SG-CIM model retrieval method and system based on improved BERT

Abstract

The invention provides an improved BERT-based SG-CIM model retrieval method and system, and is applied to the technical field of SG-CIM model retrieval. The method comprises the steps that feature extraction is conducted on input retrieval service description for an SG-CIM model through a BERT model, and retrieval features of the retrieval service description are obtained; based on the similarity between the retrieval features and various preset achievement features of model achievements for the SG-CIM model, the model achievements matched with the retrieval service description are screened out to serve as retrieval results; in the feature extraction process, if at least two words belonging to the same power business system exist in the retrieval business description, self-attention processing is carried out based on the condition of the words belonging to the same power business system. The problems that in the information retrieval process based on the keyword SG-CIM model, the retrieval precision is low, and retrieval requirements of long texts or complex descriptions are difficult to meet are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of SG-CIM model retrieval, and in particular to a SG-CIM model retrieval method and system based on improved BERT. Background Art

[0002] SG-CIM (State Grid Common Information Model) is an enterprise-level unified data model built by State Grid Corporation of China based on object-oriented modeling technology, in accordance with international standards, combining the company's core business needs, data dictionaries of in-service systems, etc., using a combination of "business demand-driven top-down" and "current situation-driven bottom-up" models. The required business-related information can be retrieved through the SG-CIM model.

[0003] At present, the retrieval of SG-CIM model results is basically done by inputting business keywords, locating SG-CIM model results containing these business keywords, and taking the most relevant SG-CIM model results as retrieval results based on relevance scoring and sorting. This process is prone to returning a large number of irrelevant results due to the lack of semantic analysis capabilities. At the same time, this process ignores the correlation between businesses, resulting in low retrieval accuracy. In addition, the existing retrieval methods are difficult to cope with the retrieval needs of long texts or complex descriptions. Summary of the invention

[0004] In order to overcome the problems of low retrieval accuracy and difficulty in coping with retrieval requirements of long texts or complex descriptions in the information retrieval process of the above-mentioned keyword-based SG-CIM model, the present invention provides a SG-CIM model retrieval method and system based on improved BERT.

[0005] On the one hand, the present invention provides an SG-CIM model retrieval method based on improved BERT, the method comprising:

[0006] Extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description;

[0007] Based on the similarity between the search feature and the preset achievement features of the model achievements for the SG-CIM model, the model achievements matching the search business description are screened out as search results;

[0008] In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

[0009] Optionally, the extracting features of the input retrieval service description for the SG-CIM model by using the BERT model to obtain retrieval features of the retrieval service description includes:

[0010] Perform multi-dimensional vector representation on the search service description through the word embedding layer of the BERT model to obtain a search embedding vector corresponding to the search service description;

[0011] The retrieval embedding vector is encoded by the encoding layer of the BERT model to obtain a retrieval encoding vector;

[0012] The retrieval encoding vector is pooled through the pooling layer of the BERT model to obtain the retrieval features of the retrieval business description.

[0013] Optionally, before performing multi-dimensional vector representation on the search service description, the method further includes:

[0014] Based on the model dictionary of the SG-CIM model, the search service description is segmented to obtain a segmentation result, wherein each word of the segmentation result corresponds to a basic word unit;

[0015] The multi-dimensional vector representation of the search service description is performed through the word embedding layer of the BERT model to obtain a search embedding vector corresponding to the search service description, including:

[0016] The word embedding layer of BERT is used to perform word embedding representation, sentence embedding representation and position embedding representation of the word basic unit corresponding to the word segmentation result, and a word embedding vector, a sentence embedding vector and a position embedding vector are obtained accordingly;

[0017] The word embedding vector, the sentence embedding vector and the position embedding vector are fused based on normalization to obtain the retrieval embedding vector.

[0018] Optionally, encoding the retrieval embedding vector through an encoding layer of a BERT model to obtain a retrieval encoding vector includes:

[0019] Based on the basic units belonging to the same power business system, self-attention processing is performed to obtain a multi-head attention matrix;

[0020] Perform residual summation and normalization on the multi-head attention matrix to obtain a normalized result;

[0021] The normalized result is processed by the feedforward neural network of the coding layer, and then residual summation and normalization are performed to obtain the overall feature representation of the retrieval service description and the feature representation of a single basic unit in the retrieval service description, that is, the retrieval coding vector.

[0022] Optionally, the self-attention processing is performed based on the basic units belonging to the same power business system to obtain a multi-head attention matrix, including:

[0023] Determine the number of basic units belonging to the same power business system in the retrieval embedding vector according to the model dictionary of the SG-CIM model; the model dictionary includes nouns of each power business system and an identifier of the power business system of each noun;

[0024] Determine the expansion factor based on the number of basic units belonging to the same power business system;

[0025] Based on the expansion factor, the key vectors of the basic units in the retrieval embedding vector belonging to the same power business system are expanded to obtain a new key matrix;

[0026] Performing self-attention processing based on the new key matrix to obtain a new self-attention matrix;

[0027] The new self-attention matrices corresponding to the multiple attention modules of the encoding layer are concatenated to obtain a multi-head attention matrix.

[0028] Optionally, the expansion factor is calculated as follows:

[0029]

[0030] Where P(n) represents the expansion factor, n is the number of basic units belonging to the same power business system;

[0031] The expression of the new self-attention matrix is:

[0032]

[0033] Among them, I-Attention is the new self-attention matrix, Q is the query matrix, and K p is the new key matrix obtained after expansion based on the expansion factor, V is the value matrix; d k represents the dimension of the new bond matrix, S softmax represents the normalization function, and the superscript T represents the matrix transpose;

[0034] The expression of the multi-head attention matrix is:

[0035] Multihead(Q,K,V)=Concat(head1,...,head i , ..., head n )W O

[0036] head i=I-Attention(QW i Q , KW i K , VW i V );

[0037] Among them, Multihead(Q, K, V) is the multi-head attention matrix, i is the attention module number of the multi-head attention of the encoding layer, head i is the new self-attention matrix of the i-th attention module in the encoding layer, head n is the new self-attention matrix of the nth attention module in the encoding layer, QW i Q , KW i K , VW i V are the weight matrices of the query matrix, the new key matrix, and the value matrix in the i-th attention module of the encoding layer, respectively, W O is the weight matrix of the output linear layer of the encoding layer.

[0038] Optionally, the performing pooling processing on the retrieval encoding vector through a pooling layer of a BERT model to obtain the extracted retrieval features of the retrieval business description includes:

[0039] Determining an attention pooling weight based on the similarity between the overall feature representation and the feature representation of the single basic unit;

[0040] Based on the attention pooling weights, weighted fusion of the overall feature representations is performed to obtain the retrieval features of the retrieval business description.

[0041] Optionally, before selecting the model achievements matching the search business description as the search results based on the similarity between the search features and the achievement features of various preset model achievements for the SG-CIM model, the method further includes:

[0042] Based on the whitening and standardization strategy, the retrieval features and the results features of the model results for the SG-CIM model are respectively reduced in dimension and standardized.

[0043] Optionally, before extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description, the method further includes:

[0044] The BERT model is used to extract features from the result set of the SG-CIM model to obtain the result features of each model result for the SG-CIM model.

[0045] On the other hand, the present invention also provides an SG-CIM model retrieval system based on improved BERT, the system comprising:

[0046] A feature extraction module, used to extract features from the input retrieval service description for the SG-CIM model through a BERT model to obtain retrieval features of the retrieval service description;

[0047] A matching and screening module, configured to screen out model achievements matching the search business description as search results based on the similarity between the search features and achievement features of various preset model achievements for the SG-CIM model;

[0048] In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

[0049] On the other hand, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0050] The memory is used to store one or more programs;

[0051] When the one or more programs are executed by the at least one processor, any one of the above-mentioned SG-CIM model retrieval methods based on improved BERT is implemented.

[0052] On the other hand, the present invention also provides a readable storage medium having an execution program stored thereon, and when the execution program is executed, the SG-CIM model retrieval method based on improved BERT described in any one of the above is implemented.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention provides an SG-CIM model retrieval method and system based on improved BERT. On the one hand, the BERT model is used to extract features of the input retrieval business description for the SG-CIM model to obtain the retrieval features of the retrieval business description; the deep bidirectional context understanding ability of the BERT model is fully utilized to generate a vector representation that captures contextual relationships and deep semantics, thereby overcoming the problem of lack of semantic understanding based on keyword retrieval technology and improving retrieval accuracy; at the same time, it can meet the needs of long texts and complex retrieval. On the other hand, in the feature extraction process, if there are at least two words in the retrieval business description that belong to the same power business system, self-attention processing is performed based on the words belonging to the same power business system, fully exploring the business relevance in the retrieval description, effectively improving the business orientation of the SG-CIM model retrieval features, and then ensuring the matching accuracy between the SG-CIM model retrieval description and the model results, and improving the retrieval accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a SG-CIM model retrieval method based on improved BERT of the present invention;

[0056] Figure 2 The second flowchart of the SG-CIM model retrieval method based on improved BERT of the present invention;

[0057] Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0058] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0059] Embodiment 1:

[0060] The present invention provides an SG-CIM model retrieval method based on improved BERT, as shown in the schematic diagram Figure 1 As shown, including:

[0061] Step S110, extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description;

[0062] Step S120, based on the similarity between the search feature and the preset achievement features of the model achievements for the SG-CIM model, the model achievements matching the search business description are screened out as search results;

[0063] In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

[0064] In this example implementation, the BERT (Bidirectional Encoder Representations from Transformers) model is a bidirectional language representation model based on the Transformer architecture proposed by the Google AI team. Through bidirectional training, it is able to simultaneously consider the contextual information in the text, thereby generating a richer and more accurate text representation. The BERT pre-training process is completed through a masked language model and a next sentence prediction task, and can then be applied to various natural language processing tasks through fine-tuning. The BERT model of the present invention is obtained by fine-tuning the pre-trained BERT model based on the retrieval description set, result set and dictionary of the SG-CIM model, and the fine-tuning process is: first construct the retrieval description set, result set and dictionary of the SG-CIM model, and then train the improved BERT model based on the retrieval description set, result set and dictionary of the SG-CIM model. This method is mainly used for the retrieval of achievements of SG-CIM model designers. The retrieval of SG-CIM model designers has strong power grid business attributes. The construction of the retrieval description set focuses on the reference of the SG-CIM model design report; the achievements of a single SG-CIM model include information such as model name, model description, and model attributes (fields); the achievement features of each model achievement for the SG-CIM model can be obtained by processing the achievement set using the trained BERT model, and the BERT model for the retrieval description and model achievements can be jointly trained until the termination condition is reached. The similarity can be cosine similarity or other physical quantities that measure the similarity between vectors. The model achievement with the highest similarity can be used as the retrieval result, or the model achievements can be arranged and output according to the similarity. The improvement of the BERT model in the present invention is that in the feature extraction process, the business relevance in the retrieval description is fully explored, and the business orientation of the SG-CIM model retrieval feature is effectively improved, thereby ensuring the matching accuracy between the SG-CIM model retrieval description and the model achievement, and improving the retrieval accuracy and efficiency; it can solve the problem of low accuracy of keyword retrieval of the current SG-CIM model, and meet the retrieval needs of model designers for SG-CIM model achievements based on business descriptions.

[0065] In an example implementation, the step S110 of extracting features from the input retrieval service description for the SG-CIM model by using the BERT model to obtain retrieval features for the retrieval service description includes:

[0066] Perform multi-dimensional vector representation on the search service description through the word embedding layer of the BERT model to obtain a search embedding vector corresponding to the search service description;

[0067] The retrieval embedding vector is encoded by the encoding layer of the BERT model to obtain a retrieval encoding vector;

[0068] The retrieval encoding vector is pooled through the pooling layer of the BERT model to obtain the retrieval features of the retrieval business description.

[0069] In this example implementation, the BERT model includes a word embedding layer, an encoding layer, and a pooling layer. The word embedding layer is used to vectorize the input data, the encoding layer is used to encode the output of the word embedding layer, and the pooling layer is used to pool the result of the encoding process. The multi-dimensional vectorized representation may include word dimensions, sentence dimensions, and position dimensions, etc. This example obtains the retrieval features of the SG-CIM model retrieval description by using the word embedding layer, encoding layer, and pooling layer of BERT. Then, the similarity between the retrieval features and the result features of the model results is measured; finally, based on the SG-CIM model retrieval description, the SG-CIM model results with the highest similarity are output, thereby laying a technical foundation for building an intelligent retrieval function based on the SG-CIM model, and meeting the SG-CIM model result retrieval needs of model designers based on business descriptions.

[0070] In an example implementation, before performing multi-dimensional vector representation on the retrieved service description in step S110, the method further includes:

[0071] Based on the model dictionary of the SG-CIM model, the search service description is segmented to obtain a segmentation result, wherein each word of the segmentation result corresponds to a basic word unit;

[0072] The multi-dimensional vector representation of the retrieval service description is performed to obtain a retrieval embedding vector corresponding to the retrieval service description, including:

[0073] The word embedding layer of BERT is used to perform word embedding representation, sentence embedding representation and position embedding representation of the word basic unit corresponding to the word segmentation result, and a word embedding vector, a sentence embedding vector and a position embedding vector are obtained accordingly;

[0074] The word embedding vector, the sentence embedding vector and the position embedding vector are fused based on normalization to obtain the retrieval embedding vector.

[0075] In this example implementation, the model dictionary of the SG-CIM model is composed of nouns of each power business system. The input retrieval business description is segmented based on the nouns in the model dictionary. Specifically, in combination with the model dictionary of the SG-CIM model, the BERT segmenter is used to decompose the retrieval business description of each SG-CIM model into a series of basic units. The basic unit can be a token corresponding to a noun. 15% of the tokens are randomly selected for masking (Mask). The BERT pre-trained model uses a full-word Mask strategy. The tokens corresponding to the retrieval business description of each SG-CIM model are input into the word embedding layer for vectorization representation in multiple dimensions, and the tokens are converted into a retrieval embedding vector. The retrieval embedding vector includes a vector of three dimensions, namely, the word embedding vector W Token , sentence embedding vector W Segment and the position embedding vector W Position , the three vectors are fully integrated using normalization. Finally, the retrieval embedding vector W input to the encoding layer of BERT is expressed as:

[0076] W=W Token +W Segment +W Position

[0077] It should be noted that in the process of fine-tuning the BERT pre-trained model, the Dropout regularization technique is used before the word embedding layer output to enhance the robustness of the retrieval embedding vector.

[0078] In some example implementations, encoding the retrieval embedding vector through an encoding layer of a BERT model to obtain a retrieval encoding vector includes:

[0079] Based on the basic units belonging to the same power business system, self-attention processing is performed to obtain a multi-head attention matrix;

[0080] Perform residual summation and normalization on the multi-head attention matrix to obtain a normalized result;

[0081] The normalized result is processed by the feedforward neural network of the coding layer, and then residual summation and normalization are performed to obtain the overall feature representation of the retrieval service description and the feature representation of a single basic unit in the retrieval service description, that is, the retrieval coding vector.

[0082] In this example implementation, self-attention processing is first performed based on the basic units belonging to the same power business system to obtain a multi-head attention matrix. Exemplarily, the specific process is as follows:

[0083] Determine the number of basic units belonging to the same power business system in the retrieval embedding vector according to the model dictionary of the SG-CIM model; the model dictionary includes nouns of each power business system and an identifier of the power business system of each noun;

[0084] Determine the expansion factor based on the number of basic units belonging to the same power business system;

[0085] Based on the expansion factor, the key vectors of the basic units in the retrieval embedding vector belonging to the same power business system are expanded to obtain a new key matrix;

[0086] Performing self-attention processing based on the new key matrix to obtain a new self-attention matrix;

[0087] The new self-attention matrices corresponding to the multiple attention modules of the encoding layer are concatenated to obtain a multi-head attention matrix.

[0088] In this example implementation, the model dictionary of the SG-CIM model is composed of nouns of various power business systems, and each power business noun is marked with the power business system to which it belongs. For example, the power business system noun "material" belongs to the ECP (Electronic Commerce Platform) and ERP (Enterprise Resource Planning) systems. "ECP" and "ERP" can be marked on the noun "material" to indicate the power business system to which it belongs. This example calculates QK in each attention module of the multi-head attention for the case where there are at least two tokens belonging to the same power business system in the retrieval description of each SG-CIM model, that is, for business features with high correlation. T Previously, the number of tokens n belonging to the same power business system in each SG-CIM model retrieval business description was determined based on the model dictionary of the SG-CIM model, and the key vector k of n tokens was n Multiply by the expansion factor P(n), the expansion factor calculation formula is as follows:

[0089]

[0090] Where P(n) represents the expansion factor, n is the number of basic units belonging to the same power business system;

[0091] At the same time, in order to better solve the problem that the calculation results are easily biased and the data is not smooth when performing dot product operations on high-dimensional vector matrices during self-attention processing, this example uses a more fine-grained calculation method during self-attention processing, as follows:

[0092]

[0093] Among them, I-Attention is the new self-attention matrix, Q is the query matrix, and K p is the new key matrix obtained after the expansion process based on the expansion factor, that is, the new key matrix composed of the product of the key vector of each token and its corresponding P(n); V is the value matrix; d k represents the dimension of the new bond matrix, S softmax In the calculation process of the new self-attention matrix, the query matrix is ​​squared to reduce the result deviation of the dot product operation of the high-dimensional vector matrix, thereby avoiding the problem of data non-smoothing.

[0094] On this basis, the self-attention matrices of multiple attention modules of the multi-head attention are spliced ​​to form a multi-head attention matrix, and its calculation formula is as follows:

[0095] Multihead(Q,K,V)=Concat(head1,...,head i , ..., head n )W O

[0096] head i =I-Attention(QW i Q , KW i K , VW i V );

[0097] Among them, i is the attention module number of the multi-head attention in the encoding layer, head i is the new self-attention matrix of the i-th attention module in the encoding layer, QW i Q , KW i K , VW i V are the weight matrices of the query matrix, the new key matrix, and the value matrix in the i-th attention module of the encoding layer, respectively, W O is the weight matrix of the output linear layer of the encoding layer.

[0098] After obtaining the multi-head attention matrix, the residual summation and normalization of the obtained multi-head attention matrix are performed, and the normalized result is processed by a layer of feedforward neural network in the encoding layer and then the residual summation and normalization are performed again to generate the overall feature representation X CLS =[x1, x2, ..., x j , ..., xN ] and the retrieval business description of each SG-CIM model single token feature representation E token =[e1, e2, ..., e j , ..., e N ], x j , e j ∈R d , where x j is the overall feature representation X CLS The jth vector in x N is the overall feature representation X CLS The Nth vector in, e j For a single feature representation E token The jth vector of N For a single feature representation E token The Nth vector of , N is the total number of vectors representing the overall feature or the total number of vectors representing a single feature, R represents a real space vector, and d is the vector dimension.

[0099] In some example implementations, the performing pooling processing on the retrieval encoding vector to obtain the retrieval feature for retrieving the service description includes:

[0100] Determining an attention pooling weight based on the similarity between the overall feature representation and the feature representation of the single basic unit;

[0101] Based on the attention pooling weights, weighted fusion of the overall feature representations is performed to obtain the retrieval features of the retrieval business description.

[0102] In this example implementation, in order to completely extract multiple important features of the retrieval service description of each SG-CIM model, attention pooling is used as the model pooling layer, and the feature representation after the final attention pooling process is:

[0103]

[0104] Among them, S represents the output result of the attention pooling layer of the SG-CIM model for retrieving business descriptions. represents a k-dimensional real vector, a j is the attention pooling weight, calculated as a j =S Softmax (sim(x j ·e j )), function sim(x j ·e j) is used to measure the similarity between the overall feature representation and the feature representation of a single token. This example improves the BERT pooling method, which generally uses max pooling and mean pooling, and introduces the idea of ​​attention into the BERT pooling layer. By using the attention pooling method, the most prominent features are generally used as the overall feature representation, which can extract multiple important vector features of the retrieval business description or model results of the SG-CIM model, and ensure the integrity of the SG-CIM model retrieval description or model result vector feature representation.

[0105] In some implementations, the BERT model is used to perform feature extraction on the result set of the SG-CIM model to obtain the result features of each model result of the SG-CIM model.

[0106] In this example implementation, the process of extracting the achievement features of the model achievements of the SG-CIM model is similar to the process of extracting the retrieval features. The BERT model used in obtaining the achievement features of the model achievements can be obtained by fine-tuning the pre-trained BERT model based on the achievement set of the SG-CIM model. The achievement features of the model achievements of a single SG-CIM model can be represented as U.

[0107] In some example implementations, in order to address the problem that the features obtained from BERT model training are conically distributed and thus affect the similarity concentration between the SG-CIM model retrieval description set and the SG-CIM model result set, before screening out the model results matching the retrieval business description as retrieval results based on the similarity between the retrieval features and the preset result features of the model results for the SG-CIM model, the retrieval features and the result features of the model results for the SG-CIM model can be respectively reduced in dimension and standardized based on a whitening and normalization strategy.

[0108] In this example implementation, a whitening standardization strategy is used to perform dimension reduction and standardization processing on the search features and result features of the SG-CIM model, respectively. The specific process is as follows:

[0109] 1) Calculate the mean μ of the retrieval features and result features of each SG-CIM model respectively S , μ U And covariance matrices ∑S, ∑U.

[0110] 2) Perform SVD singular value decomposition on the covariance matrices ∑S and ∑U of the search features and result features respectively, and obtain Y S , Z S ,Y TS and Y U , Z U , Y T U , where Y S is the orthogonal matrix of the singular value decomposition of ∑S, Y U is the orthogonal matrix of the singular value decomposition of ∑U, Z S is the diagonal matrix of the singular value decomposition of ∑S, Z U is the diagonal matrix of the singular value decomposition of ∑U. The superscript T represents the transpose of the matrix.

[0111] 3) The Whitening standardization strategy is used to reduce the dimension of the retrieval features and the results features. The standardized single SG-CIM model retrieval features and SG-CIM model results features are:

[0112]

[0113] in, is the standardized retrieval feature, is the standardized result feature, S is the search feature, U is the result feature, and W S for The first f columns, W U for The first f columns of , where f represents the dimension after dimensionality reduction.

[0114] In this example, the SG-CIM model retrieval features and the SG-CIM model result features are processed using the dimensionality reduction Whitening normalization strategy to adaptively optimize the traditional BERT model, effectively improve the similarity measurement of the SG-CIM model retrieval business description and result features, and thus ensure the matching accuracy between the retrieval description and the model results.

[0115] In some example implementations, the step S120 of selecting the model achievements matching the search business description as the search results based on the similarity between the search feature and the achievement features of each preset model achievement for the SG-CIM model includes the following process:

[0116] First, the similarity between the standardized SG-CIM model retrieval features and the achievement features is calculated. Specifically, the cosine similarity is used to measure the similarity between the SG-CIM model retrieval features and the achievement features. The formula is as follows:

[0117]

[0118] in, Respectively and The Euclidean norm of , Similarity is and The similarity.

[0119] Then, a threshold is set according to the similarity distribution. If the similarity is greater than the threshold, it indicates that the retrieval features of the SG-CIM model are similar to the results features, and the SG-CIM model results correspond to the retrieval business description / question. The retrieval business description of the SG-CIM model can be used as a benchmark, and the similarities can be sorted from large to small to obtain the model results of the SG-CIM model with the highest similarity as the retrieval result. This example matches the retrieval description features and the results features based on text similarity. Text similarity refers to evaluating the degree of similarity between two texts by comparing their content, semantics, or structure. Common methods include comparisons based on vocabulary, grammar, and semantics, as well as calculations based on vector representations, to quantify the similarity between texts, which is helpful for many tasks in information retrieval, recommendation systems, and natural language processing.

[0120] The following is a specific example to illustrate the specific steps of the SG-CIM model retrieval method based on improved BERT of the present invention. Figure 2 As shown, the following processes may be included:

[0121] (1) Obtain the SG-CIM model problem set based on the input retrieval business description.

[0122] (2) Based on the word embedding layer, encoding layer and pooling layer of the improved BERT corresponding to the description set, the retrieval description features of the SG-CIM model question set are extracted.

[0123] (3) Based on the word embedding layer, encoding layer and pooling layer of the improved BERT corresponding to the result set, the result features of the SG-CIM model result set are extracted.

[0124] (4) Use the dimension reduction Whitening normalization strategy to normalize the retrieval description features in (2) and the result features in (3).

[0125] (5) Calculate the standardized retrieval description features and result features in (4) based on cosine similarity.

[0126] (6) Taking the SG-CIM model problem as a benchmark, obtain the SG-CIM model results corresponding to the results features with the highest similarity in (5), and output the SG-CIM model results with the highest similarity.

[0127] The present invention mainly solves the problems of low retrieval accuracy of the current SG-CIM model, insufficient recognition of the relationship between SG-CIM and the same-system professional vocabulary by the traditional attention mechanism, and obvious limitations of the BERT model vector representation. The specific problems are as follows:

[0128] (1) The current SG-CIM model has low retrieval accuracy

[0129] First, keyword-based retrieval technology mainly relies on precise literal matching, which makes it difficult to capture the deep semantics and contextual relationships of keywords, resulting in insufficient processing of polysemous words and synonyms, and failing to accurately reflect the real query intent of model designers; second, this technology may ignore the importance of word order and syntactic structure, and there are timeliness issues in index construction and updating, which can easily introduce noise and irrelevant information, affecting the accuracy and recall of retrieval; finally, the processing capability for long-tail queries and complex queries is weak, which limits its applicability in dealing with diverse needs and complex scenarios. In other words, the current keyword-based retrieval technology lacks semantic analysis capabilities, is prone to returning a large number of irrelevant results, and is difficult to meet the long text or complex description retrieval needs of SG-CIM model designers.

[0130] (2) The traditional attention mechanism is not good at identifying the relationship between SG-CIM and the same system professional vocabulary

[0131] In the power grid business system, if two professional terms belong to the same business system, it often means that there is a strong business correlation between the two terms. The traditional attention mechanism only measures the degree of correlation between different terms through attention scores, which easily ignores the situation where two terms have small attention scores but belong to the same business system, resulting in the lack of business orientation of the vector feature representation of the SG-CIM model retrieval description or results.

[0132] (3) The BERT model vector representation has certain limitations

[0133] The word vectors and sentence vectors obtained by the BERT model tend to show cone-shaped distribution characteristics. Therefore, when the cosine is used to directly measure the similarity of the SG-CIM model retrieval description or result vector representation obtained by the BERT model, it is easy for the similarity values ​​of most SG-CIM model retrieval descriptions and results to be concentrated in a certain range, and thus it is impossible to effectively and accurately identify the SG-CIM results corresponding to different SG-CIM problem descriptions.

[0134] In view of the above problems, the present invention constructs an SG-CIM model retrieval description set, a result set, and a dictionary to form the basic data of the SG-CIM model retrieval technology, and fine-tunes the BERT model; then, the SG-CIM model retrieval features and result features are obtained through the trained BERT word embedding layer, encoding layer, and pooling layer; the SG-CIM model retrieval description and result vector feature representation are standardized using the dimensionality reduction Whitening normalization strategy; the similarity between the SG-CIM model retrieval description and the result is measured based on the cosine similarity; the SG-CIM model retrieval description is used as a benchmark, and the similarities are sorted from large to small to obtain the SG-CIM model result with the highest similarity, thereby solving the problem that the SG-CIM model keyword retrieval has low accuracy and cannot meet the long text or complex description retrieval needs of SG-CIM model designers. The present invention measures the similarity of the SG-CIM model retrieval description vector features and the SG-CIM model result vector features after BERT training and dimensionality reduction and Whitening standardization, thereby realizing SG-CIM model result retrieval based on the SG-CIM model retrieval description, meeting the retrieval needs of model designers based on business descriptions, accurately and quickly locating data model design results, and judging whether the existing data model results meet the current design needs.

[0135] The present invention uses the BERT word embedding layer to embed vector representations of each SG-CIM model retrieval business description and model results, and on this basis uses the adaptively improved BERT encoding layer (Encoder layer) and attention pooling layer to output vector feature representations, and uses the dimension reduction Whitening normalization strategy to standardize the SG-CIM model retrieval description and the result vector feature representation; measures the similarity between the SG-CIM model retrieval description and the model results based on cosine similarity; takes the SG-CIM model retrieval description as a benchmark, sorts them from large to small according to the similarity, and obtains the SG-CIM model results with the highest similarity. The present invention makes full use of the deep bidirectional context understanding ability of the BERT model, and generates vector representations that capture contextual relationships and deep semantics through the BERT word embedding layer, encoding layer and pooling layer, which overcomes the problem of lack of semantic understanding of the keyword-based SG-CIM model retrieval technology, and can meet the complex SG-CIM model retrieval needs of model designers. At the same time, in response to the problems such as the insufficient recognition of the relationship between SG-CIM and the same-system professional vocabulary by the BERT model attention mechanism and the obvious limitations of the BERT model vector representation, the present invention uses the expansion factor and the dimensionality reduction Whitening normalization strategy to adaptively optimize the traditional BERT model, effectively improving the business orientation and similarity measurement of the SG-CIM model retrieval description and the result vector representation, thereby ensuring the matching accuracy between the SG-CIM model retrieval description and the model results, and improving the model result query efficiency and application accuracy of model designers.

[0136] The technical solution of the present invention is specifically applied to the SG-CIM model achievement retrieval scenario of model designers, especially the model achievement retrieval based on complex power business descriptions. The SG-CIM model retrieval description and the SG-CIM model achievement feature vector representation are obtained by using the BERT word embedding layer, encoding layer and pooling layer. Then, the two types of feature vector representations are subjected to dimensionality reduction Whitening normalization processing, and the cosine similarity is used to measure the similarity of the two types of feature vector representations. Finally, based on the SG-CIM model retrieval description, the SG-CIM model achievement with the highest similarity is output, thereby laying a technical foundation for building an intelligent retrieval function based on the SG-CIM model, and meeting the SG-CIM model achievement retrieval needs of model designers based on business descriptions.

[0137] Embodiment 2:

[0138] Based on the same inventive concept, the present invention also provides a SG-CIM model retrieval system based on improved BERT, the system comprising:

[0139] A feature extraction module, used to extract features from the input retrieval service description for the SG-CIM model through a BERT model to obtain retrieval features of the retrieval service description;

[0140] A matching and screening module, configured to screen out model achievements matching the search business description as search results based on the similarity between the search features and achievement features of various preset model achievements for the SG-CIM model;

[0141] In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

[0142] In a possible implementation, the feature extraction module includes:

[0143] A vectorization submodule, used to perform multi-dimensional vectorization representation on the retrieval service description through the word embedding layer of the BERT model to obtain a retrieval embedding vector corresponding to the retrieval service description;

[0144] An encoding submodule, used for encoding the retrieval embedding vector through the encoding layer of the BERT model to obtain a retrieval encoding vector;

[0145] The pooling submodule is used to perform pooling processing on the retrieval encoding vector through the pooling layer of the BERT model to obtain the retrieval features of the retrieval business description.

[0146] In a possible implementation, the vectorization submodule includes:

[0147] A word segmentation subunit, configured to segment the search service description based on the model dictionary of the SG-CIM model to obtain a word segmentation result, wherein each word of the word segmentation result corresponds to a word basic unit;

[0148] An embedding subunit, used for performing word embedding representation, sentence embedding representation and position embedding representation of the word basic unit corresponding to the word segmentation result through the word embedding layer of BERT, and obtaining a word embedding vector, a sentence embedding vector and a position embedding vector respectively;

[0149] The fusion subunit is used to perform a normalization-based fusion process on the word embedding vector, the sentence embedding vector and the position embedding vector to obtain the retrieval embedding vector.

[0150] In a possible implementation, the encoding submodule includes:

[0151] A self-attention subunit is used to perform self-attention processing based on the basic units belonging to the same power business system to obtain a multi-head attention matrix;

[0152] The feedforward subunit is used to perform residual summing and normalization on the multi-head attention matrix to obtain a normalized result; the normalized result is processed by the feedforward neural network of the encoding layer and then residual summing and normalization are performed to obtain the overall feature representation of the retrieval service description and the feature representation of a single basic unit in the retrieval service description, that is, the retrieval coding vector.

[0153] In one possible implementation, the self-attention subunit is specifically used to:

[0154] Determine the number of basic units belonging to the same power business system in the retrieval embedding vector according to the model dictionary of the SG-CIM model; the model dictionary includes nouns of each power business system and an identifier of the power business system of each noun;

[0155] Determine the expansion factor based on the number of basic units belonging to the same power business system;

[0156] Based on the expansion factor, the key vectors of the basic units in the retrieval embedding vector belonging to the same power business system are expanded to obtain a new key matrix;

[0157] Performing self-attention processing based on the new key matrix to obtain a new self-attention matrix;

[0158] The new self-attention matrices corresponding to the multiple attention modules of the encoding layer are concatenated to obtain a multi-head attention matrix.

[0159] In a possible implementation, the expansion factor is calculated as follows:

[0160]

[0161] Where P(n) represents the expansion factor, n is the number of basic units belonging to the same power business system;

[0162] The expression of the new self-attention matrix is:

[0163]

[0164] Among them, I-Attention is the new self-attention matrix, Q is the query matrix, and K p is the new key matrix obtained after expansion based on the expansion factor, V is the value matrix; d k represents the dimension of the new bond matrix, S softmax represents the normalization function, and the superscript T represents the matrix transpose;

[0165] The expression of the multi-head attention matrix is:

[0166] Multihead(Q,K,V)=Concat(head1,...,head i , ..., head n )W O

[0167] head i =I-Attention(QW i Q , KW i K , VW i V );

[0168] Among them, i is the attention module number of the multi-head attention in the encoding layer, head i is the new self-attention matrix of the i-th attention module in the encoding layer, QW i Q , KW i K , VW i V are the weight matrices of the query matrix, the new key matrix, and the value matrix in the i-th attention module of the encoding layer, respectively, W O is the weight matrix of the output linear layer of the encoding layer.

[0169] In a possible implementation manner, the pooling submodule is specifically used for:

[0170] Determining an attention pooling weight based on the similarity between the overall feature representation and the feature representation of the single basic unit;

[0171] Based on the attention pooling weights, weighted fusion of the overall feature representations is performed to obtain the retrieval features of the retrieval business description.

[0172] In a possible implementation, the system further includes:

[0173] The standardization module is used to perform dimension reduction and standardization processing on the retrieval features and the results features of the model results of the SG-CIM model based on the whitening standardization strategy.

[0174] In a possible implementation, the feature extraction module is further used for:

[0175] The BERT model is used to extract features from the result set of the SG-CIM model to obtain the result features of each model result for the SG-CIM model.

[0176] Embodiment 3:

[0177] like Figure 3 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0178] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of an SG-CIM model retrieval method based on an improved BERT in the above embodiment.

[0179] Example 4

[0180] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of an SG-CIM model retrieval method based on improved BERT in the above embodiment.

[0181] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A SG-CIM model retrieval method based on improved BERT, characterized in that: The method comprises: Extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description; Based on the similarity between the search feature and the preset achievement features of the model achievements for the SG-CIM model, the model achievements matching the search business description are screened out as search results; In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

2. The method according to claim 1, characterized in that The step of extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description includes: Perform multi-dimensional vector representation on the search service description through the word embedding layer of the BERT model to obtain a search embedding vector corresponding to the search service description; The retrieval embedding vector is encoded by the encoding layer of the BERT model to obtain a retrieval encoding vector; The retrieval encoding vector is pooled through the pooling layer of the BERT model to obtain the retrieval features of the retrieval business description.

3. The method according to claim 2, characterized in that Before performing multi-dimensional vector representation on the search service description, the method further includes: Based on the model dictionary of the SG-CIM model, the search service description is segmented to obtain a segmentation result, wherein each word of the segmentation result corresponds to a basic word unit; The multi-dimensional vector representation of the search service description is performed through the word embedding layer of the BERT model to obtain a search embedding vector corresponding to the search service description, including: The word embedding layer of BERT is used to perform word embedding representation, sentence embedding representation and position embedding representation of the word basic unit corresponding to the word segmentation result, and a word embedding vector, a sentence embedding vector and a position embedding vector are obtained accordingly; The word embedding vector, the sentence embedding vector and the position embedding vector are fused based on normalization to obtain the retrieval embedding vector.

4. The method according to claim 3, characterized in that The encoding process of the retrieval embedding vector by the encoding layer of the BERT model to obtain the retrieval encoding vector includes: Based on the basic units belonging to the same power business system, self-attention processing is performed to obtain a multi-head attention matrix; Perform residual summation and normalization on the multi-head attention matrix to obtain a normalized result; The normalized result is processed by the feedforward neural network of the coding layer, and then residual summation and normalization are performed to obtain the overall feature representation of the retrieval service description and the feature representation of a single basic unit in the retrieval service description, that is, the retrieval coding vector.

5. The method according to claim 4, characterized in that The self-attention processing is performed based on the basic units belonging to the same power business system to obtain a multi-head attention matrix, including: Determine the number of basic units belonging to the same power business system in the retrieval embedding vector according to the model dictionary of the SG-CIM model; the model dictionary includes nouns of each power business system and an identifier of the power business system of each noun; Determine the expansion factor based on the number of basic units belonging to the same power business system; Based on the expansion factor, the key vectors of the basic units in the retrieval embedding vector belonging to the same power business system are expanded to obtain a new key matrix; Performing self-attention processing based on the new key matrix to obtain a new self-attention matrix; The new self-attention matrices corresponding to the multiple attention modules of the encoding layer are concatenated to obtain a multi-head attention matrix.

6. The method according to claim 5, characterized in that The expansion factor is calculated as follows: Where P(n) represents the expansion factor, n is the number of basic units belonging to the same power business system; The expression of the new self-attention matrix is: Among them, I-Attention is the new self-attention matrix, Q is the query matrix, and K p is the new key matrix obtained after expansion based on the expansion factor, V is the value matrix; d k represents the dimension of the new bond matrix, S softmax represents the normalization function, and the superscript T represents the matrix transpose; The expression of the multi-head attention matrix is: Multihead(Q,K,V)=Concat(head1,…,head i ,…,head n )W O head i =I-Attention(QW i Q ,KW i K ,VW i V ); Among them, Multihead(Q,K,V) is the multi-head attention matrix, i is the attention module number of the multi-head attention of the encoding layer, head i is the new self-attention matrix of the i-th attention module in the encoding layer, head n is the new self-attention matrix of the nth attention module in the encoding layer, QW i Q ,KW i K ,VW i V are the weight matrices of the query matrix, the new key matrix, and the value matrix in the i-th attention module of the encoding layer, respectively, W O is the weight matrix of the output linear layer of the encoding layer.

7. The method according to claim 4, characterized in that The pooling process is performed on the retrieval encoding vector by the pooling layer of the BERT model to obtain the extracted retrieval features of the retrieval business description, including: Determining an attention pooling weight based on the similarity between the overall feature representation and the feature representation of the single basic unit; Based on the attention pooling weights, weighted fusion of the overall feature representations is performed to obtain the retrieval features of the retrieval business description.

8. The method according to any one of claims 1 to 7, characterized in that: Before selecting the model achievements matching the search business description as search results based on the similarity between the search features and the achievement features of the preset model achievements for the SG-CIM model, the method further includes: Based on the whitening and standardization strategy, the retrieval features and the results features of the model results for the SG-CIM model are respectively reduced in dimension and standardized.

9. The method according to claim 1, characterized in that: Before extracting features of the input retrieval service description for the SG-CIM model through the BERT model to obtain retrieval features of the retrieval service description, the method further includes: The BERT model is used to extract features from the result set of the SG-CIM model to obtain the result features of each model result for the SG-CIM model.

10. An SG-CIM model retrieval system based on improved BERT, characterized in that: The system comprises: A feature extraction module, used to extract features from the input retrieval service description for the SG-CIM model through a BERT model to obtain retrieval features of the retrieval service description; A matching and screening module, configured to screen out model achievements matching the search business description as search results based on the similarity between the search features and achievement features of various preset model achievements for the SG-CIM model; In the feature extraction process, if there are at least two words in the retrieved business description that belong to the same electric power business system, self-attention processing is performed based on the words that belong to the same electric power business system.

Images