Multi-source embedded named entity recognition method fusing power vocabulary information

Abstract

The invention relates to a multi-source embedded named entity recognition method fusing power vocabulary information, which comprises the following steps of: preprocessing a text to obtain preprocessed power grid field operation unsafe behavior description text data; arranging the electric power dictionary and completing BMES labeling; training electric power embedding expressions of electric power vocabularies in the electric power dictionary, and constructing an electric power word list; a multi-source embedded named entity recognition model fused with electric power vocabulary information is built, and the model comprises three functional modules including a multi-source embedded fusion layer, a feature embedded coding layer and a label decoding layer.

Description

Technical Field

[0001] The present invention relates to the field of Chinese named entity recognition in natural language processing, and specifically to a multi-source embedded named entity recognition method that integrates power vocabulary information. The method can recognize power terminology entities in unstructured text data formed by descriptions of unsafe behaviors in power grid field operations. Background Art

[0002] In recent years, with the continued expansion of power grids and rising electricity demand, grid operation and maintenance have faced increasingly complex and challenging challenges. Within this context, grid operation safety has become increasingly prominent and has become a core challenge that the power industry urgently needs to address. Accidents occurring during operations not only seriously threaten the lives of operators but also pose a significant threat to public safety and system stability. Therefore, scientifically identifying and effectively preventing and controlling unsafe behaviors during distribution network operations has become crucial for ensuring the inherent safety of operations (Stover O, Karve P, Mahadevan S. Reliability and risk metrics to assess operational adequacy and flexibility of power grids [J]. Reliability Engineering & System Safety, 2023, 231: 1090-18). In this context, grid operation safety is particularly critical, directly impacting the personal and property safety of operators. On-site operations often involve high-voltage equipment and complex systems. Accidents can result in casualties, equipment damage, and even widespread power outages, with extremely serious consequences. Therefore, ensuring the safety of grid operations is crucial to protecting the interests of people and property. Currently, major power grid companies in my country have accumulated extensive historical data on operational plans and unsafe behavior records, covering information such as operational conditions, operational content, descriptions of on-site unsafe behaviors, and types of unsafe behaviors. This data is highly valuable and represents a crucial data asset within the power system. In-depth mining and analysis of this data can help accurately identify operational risks and provide crucial support for unsafe behavior prevention and safety management. However, these unsafe behavior records are often unstructured text and contain large amounts of data. Relying on manual analysis is not only inefficient and resource-intensive, but also susceptible to subjective factors, making it difficult to guarantee objective and accurate analysis.

[0003] Named entity recognition is a core natural language processing (NLP) technology used to identify entities such as people, locations, organizations, and dates in text. It can automatically extract named entities from text data describing unsafe behaviors in power grid field operations, and can extract important information such as workers, tools, work documents, and unsafe behaviors, providing a data foundation for subsequent analysis of typical unsafe behaviors to ensure the safety of power grid field operations. Named entity recognition methods mainly include rule-based, statistical machine learning, and deep learning methods (Cheng J, Liu J, Xu X, et al. A review of Chinese named entity recognition [J]. KSII Transactions on Internet and Information Systems (TIIS), 2021, 15 (6): 2012-2030.). In recent years, deep learning technology has made significant breakthroughs in the field of natural language processing (NLP). Compared with traditional machine learning methods, deep learning models can automatically learn deep feature representations from raw text and effectively capture long-range contextual dependencies with the help of neural network structures, thereby significantly improving the ability of feature extraction and sequence labeling, while also improving the training efficiency of the model. With the continuous maturity of related technologies, deep learning-based named entity recognition methods have rapidly developed and have been widely used in professional fields such as power generation, demonstrating good practicality and promotional value. Cong et al. address the problems of limited training data and complex entity types in named entity recognition in the Chinese power sector. They proposed an entity recognition model based on lexical enhancement and global pointers. This model enhances character representations with lexical information, combines RoBERTa with deep features, and uses global pointers for efficient entity recognition, demonstrating good results in improving recognition accuracy. However, this method still has certain limitations, such as its reliance on a high-quality vocabulary, the large number of parameters in the RoBERTa model, the high training computational resources required, and its generalization ability for low-frequency entities. (Cong L, Cui R, Dou Z, et al. Named entity recognition for power databased on lexical enhancement and global pointer [C] / / Third International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT2024). SPIE, 2024, 13181:1345-1351.)Wu Chao et al. proposed a named entity recognition model for the power dispatching field based on Transformer-BiGRU-CRF, which realizes entity recognition through character modeling. Experiments show that joint training can significantly improve performance. Although this method improves the recognition effect of power entities to a certain extent, it does not introduce vocabulary information in the power field and has certain limitations when processing professional terms and entities with fuzzy boundaries (Wu Chao, Wang Hanjun. Named entity recognition method based on GRU in the power dispatching field [J]. Computer System Applications, 2020, 29(08): 185-191.). Gu Yiran et al. proposed a Chinese named entity recognition method for the motor field that integrates BERT. By enhancing semantic representation through BERT and combining BiLSTM and CRF to realize entity recognition, it effectively improves accuracy and robustness. However, this method performs poorly in dealing with the problem of uneven data distribution and fails to effectively integrate domain knowledge, which affects the recognition effect of professional entities (Gu Yiran, Huo Jianlin, Yang Haigen, et al. Chinese named entity recognition method in the motor field based on BERT [J]. Computer Engineering, 2021, 47(08):78-83+92.). In response to the problems of low entity recognition accuracy and data scarcity in the field of power grid dispatching, Mao Hongliang et al. proposed a named entity recognition method that integrates multi-head attention mechanism and BiLSTM-CRF, which effectively improved the NER performance and provided support for the construction of knowledge graphs. However, this method relies on the construction of high-quality domain dictionaries and only uses vocabulary information as features. It lacks multi-dimensional embedding representation modeling of the input, which limits the expressive power of the model (Mao Hongliang, Aizilguli, Chen Degang. Named entity recognition in the power grid dispatching field based on multi-head attention [J]. Computer Technology and Development, 2023, 33(02):181-186+194.).

[0004] In summary, various explorations have been conducted in the Chinese power sector named entity recognition task, gradually improving the ability to recognize power terminology. However, these methods still have limitations in terms of vocabulary information integration, embedding modeling dimensions, and adaptability to uneven data distribution. In particular, further improvement is needed in the systematic expression of domain knowledge and the fusion of multimodal information.

[0005] To address the above problems, the present invention proposes a multi-source embedding named entity recognition method that integrates power vocabulary information, constructs a high-quality domain dictionary suitable for on-site operation scenarios, and comprehensively utilizes multiple embedding forms (single-word embedding, double-word embedding, general dictionary embedding, and domain dictionary embedding) to represent the semantic features of the input text, thereby optimizing the named entity recognition effect of the text describing unsafe behaviors in on-site operations in the power field. Summary of the Invention

[0006] The present invention relates to a multi-source embedded named entity recognition method that integrates power vocabulary information. This method can identify power terminology entities in unstructured text data generated by descriptions of unsafe behaviors in power grid field operations. The technical solution is as follows:

[0007] A multi-source embedding named entity recognition method integrating electric power vocabulary information includes the following steps:

[0008] Step 1: text preprocessing to obtain preprocessed text data describing unsafe behaviors in power grid field operations;

[0009] Step 2: Organize the power dictionary and complete BMES annotation: For the pre-processed text data describing unsafe behaviors in power grid field operations, organize a power dictionary related to unsafe behaviors in power grid field operations that is suitable for the field of power grid operations. Using the constructed power dictionary as a reference, perform entity annotation on the pre-processed text data describing unsafe behaviors in power grid field operations to determine the entity category.

[0010] Step 3: Train the power embedding representation of the power vocabulary in the power dictionary to construct a power vocabulary;

[0011] Step 4: Build a multi-source embedded named entity recognition model that integrates power vocabulary information, including three functional modules: multi-source embedding fusion layer, feature embedding encoding layer, and label decoding layer.

[0012] The multi-source embedding fusion layer: The input text has two main lines. One is the character sequence that is processed by the BERT model and the multi-head self-attention residual module to obtain features. The other is the character sequence that is processed by single-word embedding, double-word embedding, and the introduction of general vocabulary and power vocabulary information to obtain dictionary-enhanced features to obtain a hybrid multi-source embedding. The features of the two main lines are then spliced together to obtain a fused feature representation.

[0013] The final output of the multi-source embedding layer is the concatenation of the output of the BERT model and the multi-head self-attention residual module with the character embedding, the double-word embedding, and the general dictionary and power dictionary enhanced feature embedding obtained after introducing the general vocabulary and power vocabulary information.

[0014] The feature embedding encoding layer: based on the output of the multi-source embedding fusion layer, uses a bidirectional long short-term memory neural network as a context-aware encoder to capture the deep semantic features and context-dependent information represented by the fused features output by the multi-source embedding fusion layer, and outputs a character label score vector;

[0015] The label decoding layer: based on the conditional random field (CRF), uses the transfer relationship between entity labels to calculate the probability of the global label sequence, inputs the character label score vector output by the feature embedding encoding layer, and then obtains the overall optimal label sequence.

[0016] Furthermore, in step 4, the specific method of obtaining the dictionary enhancement features after introducing the general vocabulary and power vocabulary information is as follows:

[0017] f. To identify the input text sequence X = [c1, c2, ..., c n ] Decompose by character to obtain the corresponding character sequence;

[0018] g. According to the pre-prepared general vocabulary or power vocabulary, for each character c i Perform term matching, which includes the following four types of word segmentation roles:

[0019] B (Begin): character c i The set of all terms that appear as the first letter of a word, denoted by W B (c i ).

[0020] M (Middle): character c i The set of all terms that appear as the middle word of a word, denoted by W M (c i ).

[0021] E(End): character c i The set of all terms that appear as the last letter of a word is denoted by W E (c i ).

[0022] S (Single): character c i The set of all terms that appear as single words is denoted by W S (c i ).

[0023] h. For the corresponding character c i For the term ω that matches in the general dictionary or the power dictionary, look up its corresponding word vector representation v in the general word list or the power word list. ω .

[0024] i. For the corresponding character c i For each word segmentation role matched in the general vocabulary or the power vocabulary, the vectors of all matched terms are weighted and aggregated according to their frequency of occurrence in the corpus, i.e., word frequency;

[0025] j. Normalize the aggregated feature vector under each word segmentation role and multiply it by 4 to obtain the final dictionary enhanced features.

[0026] Furthermore, in step d, let z(ω) represent the frequency of term ω, e w(w) represents the embedding vector of the term, and the aggregate feature vectors under each word segmentation role are:

[0027]

[0028]

[0029] in, Refers to the character c i The matched term ω is used as the weight of the *participle role, Refers to the character c i The sum of the frequencies of the matched term ω in all word roles.

[0030] Furthermore, the text preprocessing in step 1 is as follows:

[0031] 4) Obtain text data describing unsafe behaviors in power grid field operations from historical inspection records;

[0032] 5) performing data cleaning and preprocessing on the text data describing unsafe behaviors of power grid field operations obtained in step 1);

[0033] 6) Reorganize and obtain text data describing unsafe behaviors in power grid field operations.

[0034] Furthermore, Step 2 organizes the power dictionary and completes the BMES annotation, as follows:

[0035] 1) Based on the pre-processed text data describing unsafe behaviors in power grid field operations, an electric dictionary related to unsafe behaviors in power grid field operations suitable for the field of power grid field operations is compiled;

[0036] 2) With the constructed power dictionary as a reference, the YEDDA annotation tool is used to perform entity annotation on the preprocessed text data describing unsafe behaviors in power grid field operations. The BMES annotation method is used for entity annotation. After analyzing the text data, five entity categories are determined: Object, Person, File, Action, and Surrounding.

[0037] 3) The text data describing unsafe behaviors in power grid field operations annotated with BMES entities is used as the training corpus for building the entity recognition model.

[0038] Furthermore, the electric embedding representation of the electric vocabulary in the trained electric vocabulary dictionary in step 3 is constructed into an electric vocabulary table, specifically as follows: based on the constructed electric vocabulary, and using the pre-processed text data describing unsafe behaviors in power grid field operations as a corpus, the Skip-Gram method based on the Word2Vec model is used to train the electric embedding representation corresponding to the electric vocabulary in the electric vocabulary dictionary, and construct it into an electric vocabulary table.

[0039] Compared with the existing technology, the present invention provides a multi-source embedded named entity recognition method that integrates power vocabulary information, which has the following beneficial effects:

[0040] Compared with the existing technology, the present invention more fully combines the prior knowledge of the power field in the named entity recognition task. It not only introduces the normalization standard of power grid operation terminology in the data preprocessing stage, but also explicitly incorporates the power dictionary information compiled by joint field experts in the model construction process. Traditional entity recognition methods mostly rely on general corpus or only use character-level embedding representation, lack of deep modeling of industry terminology, resulting in low recognition accuracy in professional scenarios, especially when dealing with polysemous words, compound words or fuzzy terminology boundaries. The effect is limited. The present invention constructs an embedding representation of power vocabulary and introduces multi-source embedding information (single-word embedding, double-word embedding, general dictionary embedding and domain dictionary embedding) in the embedding layer, which effectively improves the model's recognition ability for power field terminology. At the same time, in the construction of the named entity recognition model, a multi-head self-attention residual module is introduced after BERT embedding, which further enhances the model's modeling ability for complex semantics and long-distance dependencies. This method significantly improves the professionalism, robustness and practicality of the recognition system, and is particularly suitable for highly professional data processing scenarios such as unsafe behavior texts in power grid operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flowchart of multi-source embedded named entity recognition integrating power vocabulary information

[0042] Figure 2 YEDDA marked BMES process diagram

[0043] Figure 3 Graph showing the number of entity types in the dataset

[0044] Figure 4 Word2Vec model architecture diagram

[0045] Figure 5 Structure diagram of multi-source embedding named entity recognition model integrating power vocabulary information

[0046] Figure 6 Comparison of experimental results of model recognition on the power grid unsafe behavior description dataset

[0047] Figure 7 Comparison of model recognition experimental results on the general Weibo dataset DETAILED DESCRIPTION

[0048] The present invention designs a multi-source embedded named entity recognition method that integrates power vocabulary information, which can realize the recognition of power terminology entities in unstructured text data formed by the description of unsafe behavior in power grid field operations. Specifically, Figure 1 As shown in the figure, the recognition method introduces the prior knowledge of the constructed power dictionary, explicitly introduces the power vocabulary information in the embedding layer and integrates multiple other embedding forms to improve the professionalism and accuracy of named entity recognition. Its modeling process mainly includes: text preprocessing, coordinating the power dictionary with experts in the power grid field, BMES entity labeling, embedding representation in the Word2Vec model training store, building a multi-source embedding named entity recognition model integrating power vocabulary information, and experimental results of the named entity recognition model.

[0049] The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of this application.

[0050] Step 1, text preprocessing:

[0051] 1) Obtain 24,001 text data describing unsafe behaviors in grid field operations from historical inspection records of a power grid company;

[0052] 2) Performing data cleaning and preprocessing on the text data describing unsafe behaviors in power grid field operations obtained in step 1) (including correcting typos, normalizing industry terminology, and deleting invalid values that cannot be used for named entity recognition tasks);

[0053] 3) 15,183 text data describing unsafe behaviors in power grid field operations were reorganized and obtained.

[0054] Step 2: Organize the power dictionary and complete BMES annotation:

[0055] 1) Based on the pre-processed text data of 15,183 unsafe behaviors in power grid field operations, we collaborated with experts in the field to compile a power dictionary containing 1,084 professional terms suitable for power grid field operations, such as "fence," "goggles," "safety helmet," and "crossarm."

[0056] 2) With the constructed power dictionary as a reference, the YEDDA annotation tool was used to perform entity annotation on the preprocessed text data describing unsafe behaviors in field operations of H power grids. The BMES annotation method (B-Begin: beginning of a word, M-Middle: middle of a word, E-End: end of a word, S-Single: independent word) was used for entity annotation. After analyzing the text data, five entity categories were determined: Object, Person, File, Action, and Surrounding. The data labeling process is as follows: Figure 2As shown in the figure. In the BMES annotation method, a multi-word entity such as "safety helmet" is marked as B-Object M-Object E-Object;

[0057] 3) BMES has annotated 15,183 unstructured text data describing unsafe behaviors in power grid field operations, with a total of 427,940 characters, 9,224 of which are Object, 3,237 are Person, 4,003 are File, 9,792 are Action, and 4,858 are Surrounding. The specific distribution can be found in Figure 3 Check.

[0058] 4) The 10,183 text data describing unsafe behaviors in power grid field operations annotated with BMES entities are used as the training corpus for building the entity recognition model.

[0059] Step 3: Train the power embedding representation of the power vocabulary in the power dictionary:

[0060] 1) Based on the obtained power dictionary and the pre-processed on-site unsafe behavior description text data as the corpus, the power embedding representation corresponding to the power vocabulary in the power dictionary is trained based on the Word2Vec model. The Word2Vec model is as follows: Figure 4 shown.

[0061] 2) Word2Vec is an unsupervised learning method that maps words into a dense vector space. Its core concept is to use the contextual information of words to learn word vector representations (word vectors). This model mainly includes two word vector training methods: the continuous bag-of-words (CBOW) model and the skip-gram model (Skip-Gram). The core concept of the CBOW model is to use the contextual words of a word (i.e., several words before and after) to predict the central word. The Skip-Gram model, in contrast to the CBOW approach, takes the current central word as input and aims to predict the words that appear in its context. It mainly consists of an input layer, a projection layer, and an output layer. Word2Vec is trained on large-scale corpora to map words into a low-dimensional continuous vector space, placing semantically similar words closer in space and effectively capturing the semantic connections between words. Compared to CBOW, the Skip-Gram model's word vectors are more accurate and perform better. It is particularly effective in capturing the contextual representation of rare and specialized words in small-scale or specialized corpora. Therefore, the present invention uses the Skip-Gram model to train the embedding of electric dictionary word vectors. The goal of the Skip-Gram model is to maximize the log-likelihood function, and the stochastic gradient descent algorithm is used for optimization. After training, the matrix W' corresponds to the N×V low-dimensional dense word vector of the target word, where V is the vocabulary size. Therefore, the essence of the Skip-Gram model is to perform softmax normalization on the cosine similarity between the input vector of the input word and the output vector of the target word. The optimization objective function of this model is:

[0062]

[0063] Where V is ω j The corpus.

[0064] 3) In the Skip-Gram model, the input layer only inputs a given word ω j , taking the word as the center word and setting the window size to k, the probability of the 2k words most likely to appear in the context is:

[0065]

[0066] 4) Where: h is the column vector of the input layer matrix, which is called the input vector of the central word; u j is the row vector of the output layer matrix, called ω j The output vector of The weight used in weighted summation. Since the probability of all words needs to be calculated, j = 1, 2, ···, V, corresponding to a series of weights

[0067] 5) In the present invention, the specific parameters of the model are set as word vector dimension vector_size=50, context window size window=5, and minimum word frequency threshold (words with a word frequency lower than this value will be ignored) min_count=1.

[0068] Step 4: Build a multi-source embedded named entity recognition model that integrates power vocabulary information: Most existing named entity recognition models rely on general corpus for training, and do not fully consider the large number of professional terms and compound word expressions in the power industry. This leads to problems such as missing entity recognition or inaccurate boundary recognition when dealing with tasks such as describing unsafe behaviors in power grid field operations. To this end, a multi-source embedded named entity recognition model architecture that integrates power vocabulary information is proposed. Prior knowledge in the field of power dictionary construction is introduced, and power vocabulary information is explicitly introduced in the embedding layer and integrated with multiple other embedding forms to improve the professionalism and accuracy of named entity recognition. The model structure is as follows: Figure 5 shown.

[0069] 1) The single-word embedding (gigaword_chn.all.a2b.uni.ite50.vec), bigram embedding (gigaword_chn.all.a2b.bi.ite50.vec), and general dictionary embedding (ctb.50d.vec) used in this invention are all 50-dimensional word embeddings trained on the Chinese Gigaword or CTB corpus and are publicly available; the power dictionary embedding is obtained in step 3.

[0070] 2) Build a multi-source embedded named entity recognition model that integrates power vocabulary information:

[0071] The model as a whole consists of three functional modules:

[0072] ①Multi-source embedding fusion layer

[0073] Multi-source embeddings include BERT embeddings, single-character embeddings, two-character embeddings, universal dictionary embeddings, and power dictionary embeddings. For BERT embeddings, the BERT pre-trained model generates dynamic character vector representations of text sequences. BERT input consists of three components: token embeddings, segment embeddings, and position embeddings. Single-character and two-character embeddings, as well as universal dictionary embeddings, all use the same data as the Lattice-LSTM method. The power dictionary embedding is trained using step 3.

[0074] For general dictionaries and power dictionaries, the specific method of introducing dictionary information is as follows:

[0075] a. Input text sequence to be recognized X=[c1,c2,...,c n ] Decompose by character to obtain the corresponding character sequence;

[0076] b. According to the pre-prepared general dictionary or power dictionary, for each character c i Perform word matching. The matching content includes the following four types of word segmentation roles:

[0077] B (Begin): character c i The set of all terms that appear as the first letter of a word, denoted by W B (c i ).

[0078] M (Middle): character c i The set of all terms that appear as the middle word of a word, denoted by W M (c i ).

[0079] E(End): character c i The set of all terms that appear as the last letter of a word is denoted by W E (c i ).

[0080] S (Single): character c i The set of all terms that appear as single words is denoted by W S (c i ).

[0081] c. For the corresponding character c i For the term ω that matches in the general dictionary or the power dictionary, look up its corresponding word vector representation v in the general word list or the power word list. ω .

[0082] d. For the corresponding character c i For each word segmentation role matched in the general dictionary or the power dictionary, the vectors of all matched terms are weighted and aggregated according to their frequency of occurrence (term frequency) in the corpus. Specifically, let z(ω) represent the term frequency of term ω, e w (w) represents the embedding vector of the term, then for each role:

[0083]

[0084] in,

[0085] e. To enhance the rationality of the feature magnitude, the aggregated feature vector for each role is normalized and then multiplied by 4 to obtain the final dictionary-enhanced feature:

[0086] lexicon(c i )=4·[lex B (c i )||lex M (c i )||lex E (c i )||lex S (c i )] (7)

[0087] The final output of the multi-source embedding layer is the concatenation of the output of the BERT model and the multi-head self-attention residual module with the character embedding, bigram embedding, general dictionary embedding, and power dictionary embedding. The specific formula is:

[0088] input(c i ) = h_i \;||\; e(c i ) \;||\; bigram(c i ) \;||\; lexicon_c(c i )\;||\; lexicon_d(c i ) (8)

[0089] Among them, h i : The output of BERT and multi-head self-attention residual module of the i-th word (i.e., the backbone encoding representation); e(c i ): character embedding; bigram(c i ):Double word embedding; lexicon_c(c i ):Universal dictionary embedding; lexicon_d(c i ): Domain (power) dictionary embedding. In the present invention, the corresponding embedding dimensions are dim(h i )=256,dim(e(c i ))=50,bigram(c i )=50,lexicon_c(c i )=50,lexicon_d(c i )=50.

[0090] ② Feature embedding encoding layer

[0091] For the feature encoding layer, the feature embedding encoding layer uses the BiLSTM model to extract contextual features. BiLSTM takes the output of the multi-source embedding layer as input and uses the forward and backward LSTM to calculate the forward representation of any input character. and backward representation Then the output result of BiLSTM at the moment is obtained by splicing, and its calculation formula is as follows:

[0092]

[0093] ③Tag decoding layer

[0094] For the label decoding layer, the BiLSTM model can obtain the optimal output label, but it cannot resolve the dependency between labels. For example, the "B-Object" entity label cannot be immediately followed by the "I-File" entity label; the label of the first word in a sentence should start with "B-" or "O", not "I-". Because of the existence of the above problems, its output cannot be used as a reasonable prediction result of the model. The core role of CRF

[30] (conditional random field) is indeed to model the dependency between labels through the transfer score matrix, which can take into account the implicit rules of the label sequence and output a globally optimal label sequence.

[0095] The BiLSTM output score matrix is S, S∈R n×m , s ij Represents the i-th character x in the text sequence i First calculate the j-th label score of the text sequence X={x1,x2······x n}’s predicted label sequence Y={y1,y2······y n} score, the function formula is:

[0096]

[0097] Where A is the transfer score matrix, A∈R (m+2)×(m+2) , A yi,yi+1 For label y i Move to label y i+1 Score, S i,yi Predict the label y for the i-th character of the input text sequence i probability.

[0098] Then, the normalized activation function softmax is used to calculate the probability of the output label sequence Y. The specific formula is:

[0099]

[0100] Among them, Y X is the set of all possible label sequences, is the true label sequence.

[0101] Finally, the Viterbi Algorithm is used to obtain the global optimal label sequence Y of the text sequence X * , Y * is the label set with the highest output probability. The specific formula is as follows:

[0102]

[0103] Step 5, experimental results of named entity recognition model: In order to verify the effectiveness of the proposed multi-source embedded named entity recognition method integrating power vocabulary information, the proposed multi-source embedded named entity recognition method integrating power vocabulary information is compared with the existing mainstream named entity recognition model. The specific implementation process is as follows:

[0104] (1) Divide the data set

[0105] In the experimental verification, the text data of unsafe behavior descriptions of on-site operations annotated by BMES were divided into training set, validation set and test set in the ratio of 7:2:1.

[0106] (2) Evaluation indicators of named entity recognition models

[0107] Named entity recognition is typically evaluated using three metrics: recall, precision, and F1-score. F1-score is the harmonic mean of precision and recall, used to measure the overall performance of the model. The calculation formulas for each metric are as follows:

[0108]

[0109]

[0110] Among them, TP (True Positive) is the number of true positive samples correctly classified as positive samples, FP (False Positive) is the number of negative samples misclassified as positive samples, and FN (False Negative) is the number of true positive samples misclassified as negative samples.

[0111] The precision, recall and F1 values all range from [0,1]. The larger the value, the better the prediction effect of the model.

[0112] (3) Comparison of model experimental results

[0113] In order to verify the effectiveness of the model proposed in the present invention, the present invention selected common entity recognition models for comparison, namely BiLSTM-CRF, Lattice-LSTM, BERT-CRF, BERT-BiLSTM-CRF, a total of 4 models, among which BiLSTM-CRF was used as the baseline model. After multiple experiments, it was determined that batchsize = 32, which may be related to the fact that most of the descriptions of unsafe behaviors in power grid field operations do not exceed 32 characters. In the comparative experiment, for the sake of fairness, the model constructed by the present invention and other comparative models all used Adam as the optimizer, with a learning rate of learningrate = 0.0001 and a training round of epoch = 128. The specific comparison results are as follows: Figure 6 shown.

[0114] from Figure 6 It can be seen that the model proposed in the present invention outperforms all the comparison methods in the three core evaluation indicators of precision, recall and F1 value, which fully demonstrates that the model has higher accuracy in the entity recognition task in the text describing unsafe behaviors in power grid field operations. By comparing with the BiLSTM-CRF and Lattice-LSTM models, it is found that the introduction of vocabulary information helps to improve the model's ability to recognize entity boundaries; and the comparison results between BiLSTM-CRF, BERT-CRF and BERT-BiLSTM-CRF show that fusing BERT embedding or introducing BiLSTM modules can effectively enhance the model's ability to understand contextual semantics and recognize unregistered words, thereby significantly improving the accuracy and generalization ability of entity recognition. In addition, the comparison results with the classic Chinese vocabulary fusion baseline model Lattice-LSTM further verify that the dictionary information fusion mechanism and multi-head self-attention residual module adopted by the present invention can more fully capture the semantic features in the text context, thereby improving the overall recognition effect.

[0115] In order to further verify the adaptability and effectiveness of the proposed model, since the present invention introduces general dictionary information, an experimental evaluation is also conducted on the general field Weibo dataset. Figure 7 As shown in the figure, the model achieved an average improvement of about 1% in the three indicators of precision, recall and F1 value on this dataset, which further illustrates the robustness and rationality of the named entity recognition model in this paper in cross-domain tasks.

[0116] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-source embedded named entity recognition method integrating electric power vocabulary information comprises the following steps: Step 1: text preprocessing to obtain preprocessed text data describing unsafe behaviors in power grid field operations; Step 2: Organize the power dictionary and complete BMES annotation: For the pre-processed text data describing unsafe behaviors in power grid field operations, organize a power dictionary related to unsafe behaviors in power grid field operations that is suitable for the field of power grid operations. Using the constructed power dictionary as a reference, perform entity annotation on the pre-processed text data describing unsafe behaviors in power grid field operations to determine the entity category. Step 3: Train the power embedding representation of the power vocabulary in the power dictionary to construct a power vocabulary; Step 4: Build a multi-source embedded named entity recognition model that integrates power vocabulary information, including three functional modules: multi-source embedding fusion layer, feature embedding encoding layer, and label decoding layer. The multi-source embedding fusion layer: The input text has two main lines. One is the character sequence that is processed by the BERT model and the multi-head self-attention residual module to obtain features. The other is the character sequence that is processed by single-word embedding, double-word embedding, and the introduction of general vocabulary and power vocabulary information to obtain dictionary-enhanced features to obtain a hybrid multi-source embedding. The features of the two main lines are then spliced together to obtain a fused feature representation. The final output of the multi-source embedding layer is the concatenation of the output of the BERT model and the multi-head self-attention residual module with the character embedding, the double-word embedding, and the general dictionary and power dictionary enhanced feature embedding obtained after introducing the general vocabulary and power vocabulary information. The feature embedding encoding layer: based on the output of the multi-source embedding fusion layer, uses a bidirectional long short-term memory neural network as a context-aware encoder to capture the deep semantic features and context-dependent information represented by the fused features output by the multi-source embedding fusion layer, and outputs a character label score vector; The label decoding layer: based on the conditional random field (CRF), uses the transfer relationship between entity labels to calculate the probability of the global label sequence, inputs the character label score vector output by the feature embedding encoding layer, and then obtains the overall optimal label sequence.

2. The multi-source embedded named entity recognition method integrating electric power vocabulary information according to claim 1 is characterized in that: In step 4, the specific method of obtaining dictionary enhancement features after introducing the general vocabulary and power vocabulary information is as follows: a. Input text sequence to be recognized X=[c1,c2,...,c n ] Decompose by character to obtain the corresponding character sequence; b. According to the pre-prepared general vocabulary or power vocabulary, for each character c i Perform term matching, which includes the following four types of word segmentation roles: B (Begin): character c i The set of all terms that appear as the first letter of a word, denoted by W B (c i ). M (Middle): character c i The set of all terms that appear as the middle word of a word, denoted by W M (c i ). E(End): character c i The set of all terms that appear as the last letter of a word is denoted by W E (c i ). S (Single): character c i The set of all terms that appear as single words is denoted by W S (c i ). c. For the corresponding character c i For the term ω that matches in the general dictionary or the power dictionary, look up its corresponding word vector representation v in the general word list or the power word list. ω . d. For the corresponding character c i For each word segmentation role matched in the general vocabulary or the power vocabulary, the vectors of all matched terms are weighted and aggregated according to their frequency of occurrence in the corpus, i.e., word frequency; e. Normalize the aggregated feature vector under each word segmentation role and multiply it by 4 to obtain the final dictionary enhanced features.

3. The multi-source embedded named entity recognition method integrating electric power vocabulary information according to claim 2 is characterized in that: In step d, let z(ω) represent the frequency of term ω, e w (w) represents the embedding vector of the term, and the aggregate feature vectors under each word segmentation role are: in, Refers to the character c i The matched term ω is used as the weight of the *participle role, Refers to the character c i The sum of the frequencies of the matched term ω in all word roles.

4. The multi-source embedded named entity recognition method integrating electric power vocabulary information according to claim 1 is characterized in that: The text preprocessing in step 1 is as follows: 1) Obtain text data describing unsafe behaviors in power grid field operations from historical inspection records; 2) performing data cleaning and preprocessing on the text data describing unsafe behaviors in power grid field operations obtained in step 1); 3) Reorganize and obtain text data describing unsafe behaviors in power grid field operations.

5. The multi-source embedded named entity recognition method integrating electric power vocabulary information according to claim 1 is characterized in that: Step 2: Organize the power dictionary and complete the BMES annotation, as follows: 1) Based on the pre-processed text data describing unsafe behaviors in power grid field operations, an electric dictionary related to unsafe behaviors in power grid field operations suitable for the field of power grid field operations is compiled; 2) With the constructed power dictionary as a reference, the YEDDA annotation tool is used to perform entity annotation on the preprocessed text data describing unsafe behaviors in power grid field operations. The BMES annotation method is used for entity annotation. After analyzing the text data, five entity categories are determined: Object, Person, File, Action, and Surrounding. 3) The text data describing unsafe behaviors in power grid field operations annotated with BMES entities is used as the training corpus for building the entity recognition model.

6. The multi-source embedded named entity recognition method integrating electric power vocabulary information according to claim 1 is characterized in that: In step 3, the electric embedding representation of the electric vocabulary in the training electric vocabulary dictionary is constructed as an electric vocabulary table, specifically as follows: based on the constructed electric vocabulary, and using the pre-processed text data describing unsafe behaviors in power grid field operations as a corpus, the Skip-Gram method based on the Word2Vec model is used to train the electric embedding representation corresponding to the electric vocabulary in the electric vocabulary dictionary, and construct the electric vocabulary table.

Images