Power entity recognition method, storage medium and device based on multi-layer neural network

Abstract

The present invention discloses a method, storage medium and device for electric entity recognition based on a multi-layer neural network. The electric power corpus to be recognized is input into a pre-constructed BERT electric power entity recognition model to obtain the Huffman coding of the electric power entity label, and the entity label is obtained through Huffman coding mapping, thereby obtaining the recognized entity. The BERT language model is pre-trained through the language model training corpus; the electric power corpus data is labeled with electric power entity labels to construct the electric power entity recognition corpus; the Huffman coding of the electric power entity label is constructed according to the number of electric power entity labels in the electric power entity recognition corpus; a classification layer is added after the pre-trained BERT language model to form a BERT electric power entity recognition model, and the BERT electric power entity recognition model is re-trained through the electric power entity recognition corpus to obtain a trained BERT electric power entity recognition model. The accuracy of Chinese named entity recognition in the electric power field is improved.

Description

Technical Field

[0001] This invention relates to the field of power entity recognition technology, and specifically to a power entity recognition method, storage medium, and device based on a multi-layer neural network. Background Technology

[0002] Named Entity Recognition (NER) (also known as entity recognition, entity segmentation, and entity extraction) is a subtask of information extraction used to identify names of people, places, organizations, or other named entities in input text, based on specific needs. Its aim is to locate and classify named entities in the input text into predefined categories. Traditional named entity recognition involves recognition tasks covering three main categories (entities, time, and numbers) and seven subcategories (names, places, time, values, currency, and percentages). Traditional named entity recognition methods can be categorized into dictionary-based methods, rule-based methods, and methods based on traditional machine learning.

[0003] Early research relied on rule-based methods, but the human cost of formulating and maintaining these rules was high. Machine learning methods, such as Conditional Random Field (CRF) models, learned features by building log-likelihood models, but training was costly and slow. Deep learning models can automatically learn features; Long Short-Term Memory (LSTM) networks can learn long-range features through gating units, and attention mechanisms can focus on information more critical to the NER task from a large amount of input.

[0004] Most existing Named Entity Recognition (NER) methods are data-driven, meaning the larger the dataset, the better the model's learning performance. However, in certain domains, it's difficult to build sufficient labeled corpora, significantly reducing model effectiveness. Currently, building named entity recognition tools in the power industry suffers from insufficient labeled corpora. Furthermore, named entity recognition tasks often exhibit label imbalance, where different entities appear with significantly different frequencies. Models trained on such data tend to predict the most frequent labels, and insufficient corpora exacerbate the difficulty of handling this imbalance. Manual annotation requires specialized knowledge in the power industry, making it difficult for ordinary people to accurately identify entities in this field, resulting in high costs and slow annotation processes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method, storage medium, and device for identifying power entities based on a multi-layer neural network, which solves the problems of unbalanced power entity identification tags, inaccurate identification, and slow manual labeling.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for recognizing power entities based on multi-layer neural networks, comprising the steps of: inputting the power corpus to be recognized into a pre-constructed BERT power entity recognition model to obtain the Huffman code of the power entity label, obtaining the entity label through the Huffman code mapping, and then obtaining the recognized entity.

[0007] Furthermore, the construction steps of the BERT power entity recognition model include:

[0008] Extract massive text corpora, perform data preprocessing on massive text corpora to obtain language model training data;

[0009] Pre-training the BERT language model using language model training corpus;

[0010] Label the power entity data with power entity tags to construct a power entity recognition corpus;

[0011] Construct Huffman coding for power entity tags based on the number of power entity tags in the power entity recognition corpus;

[0012] A classification layer is added after the pre-trained BERT language model to form the BERT power entity recognition model. The BERT power entity recognition model is then retrained using power entity recognition corpus to obtain the trained BERT power entity recognition model.

[0013] Furthermore, the data preprocessing process for the massive text corpus includes:

[0014] The text is segmented into sentences and sentence pairs are constructed. Sentence pairs are connected using predefined connection tags. Predefined header tags are added to the beginning of sentence pairs, and predefined tail tags are added to the end of sentence pairs. Sentence pairs consisting of connected sentences in the original text are positive samples, and unconnected sentences are negative samples. The corpus for the sentence relationship prediction task is constructed.

[0015] In each sentence, the words in the randomly masked part are used for prediction; for the masked words, part of them are replaced with a set string label, part of them are replaced with random words, and the remaining part remains unchanged, which constitutes the corpus for the word prediction task.

[0016] Word labels are generated based on the real words in the obscured positions, and sentence relationship labels are generated based on the relationships between sentence pairs, thus obtaining the training corpus for the language model.

[0017] Furthermore, the pre-training of the BERT language model using the language model training corpus includes the following steps:

[0018] The BERT language model takes preprocessed text as input and outputs word labels and sentence relationship labels.

[0019] Calculate the loss values ​​of the BERT language model output and the true labels. Sum the loss values ​​of the word labels and the loss values ​​of the context relationship to obtain the final loss value. Train the BERT language model using the AdamW optimizer based on the final loss value. Stop training when the model's loss value on the validation set no longer decreases, save the model parameters, and obtain the BERT language model.

[0020] Furthermore, the step of labeling power entity data with power entity tags to construct power entity recognition corpus includes: manually labeling a portion of the power data to obtain a knowledge base of power entities; and using this knowledge base to perform non-manual power entity tagging on the remaining power data to obtain power entity recognition corpus.

[0021] Furthermore, the non-man-made electrical entity labeling includes:

[0022] The power entity recognition corpus was constructed using the BMEO tagging method. If a character unit is the beginning of an entity word, it is tagged as B-entity category; if a character unit is the end of an entity word, it is tagged as E-entity category; if a character unit is neither the beginning nor the end of an entity word, it is tagged as M-entity category; if a character does not belong to an entity word, it is tagged as O.

[0023] Furthermore, the classification layer includes a cascaded fully connected layer and a Sigmoid activation function. The input to the classification layer is the output of the BERT language model, and the output of the classification layer is the Huffman code of the predicted power entity label.

[0024] Further, training the BERT power entity recognition model includes:

[0025] Input the power entity recognition corpus into the BERT power entity recognition model, and output the Huffman code of the predicted power entity label. The corresponding power entity label is obtained by mapping through the Huffman code.

[0026] Cross-entropy loss is used to calculate the difference between the real labels on the power entity recognition corpus and the output labels of the BERT power entity recognition model. The BERT power entity recognition model is trained using the AdamW optimizer. When the loss of the model on the power entity recognition corpus validation set no longer decreases, training is stopped, the model parameters are saved, and the trained BERT power entity recognition model is obtained.

[0027] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform a power entity identification method based on a multilayer neural network according to any of the preceding claims.

[0028] A computing device, comprising,

[0029] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing the power entity identification method based on any of the foregoing.

[0030] The beneficial effects achieved by this invention are as follows:

[0031] 1. The Huffman coding of entity tags in this invention can effectively alleviate the problem of entity tag imbalance in the power field through the Huffman tree structure, and improve the accuracy of Chinese named entity recognition in the power field.

[0032] 2. The pseudo-labeling data annotation method in this invention can effectively reduce the human cost of entity recognition text annotation;

[0033] 3. This invention enhances the semantic representation of words through the BERT pre-trained model, reduces training parameters through fine-tuning, saves training time, and achieves good model performance even with a small amount of data. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the data annotation process in a specific embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the distribution of power entity tags in a specific embodiment of the present invention;

[0036] Figure 3 This is a flowchart illustrating the training process of the BERT power entity recognition model in a specific embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of the BERT power entity recognition model structure in a specific embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of Huffman coding in a specific embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0040] Example 1:

[0041] A method for identifying electrical entities based on a multi-layer neural network includes the following steps:

[0042] Steps 1-5 below describe the training process for the BERT power entity recognition model, as shown in the flowchart below. Figure 3 As shown;

[0043] Step 1: Crawl massive amounts of text from Wikipedia and Baidu Baike to form a massive text corpus;

[0044] Step 2: Preprocess the massive text corpus to form language model training data;

[0045] The data preprocessing process for massive text corpora mainly includes the following three parts:

[0046] (1) Perform character-level segmentation on massive text corpora;

[0047] (2) The text after character segmentation is divided into sentences and sentence pairs are constructed. Sentence pairs exceeding the preset length max-num-tokens are truncated. Sentence pairs are connected with the set connection tag [SEP]. The sentence pair is marked with the set header tag [CLS] and the sentence pair is marked with the set tail tag [SEP]. Sentence pairs formed by connected sentences in the original text are positive samples, and unconnected sentences are negative samples. The corpus for the sentence relationship prediction task is constructed.

[0048] (3) In each sentence, 15% of the words are randomly masked for prediction; for the masked words, 80% are replaced with the set string [MASK], 10% are replaced with a random word, and 10% remain unchanged; this constitutes the corpus for the word prediction task.

[0049] (4) Generate word tags based on the real words in the obscured positions in (2), and generate sentence relationship tags based on the sentence pairs in (3);

[0050] (5) For the language model training corpus constructed in the above manner, the training set and the validation set are divided in a ratio of 9:1.

[0051] Step 3: Pre-train the BERT language model using language model training corpus;

[0052] The BERT (Bidirectional Transformer Encoding) language model is an existing model that contains multiple layers of neural networks. It first constructs a vector sequence from a text sequence through an embedding layer, and then encodes the context through a transformer encoder, which can realize the transformation from a text sequence to a label sequence or from a text sequence to a single label.

[0053] The pre-training task of language models consists of two parts: word prediction and sentence-to-sentence relationship prediction.

[0054] In the character prediction task, the input is preprocessed text, and the output is the predicted character at the [MASK] label position obtained from the BERT language model. In the sentence relationship prediction task, the input is also preprocessed text, and the output is the sentence relationship at the [CLS] position obtained from the BERT language model. Since the inputs for the two tasks are the same, two types of predicted labels can be obtained for the same input data, including character labels and sentence relationship labels.

[0055] The BERT language model is optimized by calculating the loss between the output and the ground truth labels at corresponding positions. The loss includes the loss for word labels and the loss for context relationships, calculated using the cross-entropy formula. The final loss value is obtained by summing the loss values ​​for word labels and context relationships. Based on this final loss value, the BERT language model is trained on the training set using the momentum and weight decay optimizer (AdamW). Training stops when the model's loss value on the validation set no longer decreases, and the model parameters are saved.

[0056] Step 4: Label the power entity data with power entity tags (i.e. perform BMEO annotation), construct the power entity recognition corpus, and preprocess the power entity recognition corpus;

[0057] The power-related corpus to be labeled consists of the text of regulations and systems for substation maintenance issued by the State Grid Corporation of China. This text includes: General Management Regulations and Accident Prevention Measures for Substation Acceptance, Operation and Maintenance, Testing, Evaluation and Repair of State Grid Corporation of China (hereinafter referred to as "Five Connections and One Measures"). This text contains a large number of power-related named entity nouns, which were converted, cleaned and deduplicated to obtain plain text data.

[0058] A labeling pool was formed by randomly selecting one-quarter of the longer data (approximately 9800 entries) from the State Grid's "Five Connections and One Measures" corpus. The entity knowledge base statistics, as shown in Table 1, were obtained by establishing a set of labeling standards for power-related named entities. Figure 2 The data shows the statistical distribution of named entity tags for the power sector, revealing an imbalance in the tag set. Notably, the number of entities with the tag "O" (indicating the character is not an entity word) is 322,331, significantly higher than other tag classes.

[0059] Table 1: Statistical Information of the Power Entity Database

[0060]

[0061]

[0062] This step requires preprocessing the electricity-related corpus before fine-tuning the BERT model. For example... Figure 1 As shown, the annotation process for the power corpus includes:

[0063] (1) Label a portion of the power corpus data (e.g., extract 1 / 4 of the power corpus data) to obtain a knowledge base of power entities. The knowledge base stores tuples (entity type and entity name);

[0064] (2) Use the knowledge base to perform non-human pseudo-annotation on other power corpora. Non-human pseudo-annotation refers to the machine labeling the unlabeled power corpora data according to the power entity knowledge base to obtain power entity recognition corpora, i.e. pseudo-annotated corpora.

[0065] Preprocessing of the power entity recognition corpus includes:

[0066] The power entity recognition corpus is segmented at the character level, and a predefined header tag [CLS] is added to the beginning of each sentence, and a predefined tail tag [SEP] is added to the end of each sentence. (The power entity recognition corpus consists of several individual sentences.)

[0067] The preprocessed power entity recognition corpus was divided into a training set and a validation set in a ratio of 8:2.

[0068] The pseudo-annotated corpus was constructed using the BMEO tagging method. Specifically, if a character unit is the beginning of an entity word, it is labeled as B-entity category; if a character unit is the end of an entity word, it is labeled as E-entity category; if a character unit is neither the beginning nor the end of an entity word, it is labeled as M-entity category; and if a character does not belong to an entity word, it is labeled as O. For example, the annotation result for the sentence "Inspect the reactor of the series insulator" is "to O, series B-1, series M-1, insulator M-1, insulation M-1, sub-E-1, of O, electricity B-1, reactor M-1, device E-1, proceed O, perform O, check O". This method can greatly reduce labor costs.

[0069] Step 4: Construct Huffman coding for power entity tags based on the number of power entity tags in the power entity recognition corpus;

[0070] In the aforementioned named entity tags for the power industry, Huffman tree encoding is used to represent all entity tag types. These entity tag types include 28 tags: "B-0" to "B-8", "M-0" to "M-8", "E-0" to "E-8", and "O". The numbers 0 to 8 correspond to the nine numbers in Table 1. A Huffman tree is constructed based on the number of these 28 entity tags in the power industry entity recognition corpus. These 28 entity tags are stored in the leaf nodes of the Huffman tree. The Huffman code for each tag is the path from the root node to that leaf node, where a path leading to the left subtree is defined as 0, and a path leading to the right subtree is defined as 1. For example... Figure 5The Huffman code of leaf node X is [1,1,1]. However, since different leaf nodes have different lengths, for leaf nodes whose path length is less than the longest path, their Huffman codes are padded with 0s to the longest path. For example, the longest path is 5. The Huffman code of the leaf node to which the label X belongs is [1,1,1,0,0].

[0071] Huffman coding of entity labels can alleviate the problem of class imbalance. This is because when the model predicts labels, the output is its Huffman-coded path, and label mapping is performed using forward maximum matching based on the Huffman-coded path. Due to the characteristics of Huffman coding, it always selects the two entity labels with the fewest occurrences to form a subtree. The 0s and 1s on the path at the root node of this subtree point to the two selected entities, so the distribution of 0s and 1s in the labels is usually relatively balanced. By predicting the Huffman-coded path, the imbalance problem in entity class label prediction can be effectively addressed.

[0072] Step 5: After obtaining the BERT language model through pre-training, add a classification layer to form the BERT power entity recognition model. Then, retrain the BERT power entity recognition model using power entity recognition corpus to obtain the BERT power entity recognition model.

[0073] After pre-training the BERT language model, a classification layer is added to form the BERT power entity recognition model. For example... Figure 4 As shown, when the input text passes through the BERT power entity recognition model, it first passes through the embedding layer to become a text vector sequence E[1] to E

[12] , then passes through a multi-layer encoder to obtain the output T[1] to T

[12] of the BERT language model, and finally passes through classification to obtain entity labels. The classification layer includes a fully connected layer and a Sigmoid activation function connected in sequence. The input of the classification layer is the output of the BERT language model, and the output of the classification layer is the Huffman code of the predicted power entity label. The corresponding power entity label is obtained by mapping through the Huffman code. The difference between the real label on the power entity recognition corpus and the output label of the BERT power entity recognition model is calculated by using cross-entropy loss, and the BERT power entity recognition model is trained by the AdamW optimizer. When the loss of the model on the power entity recognition corpus validation set no longer decreases, the model stops training and the model parameters are saved.

[0074] Step 6: Input the corpus to be recognized into the BERT power entity recognition model to obtain the Huffman code of the power entity label. The entity label is obtained through the Huffman code mapping, and then the entities of the 9 predefined categories in the corpus are obtained.

[0075] By inputting the text to be identified sentence by sentence into the BERT power entity recognition model, the Huffman code of each character entity label in the sentence is obtained. Then, through the mapping relationship between Huffman codes and entity labels, the Huffman code of each character entity label in the text to be identified is mapped to the original entity label. By judging whether the sequence of entity labels satisfies the template of entity type set by “BX”, “BX,EX”, “BX,MX,EX” and “BX,MX,MX,EX”, where X represents the entity category (0~9) and “MX” can be repeated multiple times to meet the requirements of different lengths of entities, the characters corresponding to the subsequences that satisfy the template are combined into entities as the final predicted entities.

[0076] Example 2:

[0077] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the aforementioned methods for identifying electrical entities based on multilayer neural networks.

[0078] A computing device, comprising,

[0079] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above in the multilayer neural network-based power entity identification method.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying power entities based on a multi-layer neural network, characterized in that: The power corpus to be identified is input into the pre-built BERT power entity recognition model to obtain the Huffman coding of the power entity label, and the entity label is obtained through Huffman coding mapping, and then the identified entity is obtained; The steps of constructing the BERT power entity recognition model include: Extract massive text corpora, perform data preprocessing on the massive text corpora, and obtain language model training corpora; Pre-train the BERT language model using the language model training corpus; Label the power entity labels on the power corpus data and construct the power entity recognition corpus; Construct the Huffman coding of power entity tags according to the number of power entity tags in the power entity recognition corpus; A classification layer is added to the pre-trained BERT language model to form a BERT power entity recognition model, and the BERT power entity recognition model is retrained using the power entity recognition corpus to obtain a trained BERT power entity recognition model; The classification layer includes a fully connected layer and a Sigmoid activation function connected in series, the input of the classification layer is the output of the BERT language model, and the output of the classification layer is the Huffman code of the predicted power entity label; Training the BERT power entity recognition model, including: The power entity recognition corpus is input into the BERT power entity recognition model, and the output is the Huffman code of the predicted power entity label. The corresponding power entity label is obtained through Huffman coding mapping; The cross entropy loss is used to calculate the difference between the true label on the power entity recognition corpus and the output label of the BERT power entity recognition model, and the BERT power entity recognition model is trained by the AdamW optimizer. When the loss of the model on the power entity recognition corpus validation set no longer decreases, the training is stopped and the model parameters are saved to obtain the trained BERT power entity recognition model.

2. The method for identifying electric entities based on a multi-layer neural network according to claim 1 is characterized in that: The data preprocessing process for the massive text corpus includes: The text is divided into sentences and sentence pairs are constructed. The sentence pairs are connected with the set connection tags. The set head tag is added to the head of the sentence pair and the set tail tag is added to the tail of the sentence pair. Among them, the sentence pairs composed of the connected sentences in the original text are positive samples, and the unconnected sentences are negative samples. The corpus for the task of predicting the relationship between the upper and lower sentences is constructed. In each sentence, some words are randomly masked for prediction; for the masked words, part of them are replaced with set string labels, part of them are replaced with random words, and the rest of them remain unchanged, forming the corpus for the word prediction task; The word labels are generated according to the real words in the masked positions, and the labels of the relationship between the upper and lower sentences are generated according to the relationship between the sentence pairs, so as to obtain the language model training corpus.

3. The method for identifying electric entities based on a multi-layer neural network according to claim 2 is characterized in that: The pre-training of the BERT language model by the language model training corpus comprises the following steps: The input of the BERT language model is preprocessed text, and the output is word labels and context relationship labels; Calculate the loss value of the output of the BERT language model and the true label, add the loss value of the word label and the loss value of the context relationship to get the final loss value, and use the AdamW optimizer to train the BERT language model based on the final loss value. When the loss value of the model on the validation set no longer decreases, stop training, save the model parameters, and get the BERT language model.

4. The method for identifying electric entities based on a multi-layer neural network according to claim 1 is characterized in that: The method of labeling the power corpus data with power entity labels and constructing the power entity recognition corpus includes: manually labeling part of the power corpus data to obtain a knowledge base of power entities; and using the knowledge base to label the remaining power corpus with non-artificial power entity labels to obtain the power entity recognition corpus.

5. The method for identifying electric entities based on a multi-layer neural network according to claim 4 is characterized in that: The non-artificial power entity labeling includes: The power entity recognition corpus is constructed using the BMEO tagging form. If a character unit is the beginning of an entity word, it is marked as a B-entity category; if a character unit is the end of an entity word, it is marked as an E-entity category; if a character unit is a non-starting or non-ending character of an entity word, it is marked as an M-entity category; if a character does not belong to an entity word, it is marked as an O.

6. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform the power entity identification method based on a multi-layer neural network according to any one of claims 1 to 5.

7. A computing device, characterized in that: include, One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the power entity identification method based on a multi-layer neural network according to any one of claims 1 to 5.

Images