Power equipment defect knowledge extraction system and method based on transfer learning
By introducing the BERT-BiGRU-Att-CRF and BERT-PCNN models based on transfer learning in the field of power equipment defects, combined with the attention mechanism, the problem of low efficiency of power equipment defect knowledge extraction is solved, and efficient and accurate knowledge extraction and Intelligent decision-making support reduces labor costs.
Patent Information
- Application Number
- CN202410854130.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-12
AI Technical Summary
In the field of power equipment defects, it is difficult for the existing technology to effectively utilize massive data, resulting in low knowledge extraction efficiency, and the inability to accurately extract high-quality knowledge. In addition, traditional operation and maintenance scheduling decision-making mechanisms are difficult to cope with the rapid analysis and fault handling of complex power grid equipment.
Adopting a power equipment defect knowledge extraction system based on transfer learning, combining the BERT-BiGRU-Att-CRF named entity recognition model and the BERT-PCNN relationship extraction module, introduce an attention mechanism, improve the construction efficiency and visualization of the knowledge graph, and through transfer learning Reduce the need for manual labeling of data and improve the accuracy of data preprocessing and knowledge extraction.
It significantly improves data preprocessing efficiency and knowledge extraction accuracy, reduces labor costs, promotes intelligent decision-making and management, and enhances the comprehensive grasp of the knowledge system of power equipment.
Smart Images

Figure CN118940832A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of power equipment failure, and in particular to a system and method for extracting power equipment defect knowledge based on transfer learning. Background Art
[0002] With the advent of the big data era, the amount of data in the power system continues to increase. Along with it comes the rapid development of the development and application of smart grids and artificial intelligence technology. However, the operation and maintenance of power equipment relies on the drive of massive data, among which the data source of the distribution network covers multiple management fields such as dispatching and operation and inspection, showing the characteristics of rich types and large volume. At present, when processing big data, we often face the problem of core information being submerged or unable to be effectively utilized, resulting in the inability to locate and extract high-quality knowledge, and unable to serve upper-level applications, and the utilization rate of high-value information is low. At the same time, the traditional operation and maintenance scheduling decision-making mechanism that relies on manual experience is difficult to cope with the rapid analysis and fault handling of complex power grid equipment. Therefore, how to improve the quality and utilization of data and enhance the comprehensive grasp of the knowledge system of power equipment has become an important topic in the current power system research.
[0003] By integrating and analyzing the power equipment defect data and building a knowledge graph in the field of power equipment defects, more comprehensive and accurate information can be provided to power industry workers, thereby promoting intelligent decision-making and management in the field of power equipment defects. Knowledge extraction, as the most basic part of knowledge graph construction, can distill the information in the power equipment defect text and improve the efficiency of knowledge storage and query. However, the parsing and extraction of power texts should not only focus on the literal meaning of the text, but also on the actual situation such as industry background, industry terminology and expression. Therefore, how to dig out deeper hidden knowledge from unstructured texts and improve the accuracy of extraction is an important way to improve the efficiency of knowledge extraction in power systems.
[0004] Knowledge extraction usually includes two subtasks: entity extraction and relationship extraction. Entity extraction is also called named entity recognition. Starting from named entity recognition methods based on dictionaries, rule matching, and machine learning, some scholars have begun to focus on deep learning models such as recurrent neural networks and bidirectional long short-term memory networks. Among them, the BiLSTM-CRF structure can effectively extract entity information from fault documents. The influence of word segmentation will still cause a certain amount of error accumulation in the structure. Adding a transformer model to this structure will improve the model effect due to the introduction of an attention mechanism. In terms of relationship extraction, combining relationship extraction with remote supervision and automatic annotation can reduce the burden of manual annotation.
[0005] Compared with the prior art, this application;
[0006] Technical comparison with patent CN 113657517A "A method for detecting power defects with a small number of samples based on attention mechanism and metric learning"
[0007] Patent CN 113657517A constructs training data sets and target data, and creates a model through meta-learning technology to solve the problem of insufficient power defect sample data in real scenarios. The present invention mainly uses transfer learning technology to solve the problem of less labeled data, and creates a joint model of named entity recognition and relationship extraction to improve the efficiency of building a knowledge graph of power equipment defects.
[0008] Patent CN 113657517A introduces an attention mechanism into the relational network of the meta-learning process, which makes the network more focused on finding valuable information in the image. It is a technology for improving defect image detection in the field of power equipment images. The present invention mainly starts from the power equipment defect text, and introduces the attention mechanism into the named entity recognition and relation extraction model parts respectively, so as to better obtain the power text context information and improve the accuracy of knowledge extraction.
[0009] Technical comparison with patent CN 117828105 A "A method for constructing a knowledge graph of power defect text based on joint extraction"
[0010] Patent CN 117828105 A first constructs a joint annotation system in the field of power defects, and annotates the text through a unified annotation set containing entity labels and relationship labels to identify all entities and entity pair relationships in the text. By constructing an entity relationship extraction model (BERT-FC-BiLSTM-CRF) that integrates BERT, a fully connected layer, a bidirectional long short-term memory network, and a conditional random field, the knowledge joint extraction of power defect text is realized. The knowledge extraction model in the present invention is a fusion of the BERT-BiGRU-Att-CRF named entity module and the BERT-PCNN relationship extraction module. The entity recognition and relationship extraction modules within the model have a clear division of labor and are trained and extracted in steps.
[0011] Patent CN 117828105 A introduces a fully connected layer into the BERT-BiLSTM-CRF model to improve model performance, capture the actual structural information in overlapping relationships, and verify the improved performance through model comparison. The present invention mainly introduces an attention mechanism into the knowledge extraction model to enable the model to better learn the contextual information in the power text, improve the model's learning ability and feature extraction ability for the power text, and realize the knowledge extraction of the power equipment defect text. Summary of the invention
[0012] The technical problem to be solved by the present invention is to provide a system and method for extracting knowledge of power equipment defects based on transfer learning. Although there are many advanced algorithms for entity extraction and relationship extraction, the specific application development in the field of power equipment defects is relatively slow, and there is still a lot of room for research on related knowledge graph construction technology. Therefore, the present invention proposes to introduce the BERT-BiGRU-Att-CRF named entity recognition model and the BERT-PCNN model into the knowledge extraction process of power equipment defect text, thereby improving the construction efficiency and visualization of the knowledge graph.
[0013] To achieve the above objectives, the power equipment defect knowledge extraction system based on transfer learning includes a data acquisition module, a knowledge extraction module and a knowledge graph construction module:
[0014] The data acquisition module introduces transfer learning into model design, uses the transfer model to preprocess the data during data preprocessing, and obtains labeled data suitable for knowledge extraction of power equipment defect texts;
[0015] Knowledge extraction module, in the field of power equipment defects, studies entity discovery and relationship extraction technology based on pre-trained language models, uses a combined model of BERT and bidirectional gated attention mechanism for entity extraction, and uses BERT-PCNN for relationship extraction based on remote supervision;
[0016] The knowledge graph construction module constructs a triple data set of the knowledge graph of power equipment defects, improves the construction efficiency and visualization of the knowledge graph, and enriches the systematic management methods in the field of power equipment defect operation and maintenance.
[0017] The data annotation method based on transfer learning in this paper proposes to use the BERT-BiLSTM-CRF model that has been trained in other fields, and use its mature language expression ability to migrate to the power equipment defect text annotation task required by this paper. A small amount of manually annotated data is used to train an annotation model that masters the rules of this data annotation task, and then the remaining data is annotated to reduce the labor consumption and labor cost in the process of obtaining annotated data and improve the data preprocessing speed.
[0018] As a further improvement of the system of the present invention, the knowledge extraction module includes the following steps:
[0019] S01. Entity extraction using BERT-BiGRU-Att-CRF;
[0020] The preprocessed power equipment defect text is input as the original text into the entity extraction model. After BERT encoding, BiGRU layer feature extraction and CRF layer label prediction, the entity information contained in the power text is obtained, thereby completing the entity recognition task. Among them, the role of the BiGRU-Attention layer is to realize the global feature extraction of the output vector of the BERT model. The BERT embedding vector T is used as the input of the BIGRU, and each character encoding vector in the set is updated in time series through the GRU unit. The one-way propagation formula of the GRU is:
[0021] r t =σ(W r x t +U r h t-1 ])
[0022] z t =σ(W z x t +U z h t-1 ])
[0023] where x t Input word vector for text sequence at time t, h t-1 represents the output state of the hidden layer at time t-1, W r ,U r To reset the gate, t 、h t-1 The weight matrix, W z ,U z For the update gate, t 、h t-1 The weight matrix, σ is the sigmoid activation function, and the reset gate activation result r is obtained. t And update the gate activation result z t Then calculate the information that needs to be remembered and forgotten in the previous time step and update the hidden state of the current time step;
[0024]
[0025]
[0026] in represents the candidate hidden state calculated by the tanh activation function, h t Represents the output of the gated recurrent unit at the current time step, which is the sum of the information in the previous time step that needs to be retained in the current hidden state and the information in the current time step;
[0027] The attention mechanism model is the output vector h of the BiGRU layer t Assign weight, c tIt represents the common output of the BiGRU-Attention comprehensive structure and is calculated as follows;
[0028]
[0029]
[0030] In the above formula, a t,i It refers to the attention function in the attention mechanism; the score function is an alignment model, which sets the specific weight value that the output should give to the input hidden state by judging the degree of matching between the input and output at a certain moment;
[0031] In order to improve the performance of obtaining long-distance text features in the BIGRU model, the present invention introduces an attention mechanism into the model to mine the degree of contextual relevance of different characters in a sentence, thereby better solving the long-range dependency problem caused by excessive word length. The attention mechanism can optimize the extraction results of local features by increasing the feature weights of semantics related to entities.
[0032] S02, use BERT-PCNN for relationship extraction;
[0033] BERT is used as the model embedding layer to convert text data into corresponding numerical vectors. The PCNN layer is used to extract text features and predict relationships. The BERT-PCNN relationship extraction model with attention mechanism is hierarchically divided into embedding layer, self-attention layer, convolution layer, segmented maximum pooling layer and classification layer. After the BERT embedding layer, the entity vector information contained in the text vector is integrated into the text vector through vector weighting, and the association relationship between the characters in the text is learned through the attention mechanism, so that the vector is rich in relevant entity information before entering the PCNN feature extraction.
[0034] In the attention mechanism, for the input sequence X = {x 1 ,x 2 ,...,x n}, where x i Represents the vector of the i-th position, and defines the weight matrix W Q ,W K ,W V , through learning and updating, the Dot-product method can be used to calculate any two vectors x i and x j The degree of correlation between The calculation formula is as follows;
[0035]
[0036] Use the softmax activation function to normalize the degree of association between vectors to obtain the degree of association The calculation formula is as follows;
[0037]
[0038] The input vector x i With the weight matrix W V Multiply them to get V i , when using V i With every one of you Multiply and sum to get the output vector Y = {y 1 ,y 2 ,...,y n}, where y i Contains the input vector X and each vector x i The correlation relationship between them can be calculated by the following formula:
[0039]
[0040] The convolutional layer in the model converts the output vector sequence Y = {y 1 ,y 2 ,...,y n}A piecewise convolution operation is performed in conjunction with the weight vector w, where the weight matrix w is considered as a convolution filter.
[0041] The present invention uses the maximum pooling layer to filter the features obtained by the convolution layer and output the local optimal features to reduce the problem of excessively high data dimensions after the convolution operation. Then, the output feature vector of the maximum pooling layer is processed in the classification layer, and the most likely relationship is predicted by the softmax classifier.
[0042] As a further improvement of the system of the present invention, the knowledge graph construction module includes the integration of entity relationship triples and knowledge graph construction, including the following steps:
[0043] T01, knowledge integration;
[0044] T02, integrate knowledge triples into csv files;
[0045] T03. Import the csv file into the neo4j system;
[0046] T04, visualization of graphs.
[0047] The present invention provides a method for extracting knowledge of power equipment defects based on transfer learning, and the specific steps are as follows:
[0048] 1) Obtaining annotated data suitable for knowledge extraction of power equipment defect texts;
[0049] 2) Use the labeled data to train the entity relationship extraction model, and use the optimal model to extract power knowledge triples from the power equipment defect text;
[0050] 3) Based on the extracted knowledge triples, data purification and correction are performed through knowledge fusion to construct a knowledge graph of power equipment defects.
[0051] As a further improvement of the method of the present invention, step S3) integrates the knowledge triples into a csv file and imports it into the neo4j system using the cypher language to construct a knowledge graph in the field of power equipment defects.
[0052] The beneficial effects of the present invention are mainly reflected in the following aspects:
[0053] 1) Improve data preprocessing efficiency:
[0054] By introducing transfer learning technology, the present invention can use model parameters trained in other fields to quickly adapt to the data annotation task of power equipment defect text. This method significantly reduces the need for manual data annotation, reduces annotation costs, and improves the speed and efficiency of data preprocessing.
[0055] 2) Improve the accuracy of knowledge extraction:
[0056] The BERT-BiGRU-Att-CRF model and the BERT-PCNN model are used for entity extraction and relationship extraction. Combining the pre-training capability of BERT and the characteristics of the deep learning model, the present invention can more accurately extract key information and relationships from the power equipment defect text, thereby improving the accuracy of knowledge extraction.
[0057] 3) Enriching knowledge graph construction:
[0058] Based on the acquired power equipment knowledge graph triple data set, the present invention can construct a structured power equipment defect knowledge graph to provide more comprehensive and accurate information for power industry workers. This helps to improve the overall grasp of the power equipment knowledge system and provide support for intelligent decision-making and management.
[0059] 4) Promote intelligent decision-making and management:
[0060] By constructing a knowledge graph of power equipment defects, the present invention can establish power equipment data association analysis, capture the association relationship between data, and visualize the data. This helps operation and maintenance personnel to more intuitively understand and analyze the defects of power equipment and promote the realization of intelligent decision-making and management.
[0061] 5) Reduce labor costs:
[0062] Through the application of transfer learning and automatic labeling technology, the present invention reduces the need for manual labeling of data, reduces the labor cost and time cost of data labeling, and improves work efficiency.
[0063] In summary, the present invention has achieved remarkable beneficial effects in the field of power equipment defect knowledge extraction by combining transfer learning and deep learning technology. It not only improves the efficiency and accuracy of data preprocessing and knowledge extraction, but also promotes the realization of intelligent decision-making and management, and reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 It is the entity relationship extraction framework diagram in the present invention;
[0065] Figure 2 It is an example of data preprocessing in the present invention;
[0066] Figure 3 This is an example of the electric power equipment defect text knowledge graph constructed in the present invention. DETAILED DESCRIPTION
[0067] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0068] As a specific embodiment of the present invention, the present invention provides a method for extracting knowledge of power equipment defects based on transfer learning. The entity relationship extraction framework diagram of the present invention is as follows: Figure 1 Shown
[0069] In the data preprocessing module, the main data sources are the unstructured text of the power transmission and transformation defect terminology specifications issued by the State Grid Corporation of China and the fault operation and maintenance records of a certain power company. The specific preprocessing steps are as follows:
[0070] A01) Data preprocessing. First, the power equipment defect text data is traversed and cleaned to reduce the computational complexity and make it reach the standard format, and then enter the next step of determining data attributes. While ensuring that the data is non-redundant, based on the power transmission and transformation defect terminology specification, the power company's equipment failure operation and maintenance records are converted into data, and the rule data of failure and disposal are screened out. After deduplication, a total of 15,916 data are obtained.
[0071] A02) Determine entities and labels. In the identification of power equipment fault entities, through the analysis of equipment operation and inspection data and actual application requirements, the key information that needs to be extracted from the text is determined: six types of entities: primary equipment, equipment category, fault component, fault phenomenon, fault cause, and fault degree. The interpretation of each entity is shown in Table 1.
[0072] Table 1 Entity category labeling standard for entity relationship extraction of power equipment fault
[0073]
[0074] According to the BIO annotation system and combined with the actual annotation situation, 13 categories of annotation labels are determined. The annotation definitions of each category of entities are shown in Table 2. Each label directly represents the meaning of the entity itself.
[0075] Table 2 Entity annotation definitions
[0076]
[0077] Among them, B is used as the beginning of the entity, I is used as the continuation of the entity content, and O represents a word that is not of interest. The training set and the test set are marked accordingly. The annotation examples are shown in Table 3.
[0078] Table 3 Example of text annotation of power equipment defects
[0079]
[0080] A03) Use the migration model to annotate data. This paper migrates to the power equipment defect text entity annotation task based on the BERT-BiLSTM-CRF algorithm. The BERT-BiLSTM-CRF entity annotation model mainly consists of 3 parts: BERT word vector pre-training language model, BiLSTM semantic extraction layer and CRF decoding layer. Some annotation results are shown in Figure 2. Figure 2 shown.
[0081] In the entity relationship extraction module, the specific implementation steps of entity extraction are as follows:
[0082] B01) Model training and verification. The preprocessed text is vectorized to make it structured data that can be recognized by computers; then the features that can represent the semantics of the characters are selected; finally, the selected vectorized features are input into the model for training. The model is trained for 35 rounds, and the optimal model is achieved at epoch=35, with a loss value of 0.02.
[0083] B02) Model prediction. Input the preprocessed test data into the trained classifier for evaluation. If the evaluation result is not ideal, the model parameters should be adjusted and iterated continuously until the relatively optimal prediction result is output. Finally, the trained entity extraction model is used to perform entity recognition and extraction on the prediction set data. The prediction result has an accuracy rate of 94%, a recall rate of 93%, and an F1 value of 93%. The overall effect is good.
[0084] When extracting relations, based on the existing text data of power equipment defects and the obtained entity data set, the Bert-PCNN model was trained using a remote supervision method for 30 rounds. In the 17th round of training, the accuracy of the model was stable at more than 90%, indicating that the model can learn knowledge from the training data well and has good stability. Finally, the optimal model was used to predict relations for the remaining data, with a prediction accuracy of 93.55%.
[0085] The knowledge graph construction module imports the power text into the entity relationship extraction model of the present invention to obtain entity relationship triple data suitable for the knowledge graph of power equipment defects, and then uses the cypher language to import the data into the neo4j system. The constructed knowledge graph example is shown in the figure below Figure 3 shown.
[0086] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.
Claims
1. A power equipment defect knowledge extraction system based on transfer learning, including a data acquisition module, a knowledge extraction module and a knowledge graph construction module, characterized in that: The data acquisition module introduces transfer learning into model design, uses the transfer model to preprocess the data during data preprocessing, and obtains labeled data suitable for knowledge extraction of power equipment defect texts; Knowledge extraction module, in the field of power equipment defects, studies entity discovery and relationship extraction technology based on pre-trained language models, uses a combined model of BERT and bidirectional gated attention mechanism for entity extraction, and uses BERT-PCNN for relationship extraction based on remote supervision; The knowledge graph construction module constructs a triple data set of the knowledge graph of power equipment defects, improves the construction efficiency and visualization of the knowledge graph, and enriches the systematic management methods in the field of power equipment defect operation and maintenance.
2. According to claim 1, the power equipment defect knowledge extraction system based on transfer learning is characterized in that: The knowledge extraction module comprises the following steps: S01. Entity extraction using BERT-BiGRU-Att-CRF; The preprocessed power equipment defect text is input as the original text into the entity extraction model. After BERT encoding, BiGRU layer feature extraction and CRF layer label prediction, the entity information contained in the power text is obtained, thereby completing the entity recognition task. Among them, the role of the BiGRU-Attention layer is to realize the global feature extraction of the output vector of the BERT model. The BERT embedding vector T is used as the input of the BIGRU, and each character encoding vector in the set is updated in time series through the GRU unit. The one-way propagation formula of the GRU is: r t =σ(W r x t +U r h t-1 ]) z t =σ(W z x t +U z h t-1 ]) where x t Input word vector for text sequence at time t, h t-1 represents the output state of the hidden layer at time t-1, W r ,U r To reset the gate, t 、h t-1 The weight matrix, W z ,U z For the update gate, t 、h t-1 The weight matrix, σ is the sigmoid activation function, and the reset gate activation result r is obtained. t And update the gate activation result z t Then calculate the information that needs to be remembered and forgotten in the previous time step and update the hidden state of the current time step; in represents the candidate hidden state calculated by the tanh activation function, h t Represents the output of the gated recurrent unit at the current time step, which is the sum of the information in the previous time step that needs to be retained in the current hidden state and the information in the current time step; The attention mechanism model is the output vector h of the BiGRU layer t Assign weight, c t It represents the common output of the BiGRU-Attention comprehensive structure and is calculated as follows; In the above formula, a t,i It refers to the attention function in the attention mechanism; the score function is an alignment model, which sets the specific weight value that the output should give to the input hidden state by judging the degree of matching between the input and output at a certain moment; S02, use BERT-PCNN for relationship extraction; BERT is used as the model embedding layer to convert text data into corresponding numerical vectors. The PCNN layer is used to extract text features and predict relationships. The BERT-PCNN relationship extraction model with attention mechanism is hierarchically divided into embedding layer, self-attention layer, convolution layer, segmented maximum pooling layer and classification layer. After the BERT embedding layer, the entity vector information contained in the text vector is integrated into the text vector through vector weighting, and the association relationship between the characters in the text is learned through the attention mechanism, so that the vector is rich in relevant entity information before entering the PCNN feature extraction. In the attention mechanism, for the input sequence X = {x1, x2, ..., x n }, where x i Represents the vector of the i-th position, and defines the weight matrix W Q ,W K ,W V , through learning and updating, the Dot-product method can be used to calculate any two vectors x i and x j The degree of correlation between The calculation formula is as follows; Use the softmax activation function to normalize the degree of association between vectors to obtain the degree of association The calculation formula is as follows; The input vector x i With the weight matrix W V Multiply them to get V i , when using V i With every one of you Multiply and sum to get the output vector Y = {y1, y2, ..., y n }, where y i Contains the input vector X and each vector x i The correlation relationship between them can be calculated by the following formula: The convolutional layer in the model converts the output vector sequence Y = {y1, y2, ..., y n A piecewise convolution operation is performed in conjunction with the weight vector w, where the weight matrix w is considered as a convolution filter.
3. The power equipment defect knowledge extraction system based on transfer learning according to claim 1 is characterized in that: The knowledge graph construction module includes the integration of entity relationship triples and knowledge graph construction, including the following steps: T01, knowledge integration; T02, integrate knowledge triples into csv files; T03. Import the csv file into the neo4j system; T04, visualization of graphs.
4. The method of the power equipment defect knowledge extraction system based on transfer learning according to any one of claims 1 to 3, characterized in that: The specific steps are as follows: 1) Obtaining annotated data suitable for knowledge extraction of power equipment defect texts; 2) Use the labeled data to train the entity relationship extraction model, and use the optimal model to extract power knowledge triples from the power equipment defect text; 3) Based on the extracted knowledge triples, data purification and correction are performed through knowledge fusion to construct a knowledge graph of power equipment defects.
5. The method of the power equipment defect knowledge extraction system based on transfer learning according to claim 4 is characterized in that: Step S3) Integrate the knowledge triples into a csv file and import it into the neo4j system using the cypher language to construct a knowledge graph in the field of power equipment defects.
Citation Information
Patent Citations
BERT-BiGRU-IDCNN-CRF named entity identification method based on attention mechanism
CN112733541A
Chinese triple extraction method based on BERT model
CN113901820A
Knowledge graph construction method based on food safety field
CN117634610A
Electric power defect text knowledge graph construction method based on joint extraction
CN117828105A
Cited By
Nuclear power unit quality defect source analysis and total value chain tracing method
CN120258632A
Enterprise-level knowledge center system construction method and device based on large language model, computer equipment, storage medium and computer program product
CN121144627A
BERT model and linear attention-based intention recognition method and system
CN121256466A
Data migration method, system and equipment for historical data of oiling laboratory and medium
CN121387859A
A data migration method, system, device and medium for oiling laboratory historical data
CN121387859B
