A method for optimizing fault diagnosis of power main equipment based on ChatGPT fine-tuning pre-training model

Abstract

The present invention relates to a method for optimizing the diagnosis of defects in main power equipment by fine-tuning a pre-trained model based on ChatGPT. A corpus containing 80% correct sentences and 20% grammatically incorrect sentences generated by the ChatGPT API is constructed. Domain-specific long texts are converted into the CoLA dataset format. Adapter BERT is used as the infrastructure for fine-tuning. The base model uses bert-Chinese-base. After meeting the accuracy threshold, all layer parameters are frozen and the model is used as a feature extractor. The final vector representation is obtained by summarizing the output of the last hidden layer. A KGC model with excellent performance is selected and only its input layer is modified. A fully connected layer is introduced between the output layer of the pre-trained language model and the KGC model. The fully connected layer is trained and its parameters are updated during the KGC task iteration process. Finally, in the KGC model, the probabilities of candidate entities are calculated and ranked, and the top k entities are selected as the final answer. Experimental results show that this method effectively improves the performance of KGC in diagnosing defects in main power equipment.

Description

Technical Field

[0001] The present invention relates to the field of power cognitive intelligence technology, and in particular to a method for optimizing power main equipment defect diagnosis based on a ChatGPT fine-tuning pre-training model. Background Art

[0002] Defect diagnosis plays a critical role in ensuring stable power system operation. Defects in primary power equipment can have serious consequences, such as power outages, equipment damage, and significant impacts on daily life and the economy. Therefore, rapid and accurate detection of these defects is crucial to maintaining system reliability and mitigating negative impacts on public life and the economy.

[0003] As an innovative knowledge engineering technology, knowledge graphs provide a structured approach for describing defect events, attributes, and their relationships. This enables a more comprehensive representation of complex defect relationships. In the context of power main equipment defect diagnosis, our goal is to predict defective components based on given defect phenomena and their relationships. Furthermore, the model can take entities and given relationships in a specific knowledge graph as input and output the tail entity under that relationship. To this end, we propose a method for extracting triples from tabular defect data, constructing a knowledge graph, and employing a knowledge graph completion algorithm for defect diagnosis.

[0004] In the field of power equipment, we observed that fine-tuning a pre-trained model twice yielded superior results. We fine-tuned a language model for a specific task, choosing sentence classification. Specifically, we constructed the CoLA dataset, consisting of 80% original text labeled as grammatically correct sentences, and the remaining 20% ​​generated by ChatGPT and labeled as incorrect. This approach explored how to generate a binary classification dataset from a base corpus and fine-tune a language model to better capture expert knowledge.

[0005] To conserve computational resources during fine-tuning and enable CPU-based execution, we chose to use the AdapterBERT model. This approach significantly reduces computational costs. In this approach, we designed a fully connected layer that is added after the BERT output and connected to the KGC model to reduce the input dimensionality. This further reduces the computational cost of the KGC algorithm. Our goal is to reduce the application's computational requirements, making it easier to deploy on commodity hardware. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings of the prior art by providing a method for optimizing the diagnosis of defects in primary power equipment based on a fine-tuned pre-trained ChatGPT model. This method accurately predicts the phenomena and relationships of primary power equipment defects, improving system reliability and mitigating negative impacts on people's lives and the economy. By leveraging secondary fine-tuning pre-training in specialized fields and combining it with the Adapter BERT model to reduce computational costs, this method achieves efficient and accurate recognition and processing capabilities even in resource-limited environments.

[0007] To achieve the above objectives, the technical solution of the present invention is: a method for optimizing the fault diagnosis of main power equipment based on the ChatGPT fine-tuning pre-training model, comprising the following steps:

[0008] (1) Generate a fine-tuning corpus based on ChatGPT. 80% of the correct sentences will be directly extracted from the original power corpus and labeled correctly; the remaining 20% ​​are grammatically incorrect sentences generated through the ChatGPT API.

[0009] (2) Convert the corpus in step (1) into the CoLA dataset format;

[0010] (3) Adopting the Adapter BERT model architecture, using bert-Chinese-base as the basic pre-training model, and fine-tuning using the CoLA dataset;

[0011] (4) After meeting the fine-tuning accuracy threshold, freeze all layer parameters and use the pre-trained model as a feature extractor;

[0012] (5) Input the knowledge graph nodes and edges into the pre-trained model, and pool the output of the last hidden layer to obtain the final vector representation;

[0013] (6) Using the trained knowledge graph completion model, namely the KGC model, as the basis, the knowledge graph completion method is used to diagnose the defects of the main power equipment, and only the input layer of the KGC model is modified;

[0014] (7) A fully connected layer is introduced between the output layer of the pre-trained model and the KGC model; the fully connected layer updates its parameters during the KGC model training process;

[0015] (8) In the KGC model, the answer selection process will calculate the probabilities of all candidate entities, rank them and select the top k entities as the final answers, and analyze the final results to test the performance of the model.

[0016] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention effectively improves the performance of KGC in diagnosing defects of main power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the method of the present invention.

[0018] Figure 2 This is a model architecture diagram of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, the present invention provides a method for optimizing the fault diagnosis of power main equipment based on ChatGPT fine-tuning pre-training model, and its model diagram is as follows Figure 2 The present invention comprises the following steps:

[0021] Step 1: Processing the defect dataset is a prerequisite. The data comes from the State Grid Fujian Electric Power Research Institute and primarily contains defect information for primary power equipment. We manually collected the data, structured it, and converted it into triples. The dataset contains 58,820 triples, 53,889 entities, and 12 different relations.

[0022] Step 2: Pre-training dataset processing. We used a 4.96 MB power-specific corpus for fine-tuning, including a substation operation and maintenance question bank, substation primary equipment defect classification standards, and distribution network fault analysis cases. The corpus was divided into training, validation, and test sets in a ratio of 8:1:1.

[0023] Step 3: Corpus generation based on ChatGPT. Domain-specific long text data is converted to the CoLA dataset format, and BERT is then fine-tuned. Specifically, 80% of sentences are marked as correct and extracted directly from the original corpus. For the remaining 20%, ChatGPT generates grammatically incorrect sentences. The key technology lies in prompt design, which includes four parts: task definition, specific task description and examples, additional requirements, and corpus (i.e., original content generation). It significantly impacts the quality of the data returned by ChatGPT.

[0024] Step 4: Fine-tune the pre-trained model. To save computational resources during fine-tuning and run on a CPU, we chose to use Adapter BERT [1], which adds an adapter layer to BER without directly fine-tuning parameters. This approach significantly reduces computational cost. We selected the bert-Chinese-base pre-trained model, set the learning rate to 3e-4, and the train_batch_size to 32. The sentence length was set to 128, and sentences exceeding this limit were truncated and shorter sentences were padded. Fine-tuning the pre-trained language model was performed on a CPU and took about 30 minutes.

[0025] Step 5. Model training. DistMult, KG2E, and NodePiece were selected as knowledge graph completion models and implemented using the pykeen library. A maximum of 300 training rounds were set, with evaluations performed every 10 rounds. If the model does not improve after 5 consecutive evaluations, training will stop early. To further reduce the computational cost of the KGC algorithm, we designed a fully connected layer to add after the BERT output, and then connected it to the KGC model to reduce the input dimension. Doing so can reduce the computational requirements of the application and make it easier to deploy on hardware with general performance. A fully connected layer is used to connect the output of BERT with the output of the KGC model, with an input dimension of 768 and an output dimension of 300. The KGC model is trained on a GPU using four Tesla A100 graphics cards, with each round taking approximately 1.5 seconds.

[0026] Step 6: Model Testing. Two test experiments were conducted. The first experiment directly used the knowledge graph completion model. The second experiment fine-tuned the pre-trained model using the CoLA dataset generated based on the power system long text corpus. The fine-tuned pre-trained model was then used as a feature extractor to extract the initial embedding for the knowledge graph completion task. This second method is the one proposed in this paper.

[0027] Step 7: Model Evaluation. We used commonly used knowledge graph completion metrics, such as accuracy (Hits@k) and mean rank, to compare the proposed method with knowledge graph completion models such as DistMult, KG2E, and NodePiece. Experimental results on the power system defect knowledge graph completion task showed that the proposed method outperformed the compared methods.

[0028] Step 8: Practical Application. We applied the proposed method to a real-world power system defect diagnosis scenario. Using knowledge graph completion technology, we provided more accurate defect predictions for power equipment. Power operation and maintenance personnel can develop targeted maintenance plans based on the defect prediction results, improving equipment reliability and safety.

[0029] References:

[0030] [1]HoulsbyN, Giurgiu A, JastrzebskiS, et al. Parameter-EfficientTransfer Learning for NLP[J]. 2019..

[0031] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for optimizing power main equipment defect diagnosis based on ChatGPT fine-tuning pre-training model, characterized in that: The steps include: (1) Generate a fine-tuning corpus based on ChatGPT. 80% of the correct sentences will be directly extracted from the original power corpus and labeled correctly; the remaining 20% ​​are grammatically incorrect sentences generated through the ChatGPT API. When generating a fine-tuning corpus based on ChatGPT, grammatically incorrect sentences are generated by designing prompts including task definition, specific task description and examples, additional requirements, and corpus. (2) Convert the corpus in step (1) into the CoLA dataset format; (3) Adopting the Adapter BERT model architecture, using bert-Chinese-base as the basic pre-training model, and fine-tuning using the CoLA dataset; (4) After meeting the fine-tuning accuracy threshold, freeze all layer parameters and use the pre-trained model as a feature extractor; (5) Input the knowledge graph nodes and edges into the pre-trained model, and pool the output of the last hidden layer to obtain the final vector representation; (6) Introduce a fully connected layer between the output layer of the pre-trained model and the KGC model to train the KGC model and the fully connected layer; The fully connected layer updates its parameters during the KGC model training process; Based on the implementation of the trained KGC model, the knowledge graph is completed to diagnose the defects of the main power equipment; (7) In the KGC model, the answer selection process will calculate the probabilities of all candidate entities, rank them and select the top k entities as the final answers, and analyze the final results to test the performance of the model.

Images