Intelligent question and answer method for power equipment knowledge base

Abstract

The invention discloses a power equipment knowledge base intelligent question and answer method, which comprises the following steps: acquiring a pdf file from a power system operation and maintenance team, reading PDF file content by using a text preprocessing library space, and training through a banito model to obtain a training data set; based on an open-source large model Llmal3, performing model fine tuning on the training data set by adopting an LORA method to optimize the model; the method comprises the following steps: performing text extraction and statement segmentation on a PDF file, encoding Chinese statements by using a Pinecone vector search mode, storing the Chinese statements as a Pinecone vector library, performing Pinecone encoding, and establishing a knowledge base; when the equipment in the transformer substation breaks down, the staff can easily analyze the equipment state, determine the fault position and process the equipment fault in a short time without looking up related equipment data, so that huge economic loss and severe social influence cannot be caused to companies.

Description

Technical Field

[0001] The present invention relates to the fields of big data, artificial intelligence big models, and electric power operation and maintenance, and specifically to an intelligent question-answering method for an electric power equipment knowledge base. Background Art

[0002] With the rapid development of society, all walks of life are increasingly dependent on electricity, and users' requirements for the safe operation of power grids are constantly increasing. Work safety has become a key element to ensure the sustainable development of power grid enterprises. In recent years, the scale of power grids has expanded rapidly. However, due to issues such as capital budget and equipment cost, there is a common situation of mixing old and new equipment in power grids. In addition, the technologies used by equipment produced in different eras are not the same, and the maintenance methods vary greatly, making it difficult to build a unified management standard. This greatly increases the difficulty for power grid employees to manage and maintain equipment. Even old employees with many years of field experience cannot accurately grasp the information of each device. When equipment in a substation fails, it is difficult for staff to analyze the equipment status, determine the fault location, and handle the equipment failure in a short time without consulting relevant equipment information. This is likely to lead to the deterioration of the accident and the expansion of the scope of influence, thereby causing huge economic losses and adverse social impacts on the company. The establishment of an intelligent question-answering system for the knowledge base of power equipment through a large language model provides an excellent solution to the above problems. The Pinecone method used in the present invention is an open source library in Python, which can realize the function of similar vector search. The PDF file comes from the substation equipment archives pdf, safety management regulations pdf, and operation and maintenance behavior guidelines pdf stored by the power system operation and maintenance team. The substation equipment archives PDF mainly include: Gezhenbao main transformer manual, Gezhenbao 66kV equipment archives, safety management regulations PDF mainly include: State Grid Corporation Power Safety Work Regulations-Substation Part, and Operation and Maintenance Behavior Guidelines pdf mainly include: Gezhenbao Substation On-site Operation Special Regulations. The test question bank comes from the power grid system employee test training materials. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that when equipment in a substation fails, it is difficult for staff to analyze the equipment status, determine the fault location, and handle the equipment failure in a short period of time without consulting relevant equipment information. This is likely to lead to the deterioration of the accident and the expansion of the impact range, thereby causing huge economic losses and adverse social impacts to the company.

[0004] In order to solve the above problems, the present invention proposes an intelligent question-answering method for a power equipment knowledge base, comprising the following steps:

[0005] S1: Create a training dataset;

[0006] S1-1: Use the text preprocessing library spacy to read the PDF file content and segment it according to semantics to obtain the sentence content;

[0007] S1-2: Prepare the bonito model, input the training task, and use the extraction question-answering mode. The longest token of the model is set to 256;

[0008] S1-3: Input the sentence content into the bonito model to generate a preliminary redundant data set, manually filter out the valid data and filter out the dirty data, which is the data in which the generated question-answering answers are wrong or have no answers;

[0009] The question and answer data collected manually from the test question bank is added to the Bonito model to automatically generate valid data. The set of valid data is the training data set.

[0010] S2: Model fine-tuning;

[0011] Based on the open source large model Llmal3, the LORA method is used to fine-tune the model for the training data set obtained in step S1, including the following steps:

[0012] S2-1: Given a pre-trained autoregressive language model P parameterized by Φ Φ (y|x), P Φ (y|x) can be a general multi-task learner, which can be a GPT model based on the Transformer architecture; each downstream task is represented by a training dataset of context-target pairs as:

[0013] Z={(x i ,y i )} i＝1,...,N

[0014] where x i and i is a tag sequence. During full fine-tuning, the model pre-training weights are initialized to Φ0 and updated to Φ0+ΔΦ through continuous gradient following to maximize the conditional language modeling objective:

[0015]

[0016] S2-2: If the pre-trained autoregressive language model is large, training the model through full fine-tuning requires a lot of computing power. A more efficient method is used to optimize the parameters, including the following steps:

[0017] The task-specific parameter increment ΔΦ = ΔΦ(Θ) is further encoded by a smaller parameter set Θ, where |Θ| = |Φ0|, thus transforming the task of finding ΔΦ into optimizing the parameter set Θ:

[0018]

[0019] S2-3: For the pre-trained weight matrix W0∈R d×r , by decomposing the iterative matrix, namely:

[0020] W0+ΔW=W0+BA

[0021] To suppress the update speed of the model, where B∈R d×r ,A∈R r×k , and the rank of the matrix is ​​r = min(d,k);

[0022] During the training process, the pre-trained weight matrix W0 is not updated with the gradient, and only the parameters in the process matrices A and B are updated. When h=W0x, the forward transfer expression after optimization of the present invention is:

[0023] h=W0x+ΔWx=W0x+BAx;

[0024] S3: Knowledge base building and question answering;

[0025] The original PDF file is extracted and sentence segmented using the method of step S1, and the Chinese sentences are encoded using the Pinecone vector search method and stored as a Pinecone vector library. When a question is raised, the program also Pinecone encodes the question, compares the similarity with the code in the vector library, finds the vector with the highest similarity, and completes the establishment of the knowledge base.

[0026] Beneficial purpose of the present invention: The present invention innovatively develops an end-to-end large-model dialogue robot dedicated to the power system, uses advanced algorithms such as Bonito, Llama, Lora, etc., and realizes a computer program that integrates data set establishment, knowledge base construction, model training, and question-answering dialogue. It is the first time that the knowledge base and large-model fine-tuning are organically combined to improve the accuracy of the question-answering robot. It allows power workers to provide professional fault solutions without having to fully grasp the complicated equipment information with a wide time span and uneven technology. They only need to understand the general information of the equipment to provide professional fault solutions, which can effectively reduce the work pressure of front-line workers and improve the efficiency of daily operation and maintenance of equipment and abnormal fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the main interface of the conversation robot of the present invention;

[0028] Figure 2 It is a schematic diagram of the model training interface of the present invention;

[0029] Figure 3 It is a schematic diagram of the algorithm flow of the present invention;

[0030] Figure 4 This is a schematic diagram of the present invention using the Bonito model to automatically generate a training data set;

[0031] Figure 5 It is a schematic diagram of the model fine-tuning process of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below with reference to the accompanying drawings;

[0033] An intelligent question-answering method for a power equipment knowledge base comprises the following steps:

[0034] S1: Create a training dataset;

[0035] S1-1: Use the text preprocessing library spacy to read the PDF file content and segment it according to semantics to obtain the sentence content;

[0036] S1-2: Prepare the bonito model, input the training task, and use the extraction question-answering mode. The longest token of the model is set to 256;

[0037] S1-3: Input the sentence content into the bonito model to generate a preliminary redundant data set, manually filter out the valid data and filter out the dirty data, which is the data in which the generated question-answering answers are wrong or have no answers;

[0038] The question and answer data collected manually from the test question bank is added to the Bonito model to automatically generate valid data. The set of valid data is the training data set.

[0039] S2: Model fine-tuning;

[0040] Based on the open source large model Llmal3, the LORA method is used to fine-tune the model for the training data set obtained in step S1, including the following steps:

[0041] S2-1: Given a pre-trained autoregressive language model P parameterized by Φ Φ (y|x), P Φ (y|x) can be a general multi-task learner, which can be a GPT model based on the Transformer architecture; each downstream task is represented by a training dataset of context-target pairs as:

[0042] Z={(x i ,y i )} i＝1,...,N

[0043] where x i and iis a tag sequence. During full fine-tuning, the model pre-training weights are initialized to Φ0 and updated to Φ0+ΔΦ through continuous gradient following to maximize the conditional language modeling objective:

[0044]

[0045] S2-2: If the pre-trained autoregressive language model is large, training the model through full fine-tuning requires a lot of computing power. A more efficient method is used to optimize the parameters, including the following steps:

[0046] The task-specific parameter increment ΔΦ = ΔΦ(Θ) is further encoded by a smaller parameter set Θ, where |Θ| = |Φ0|, thus transforming the task of finding ΔΦ into optimizing the parameter set Θ:

[0047]

[0048] S2-3: The neural network in the open source large model Llmal3 contains dense layers that perform matrix multiplication, and the weight matrix in the dense layer has full rank; when adapting to specific tasks, the pre-trained autoregressive language model has a lower intrinsic dimension and can still be effectively learned even if randomly projected to a smaller subspace. The update of the weights in the training model proposed in this invention has a lower intrinsic rank during the adaptation process;

[0049] For the pre-trained weight matrix W0∈R d×r , by decomposing the iterative matrix, namely:

[0050] W0+ΔW=W0+BA

[0051] To suppress the update speed of the model, where B∈R d×r ,A∈R r×k , and the rank of the matrix is ​​r = min(d,k);

[0052] During the training process, the pre-trained weight matrix W0 is not updated with the gradient, and only the parameters in the process matrices A and B are updated. When h=W0x, the forward transfer expression after optimization of the present invention is:

[0053] h=W0x+ΔWx=W0x+BAx;

[0054] S3: Knowledge base building and question answering;

[0055] After fine-tuning the model using the LORA method, it already has a certain ability to answer questions and can initially solve most operation and maintenance problems. However, in order to further improve the accuracy of the model's answers, the concept of knowledge base is introduced based on model fine-tuning to further enhance the accuracy and speed of the model's answering of questions.

[0056] The original PDF file is extracted and sentence segmented using the method of step S1, and the Chinese sentences are encoded using the Pinecone vector search method and stored as a Pinecone vector library. When a question is raised, the program also Pinecone encodes the question, compares the similarity with the code in the vector library, finds the vector with the highest similarity, and completes the establishment of the knowledge base.

[0057] These vectors with high similarity can be considered as the knowledge most relevant to answering the question. The large model can use this knowledge to correct its own answers and give more accurate answers.

[0058] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent question-answering method for a power equipment knowledge base, characterized in that: The following steps are involved: S1: Establish training data set; S1-1: Use the text preprocessing library spacy to read the PDF file content and segment it according to semantics to obtain the sentence content; S1-2: Prepare the bonito model, input the training task, and use the extraction question-answering mode. The longest token of the model is set to 256; S1-3: Input the sentence content into the bonito model to generate a preliminary redundant data set, manually filter out the valid data and filter out the dirty data, which is the data in which the generated question-answering answers are wrong or have no answers; The question and answer data collected manually from the test question bank is added to the Bonito model to automatically generate valid data. The set of valid data is the training data set. S2: Model fine-tuning; Based on the open source large model Llmal3, the LORA method is used to fine-tune the model for the training data set obtained in step S1, including the following steps: S2-1: Given a pre-trained autoregressive language model P parameterized by Φ Φ (y|x), P Φ (y|x) can be a general multi-task learner, which can be a GPT model based on the Transformer architecture; each downstream task is represented by a training dataset of context-target pairs as: Z={(x i ,y i )} i＝1,...,N where x i and i is a tag sequence. During full fine-tuning, the model pre-training weight is initialized to Φ0 and updated to Φ0+ΔΦ through continuous gradient following to maximize the conditional language modeling objective. The formula is as follows: S2-2: If the pre-trained autoregressive language model is large, training the model through full fine-tuning requires a lot of computing power. A more efficient method is used to optimize the parameters, including the following steps: The task-specific parameter increment ΔΦ = ΔΦ(Θ) is further encoded by a smaller parameter set Θ, where |Θ| = |Φ0|, thus transforming the task of finding ΔΦ into optimizing the parameter set Θ: S2-3: For the pre-trained weight matrix W0∈R d×r , by decomposing the iterative matrix, namely: W0+ΔW=W0+BA To suppress the update speed of the model, where B∈R d×r ,A∈R r×k , and the rank of the matrix is ​​r = min(d,k); During the training process, the pre-trained weight matrix W0 is not updated with the gradient, and only the parameters in the process matrices A and B are updated. When h=W0x, the forward transfer expression after optimization of the present invention is: h=W0x+ΔWx=W0x+BAx; S3: Knowledge base building and question answering; The original PDF file is extracted and sentence segmented using the method of step S1, and the Chinese sentences are encoded using the Pinecone vector search method and stored as a Pinecone vector library. When a question is raised, the program also Pinecone encodes the question, compares the similarity with the code in the vector library, finds the vector with the highest similarity, and completes the establishment of the knowledge base.

Images