Large language model corpus construction method for power supply industry

Abstract

The invention provides a power supply industry-oriented large language model corpus construction method. The method comprises the following steps of 1, deploying an RAG system in a working environment of a power supply industry; 2, performing knowledge retrieval by using the RAG system to obtain a question text and a plurality of answer texts, namely question and answer texts corresponding to the question text; 3, vectorizing the question text to obtain a question vector; step 4, all problem vectors are clustered, and each cluster is coded to obtain a cluster code; 5, encoding all question and answer texts according to clustering categories, and segmenting the question and answer texts into question fragments according to a time sequence; step 6, extracting the question of the corpus in the question segment, generating an answer, and calculating the confidence of the answer; and 7, cataloguing the question, the corresponding answer and the confidence into a corpus record, and storing all corpus records to obtain a large language model corpus for the power supply industry.

Description

Technical Field

[0001] The present invention relates to a method for constructing a large language model corpus, in particular to a method for constructing a large language model corpus for the power supply industry. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Large Language Model (LLM) has the ability to interact with users through dialogue and question-and-answer sessions and can understand user needs. LLM can be used to automatically process documents, such as automatically generating summaries or key points; it can also be used to create intelligent search tools to help quickly find the required information, or answer user questions through a question-and-answer system. However, the cost of LLM training is extremely high. Therefore, in the development of applications for specific industries, free pre-trained LLM is usually used as a base tool to provide support for upper-level applications. Since pre-trained LLM is trained with an open corpus, it lacks data from specific industries or accounts for a small proportion; although it can meet general application needs, it performs generally in specific industry applications, which requires fine-tuning it in combination with the industry knowledge base. Although the corpus required for fine-tuning LLM is not as large as that for training general models, its data set is still large. If manual annotation is used for fine-tuning, the cost of collecting corpora is still high.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for constructing a large language model corpus for the power supply industry in view of the shortcomings of the prior art.

[0006] In order to solve the above technical problems, the present invention discloses a method for constructing a large language model corpus for the power supply industry, comprising the following steps:

[0007] Step 1: deploy the RAG system in the working environment of the power supply industry;

[0008] Step 2, using the RAG system to perform knowledge retrieval to obtain a question text and a plurality of corresponding answer texts, namely, question and answer texts;

[0009] Step 3, vectorize the question text to obtain the question vector;

[0010] Step 4: cluster all question vectors and encode each cluster to obtain cluster code;

[0011] Step 5: Encode all question and answer texts according to the clustered categories and divide them into question segments in time sequence;

[0012] Step 6, extracting the question of the corpus from the question segment, generating an answer, and calculating the confidence of the answer;

[0013] Step 7: catalog the question, the corresponding answer and the confidence level into a corpus record, and save all the corpus records to obtain a large language model corpus for the power supply industry.

[0014] Further, the RAG system described in step 1 includes:

[0015] Local knowledge base, local knowledge base retrieval module, pre-trained LLM model, interactive question and answer module, storage and export of interactive question and answer text module.

[0016] Furthermore, the use of the RAG system for knowledge retrieval described in step 2 includes: collecting interactive question and answer texts in the knowledge retrieval process and performing preprocessing.

[0017] Furthermore, the preprocessing described in step 2 includes:

[0018] The interactive question and answer texts in the knowledge retrieval process are classified by users, arranged in text order, and marked with questions and answers to obtain question texts and answer texts.

[0019] Furthermore, the clustering described in step 4 uses the K-Means clustering method.

[0020] Furthermore, in the K-Means clustering method described in step 4, the distance adopts the cosine distance.

[0021] Furthermore, the cosine distance described in step 4 is calculated as follows:

[0022]

[0023] Where d represents the cosine distance between vector A and vector B, A i represents the i-th element in vector A, B i represents the i-th element in vector B, and n represents the number of elements in vector A and vector B.

[0024] Furthermore, the problem of extracting the corpus from the question segment described in step 6 is to use the first question in the question segment as the corpus.

[0025] Furthermore, the answer generation described in step 6 is to use the LLM model to merge all the answers corresponding to the question in the corpus in the question segment to generate a summary as the answer corresponding to the question.

[0026] Furthermore, the calculation of the confidence of the answer in step 6 includes:

[0027] Different confidence levels are assigned to the corpus according to the number of questions asked in each segment. The greater the number of questions asked, the lower the confidence level.

[0028] Beneficial effects:

[0029] 1. The present invention enhances the capabilities of the model by accessing an external knowledge base, so that the model can generate more accurate and context-relevant responses.

[0030] 2. The present invention collects questions and answers from industry users in their daily use of LLM, and automatically processes the collected corpus according to question rounds, question and answer changes, and the correlation between questions and answers, thereby creating a high-quality fine-tuned corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0032] Figure 1 It is a schematic diagram of the workflow of the present invention.

[0033] Figure 2 It is a system structure block diagram for implementing the method of the present invention.

[0034] Figure 3 This is the document import interface diagram for the system to build a local knowledge base.

[0035] Figure 4 It is the operation interface diagram of the system using documents to build a local knowledge base.

[0036] Figure 5 This is a diagram of the interactive question-and-answer interface of the system using the local knowledge base and LLM. DETAILED DESCRIPTION

[0037] The general idea of ​​the present invention is as follows:

[0038] By collecting questions and answers from industry users in their daily use of LLM, the collected corpus is automatically processed according to the question rounds, question and answer changes, and the correlation between questions and answers, thereby creating a high-quality fine-tuned corpus.

[0039] The present invention adopts a retrieval-augmented generation system (RAG), which is an artificial intelligence technology that combines information retrieval technology with a language generation model. It is based on an existing large-scale language model and enhances the model's capabilities by accessing an external knowledge base, so that the model can generate more accurate and context-sensitive responses. Its work is roughly divided into three steps: data preparation stage: The first step is data extraction, text segmentation, vectorization, and data storage. In this process, data from different sources are processed into a unified format and converted into vector form and stored in a special database for subsequent retrieval. The second step is that when a user asks a question or request, the RAG system first uses the retrieval model to find the information most relevant to the user's request from the knowledge base. This information is then integrated into the user's original request to form an enhanced prompt. The third step is that the enhanced prompt is submitted to the large language model, and the model generates the final answer based on this prompt containing external information.

[0040] The technical solution proposed in this invention is a method for constructing a large language model fine-tuning corpus for industry applications, which is used to improve the ability of pre-trained LLM to adapt to industry applications and reduce the cost of manually annotating corpora. Figure 1 As shown, the specific steps are as follows:

[0041] Step 1: Deploy the RAG system in the industry user work environment; the system has a local knowledge base, local knowledge base retrieval function, pre-trained LLM, interactive question and answer function, and storage and export of interactive question and answer text functions;

[0042] Step 2: The collected interactive question-and-answer texts are classified by user, arranged in order, and marked with questions and answers;

[0043] Step 3: Vectorize the question and answer texts separately, that is, convert each question and its corresponding multiple answers into vectors;

[0044] Step 4: Set the number of clusters, cluster the text vectors corresponding to all users' questions using an unsupervised clustering algorithm, and encode each cluster;

[0045] Step 5: Encode user questions by cluster and divide them into multiple segments in time sequence. The segmentation method is to classify questions that are connected in time and belong to the same category into one segment, that is, the questions in each segment belong to the same category, which means that the user is asking questions about the same topic. If two consecutive questions belong to different categories, it means that the user has switched topics.

[0046] Step 6: Take the first question in the user question segment as the question of the corpus, because the first question is often the topic that users are most concerned about; use the LLM model to merge all the answers in the segment to generate a summary as the answer to the question, because the user completes a round of conversation, which means that the user has found the answer, and the answer should be included in the answers returned multiple times; and assign different confidence levels to the corpus records according to the number of questions asked in each segment. The greater the number of questions asked, the lower the confidence level, because multiple questions indicate that the previous answers did not fully meet the user's expectations;

[0047] Step 7: The questions, answers and confidence levels collected above are catalogued into triples as a corpus record, and exported as corpus for model fine-tuning.

[0048] Example:

[0049] In order to make the pre-trained LLM better adapt to industry applications, the model needs to be fine-tuned, which requires the collection of text of industry knowledge. Taking the power supply industry as an example, combined with the attached drawings, the specific implementation plan is described in detail according to the technical solution.

[0050] According to the processing flow, the RAG system needs to be deployed in the user's working environment first. The RAG system can be divided into a knowledge base subsystem and a knowledge retrieval subsystem. The data used to build the local knowledge base are knowledge documents from the industry field, such as basic knowledge of the power grid, such as the composition, structure, and operating principle of the power transmission and distribution system, power supply industry standards and specifications, power supply equipment and technology, etc.; second, work documents, reports, logs, etc. from within the organization; third, production data; and fourth, structured data from business systems. The knowledge base system must have the ability to dynamically update knowledge, because local documents and materials will continue to accumulate, and the knowledge contained in these documents needs to be dynamically added to the knowledge base. The knowledge base can be stored in a graph database, such as Neo4j, using manual and auxiliary tools to extract entities and relationships from documents, and store them in a graph database to form a local knowledge graph. During interactive question-answering, the question text is converted into a graph query language, such as Cypher, and relevant information is retrieved from the knowledge graph. A vector database, such as Faiss, can also be used to store text and build a knowledge vector library. The document is segmented into multiple text fragments by paragraph or sentence, and the segmented fragments are vectorized and stored in the knowledge base. During interactive question-answering, the question text is converted into a vector, and the vector is used to retrieve the related text paragraphs. During interactive question-answering, the question text and the retrieved results are filled into a pre-customized template to construct the LLM prompt, and the LLM will return the result expressed in natural language. During the operation of the system, the interactive question-answering text of users is continuously collected in time sequence to form the material of the corpus.

[0051] The system provides users with the function of archiving internal documents or industry data (such as Figure 3 ); then use the archived files to build a local knowledge base (such as Figure 4 As shown); Combine local knowledge base and large language model to realize interactive question answering (as shown Figure 5 As shown in Figure 2, the conversation text of the interactive question and answer session is used to construct the material of the corpus.

[0052] Then, the user question text vector in the text material is clustered with the number of clusters K, where K is an empirical value and estimated based on the number of topics queried by the user. The k-means clustering algorithm is used for clustering, and the distance is the cosine distance. The calculation method of the cosine distance d between the A and B vectors is:

[0053]

[0054] Where d represents the cosine distance between vector A and vector B, A i represents the i-th element in vector A, B i represents the i-th element in vector B, and n represents the number of elements in vector A and vector B; and encodes each cluster; and uses an unsupervised clustering algorithm to classify user texts.

[0055] Then, the question text of each user is segmented in chronological order, and two consecutive questions that do not belong to the same category are used as segmentation points. That is, the segmentation method is to classify consecutive questions that belong to the same category into one segment.

[0056] Then, the first question in the user question segment is used as the question of the corpus, because the first question is often the topic that users are most concerned about; the joint uses the LLM model to merge all the answers in the segment to generate a summary as the answer to the question, because the user's completion of a conversation means that the user has found the answer, and the answer should be included in the multiple returned answers; and different confidence levels are assigned to the corpus according to the number of questions asked in each segment. The greater the number of questions asked, the lower the confidence level, because multiple questions indicate that the previous answers did not fully meet the user's expectations.

[0057] The confidence value = -0.1*number of times + 1. If the number of times is greater than 10, the confidence value is 0.

[0058] Finally, the collected (question, answer, confidence) is cataloged into triples as a corpus record, and all records are exported to build a corpus for model fine-tuning.

[0059] The systems that realize the above functions can be divided into Figure 2 The four modules shown are:

[0060] The RAG module is the basis for building a local knowledge base and providing interactive question and answer services;

[0061] The question and answer collection module saves each user's question and answer text in a specified format, such as json, xml;

[0062] The question-answer processing module clusters and classifies the question texts and merges the answers in the same segment into one answer. It can use RAG's existing LLM or other LLM models to merge multiple paragraphs of text to reduce duplication and redundancy of information.

[0063] The main function of the corpus export module is to export corpus records into files in an agreed format as a corpus; the system can provide a graphical operation interface to facilitate user operations, such as providing file import and corpus export interfaces.

[0064] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a method for constructing a large language model corpus for the power supply industry provided by the present invention and some or all of the steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0065] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0066] The present invention provides a method and idea for building a large language model corpus for the power supply industry. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A method for constructing a large language model corpus for the power supply industry, characterized in that: The following steps are involved: Step 1: deploy the RAG system in the working environment of the power supply industry; Step 2, using the RAG system to perform knowledge retrieval to obtain a question text and a plurality of corresponding answer texts, namely, question and answer texts; Step 3, vectorize the question text to obtain the question vector; Step 4: cluster all question vectors and encode each cluster to obtain cluster code; Step 5: Encode all question and answer texts according to the clustered categories and divide them into question segments in time sequence; Step 6, extracting the question of the corpus from the question segment, generating an answer, and calculating the confidence of the answer; Step 7: catalog the question, the corresponding answer and the confidence level into a corpus record, and save all the corpus records to obtain a large language model corpus for the power supply industry.

2. According to claim 1, a method for constructing a large language model corpus for the power supply industry is characterized in that: The RAG system described in step 1, comprising: Local knowledge base, local knowledge base retrieval module, pre-trained LLM model, interactive question and answer module, storage and export of interactive question and answer text module.

3. The method for constructing a large language model corpus for the power supply industry according to claim 2, characterized in that: The use of the RAG system for knowledge retrieval described in step 2 includes: collecting interactive question and answer texts in the knowledge retrieval process and performing preprocessing.

4. The method for constructing a large language model corpus for the power supply industry according to claim 3 is characterized in that: The pre-processing described in step 2 includes: The interactive question and answer texts in the knowledge retrieval process are classified by users, arranged in text order, and marked with questions and answers to obtain question texts and answer texts.

5. The method for constructing a large language model corpus for the power supply industry according to claim 4, characterized in that: The clustering described in step 4 uses the K-Means clustering method.

6. The method for constructing a large language model corpus for the power supply industry according to claim 5, characterized in that: In the K-Means clustering method described in step 4, the distance uses the cosine distance.

7. The method for constructing a large language model corpus for the power supply industry according to claim 6, characterized in that: The cosine distance described in step 4 is calculated as follows: Where d represents the cosine distance between vector A and vector B, A i represents the i-th element in vector A, B i represents the i-th element in vector B, and n represents the number of elements in vector A and vector B.

8. The method for constructing a large language model corpus for the power supply industry according to claim 7, characterized in that: The problem of extracting corpus from the question segment described in step 6 is to use the first question in the question segment as the corpus.

9. The method for constructing a large language model corpus for the power supply industry according to claim 8, characterized in that: The generating of the answer described in step 6 is to use the LLM model to merge all the answers corresponding to the question in the corpus in the question segment to generate a summary as the answer corresponding to the question.

10. The method for constructing a large language model corpus for the power supply industry according to claim 9, characterized in that: The calculation of the confidence of the answer described in step 6 includes: Different confidence levels are assigned to the corpus according to the number of questions asked in each segment. The greater the number of questions asked, the lower the confidence level.

Images