A reading comprehension method and system for the power industry based on a large-scale language model
By training and fine-tuning a large-scale language model, the reading comprehension challenge in the field of power knowledge was solved, achieving efficient and accurate reading comprehension tasks in the power sector.
Patent Information
- Application Number
- CN202311046961.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-08-16
AI Technical Summary
The application of large-scale language models in the field of power knowledge is limited by the inability to answer open-ended questions and the poor performance of traditional extractive reading comprehension models in understanding professional knowledge in the power field, and the computational cost is high.
We construct a reading comprehension method for the power industry based on a large-scale language model. By training an extractive reading comprehension model and a large-scale language model for the power industry, and combining them with a power industry dataset for pre-training and fine-tuning, we can achieve segmentation processing and answer generation for long texts in the power industry.
It breaks through the limitations of existing technologies in terms of incorrect answers and open-ended question responses, and constructs a reading comprehension model and system for practical application in the field of power knowledge, thereby improving the accuracy and efficiency of answers.
Smart Images

Figure CN117113937B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of reading comprehension of electrical knowledge, and in particular to a method and system for reading comprehension in the electrical field based on a large-scale language model. Background Technology
[0002] Large-scale language models have rapidly become a major research topic due to their excellent natural language understanding capabilities, lifelike response generation, and few-shot or zero-shot learning abilities. However, their practical applications in vertical fields such as medicine, law, and power are less than ideal because they lack domain-specific expertise and their outputs contain certain "illusion" errors. Furthermore, currently deployed large-scale language models, such as ChatGPT, have parameter scales reaching hundreds of billions. Deploying such ultra-large-scale models is difficult and computationally expensive.
[0003] The results of reading comprehension tasks can be applied in practice to help people read documents and extract key information. Traditional extractive reading comprehension models are mostly based on attention mechanisms or pre-trained language models, such as BERT. They perform well for questions with explicit answers in a given context, but their performance is poor for questions involving knowledge summarization and open-ended questions. Furthermore, general-purpose reading comprehension models generally perform poorly in understanding context within specific domains.
[0004] The field of electrical engineering knowledge contains many specialized terms that are easily confused, posing a significant challenge to the comprehension capabilities of artificial intelligence models. The varying lengths of documents in this field, including lengthy texts, also present a challenge to traditional extractive reading comprehension models. In practical applications, the reliability of electrical engineering knowledge is paramount for safety reasons. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to overcome the limitations of the existing related technologies that have not been applied to the field of reading comprehension of power knowledge. The present invention provides a reading comprehension method and system for the power field based on a large-scale language model, so as to overcome the limitations of existing large-scale model technology in answering errors and traditional extraction-based reading comprehension models in being unable to answer open-ended questions, and to construct a reading comprehension model and system that can be practically applied in the field of power knowledge.
[0006] Therefore, the technical solution adopted by the present invention is as follows:
[0007] In a first aspect, this invention proposes a reading comprehension method for the power industry based on a large-scale language model, comprising the following steps:
[0008] 1) Obtain relevant text data in the power field and construct dataset D1 for model pre-training and dataset D2 for reading comprehension tasks;
[0009] 2) Based on the datasets D1 and D2 obtained in step 1), train an extractive reading comprehension model M1 and a large-scale language model M2 for the power industry, respectively;
[0010] 3) Divide the long texts in the field of electricity that are to be read and understood into paragraphs;
[0011] 4) Take the relevant questions Q of the long text in the power field to be read and the long text segmentation results obtained according to step 3) as input, and use the trained extractive reading comprehension model M1 to obtain the answer prediction results of all text segments;
[0012] 5) Determine the number of predicted answers r in the prediction results of all text segments obtained in step 4). If r = 1, output the corresponding predicted answer as the final answer. If r > 1, execute step 6). If r = 0, execute step 7).
[0013] 6) Select the text segments containing the predicted answer from the answer prediction results to form a new text, and use the new text and question Q as input to the large-scale language model M2 to generate the final answer.
[0014] 7) Based on the document matching database of question Q, the matched documents and question Q are used as input to the large-scale language model M2, and the final answer is generated through the large-scale language model M2.
[0015] Furthermore, the dataset D1 used for model pre-training is plain text data in the power field, used for masked language model training and causal language model training of the extractive reading comprehension model and the large-scale language model in the power field, respectively; each sample in the dataset D2 used for the reading comprehension task contains a question Q, context T, and answer A.
[0016] Furthermore, the training process of the extractive reading comprehension model M1 includes:
[0017] Using dataset D1, a masked language model is pre-trained on an extractive reading comprehension model based on a transformer encoder architecture, enabling the extractive reading comprehension model to incorporate professional knowledge in the power field.
[0018] The pre-trained extractive reading comprehension model was further trained using dataset D2 to obtain the trained extractive reading comprehension model M1.
[0019] Furthermore, the aforementioned pre-training of a masked language model on an extractive reading comprehension model based on a transformer encoder architecture using dataset D1 includes:
[0020] Preprocessing of dataset D1: The original text is divided into several segments according to natural paragraphs, the text is converted into word sequences by a word segmenter, and the word sequences are randomly masked to obtain masked text segments;
[0021] The embedded representation of the masked text paragraph is obtained through the embedding layer. The embedded representation of the masked text paragraph is then input into the transformer encoder to obtain the hidden representation of the masked text paragraph.
[0022] Using the hidden representation of the masked text paragraph as input, the words at the masked positions are predicted by the masked language model prediction head, and the corresponding words in the original text paragraph are used as labels to calculate the cross-entropy loss.
[0023] Based on cross-entropy loss, the parameters of the embedding layer, transformer encoder, and masked language model prediction head in the extractive reading comprehension model are updated through gradient backpropagation, thus completing the pre-training of the extractive reading comprehension model.
[0024] Furthermore, the pre-trained extractive reading comprehension model is further trained using dataset D2 to obtain a trained extractive reading comprehension model M1, including:
[0025] Preprocessing of dataset D2: The context T and question Q of each sample in dataset D2 are converted into word sequences by a word segmenter and connected by the sentence separator of the language model;
[0026] The embedded representation of the word sequence after sample preprocessing is obtained through the pre-trained embedding layer, and then input into the pre-trained transformer encoder to obtain the hidden representation of the sample.
[0027] Using the hidden representation of the sample as input, the start and end positions of the answer are predicted by the extractive reading comprehension classification head. The start and end positions of the sample's true answer A, which correspond to the context, are used as labels to calculate the cross-entropy loss.
[0028] Based on cross-entropy loss, the parameters of the embedding layer, transformer encoder, and extractive reading comprehension classifier in the extractive reading comprehension model are updated through gradient backpropagation to complete the training of the extractive reading comprehension model and obtain the trained extractive reading comprehension model M1.
[0029] Furthermore, the training process of the large-scale language model M2 in the power field includes:
[0030] Obtain the Chinese extended vocabulary corresponding to dataset D1, and merge it with the original vocabulary of the large-scale language model to obtain a new vocabulary;
[0031] Using dataset D1, a large-scale language model based on the transformer decoder architecture is pre-trained, enabling the large-scale language model to incorporate professional knowledge in the power field.
[0032] The pre-trained large-scale language model was further trained using dataset D2, and the large-scale language model M2 was obtained through fine-tuning of instructions.
[0033] Furthermore, the aforementioned pre-training of a large-scale language model based on a transformer decoder architecture using dataset D1 includes:
[0034] Preprocessing of dataset D1: Based on the new vocabulary obtained by fusing the Chinese extended vocabulary corresponding to dataset D1 and the original vocabulary of the large-scale language model, the text data in dataset D1 is converted into word sequences by a word segmenter, and each 4096 words are extracted as a word sequence sample.
[0035] The embedding representation of word sequence samples is obtained through the embedding layer, and the embedding representation is input into the transformer decoder to obtain the hidden representation of word sequence samples;
[0036] Using the hidden representation of the word sequence sample as input, the causal language model prediction head predicts the word at the current position based on the word sequence before the current position. The word at the current position in the original word sequence is used as the label, and the cross-entropy loss is calculated.
[0037] Based on cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model are updated through gradient backpropagation, thus completing the pre-training of the large-scale language model.
[0038] Furthermore, the method of using dataset D2 to further train the pre-trained large-scale language model, and training it through instruction fine-tuning to obtain the large-scale language model M2, includes:
[0039] Preprocess dataset D2: Construct the question Q and context T of each sample in dataset D2 into the input of a large-scale language model with instructions;
[0040] The answer to question Q is generated in streaming fashion from a pre-trained large-scale language model;
[0041] Using the sample's true answer A as the label, calculate the cross-entropy loss between the generated answer and the true answer;
[0042] Based on cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model are updated through gradient backpropagation to complete the training of the large-scale language model and obtain the trained large-scale language model M2.
[0043] Furthermore, the number of predicted answers r included in the prediction results of all text segments obtained in the judgment step 4) is specifically as follows:
[0044] The prediction result p for each paragraph in the answer prediction result P that iterates through all text segments. i If p i If the start and end positions contained therein can be successfully mapped to the corresponding paragraph, it is considered that the answer has been found.
[0045] Secondly, this invention proposes a reading comprehension system for the power industry based on a large-scale language model, which is used to implement the aforementioned reading comprehension method for the power industry based on a large-scale language model.
[0046] The beneficial effects of this invention are as follows: This invention breaks through the limitations of existing large model technology in answering errors and traditional extractive reading comprehension models in being unable to answer open-ended questions, and constructs a reading comprehension model and system that can be practically applied in the field of power knowledge. Attached Figure Description
[0047] Figure 1 This is a schematic diagram of the overall architecture of a reading comprehension model for the power industry based on a large-scale language model, according to the present invention. Detailed Implementation
[0048] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.
[0049] like Figure 1 As shown, this invention is a reading comprehension method for the power industry based on a large-scale language model, comprising the following steps:
[0050] Step 1: Obtain relevant text data in the power industry and construct dataset D1 for model pre-training and dataset D2 for reading comprehension tasks.
[0051] In this embodiment, the dataset D1 used for pre-training contains 100,000 documents related to the power industry. It is plain text data and is used to train the extractive reading comprehension model and the large-scale language model in the power industry, respectively, using masked language model training and causal language model training.
[0052] The dataset D2 is used for reading comprehension tasks, where each sample contains a question Q, context T, and answer A. In this embodiment, the dataset was constructed by having professionals in the power industry annotate 5,000 question-answer pairs by reading relevant literature in the power industry, and then generating 500,000 pairs based on these human-annotated question-answer pairs using ChatGPT.
[0053] Step 2: Based on the datasets D1 and D2 obtained in Step 1, train an extractive reading comprehension model M1.
[0054] First, based on the pre-trained dataset D1 obtained in step one, an extractive reading comprehension model based on a transformer encoder architecture is pre-trained, enabling the model to incorporate professional knowledge from the power industry. In this embodiment, BERT is selected as the base model for this extractive reading comprehension model.
[0055] Secondly, the pre-trained extractive reading comprehension model obtained above is trained again using the dataset D2 obtained in step one for the reading comprehension task, resulting in the trained extractive reading comprehension model M1.
[0056] In an optional embodiment of the present invention, the pre-trained dataset D1 obtained in step one is pre-trained using an extractive reading comprehension model based on a transformer encoder architecture, specifically as follows:
[0057] Based on the pre-training dataset D1 obtained in step one, it is preprocessed as follows: it is divided into several segments according to natural paragraphs, the text is converted into a token sequence by the tokenizer, segments with more than 512 tokens are deleted, and 15% of the tokens are randomly masked, that is, [MASK] is used to replace the original tokens in the token sequence to obtain the masked text segments.
[0058] Next, the masked text paragraph is first passed through the embedding layer to obtain the embedded representation, and then through the transformer encoder to obtain the hidden representation of the masked text paragraph;
[0059] Then, the token at the [MASK] position is predicted using the masked language model prediction header. The token at the corresponding position in the original text is used as the tag y. The loss for y is calculated using the cross-entropy loss function;
[0060] Finally, based on the cross-entropy loss, the parameters of the embedding layer, transformer encoder, and masked language model prediction head in the extractive reading comprehension model are updated through gradient backpropagation, thus completing the pre-training of the extractive reading comprehension model.
[0061] In an optional embodiment of the present invention, the pre-trained extractive reading comprehension model obtained above is retrained based on the dataset D2 obtained in step one for the reading comprehension task, specifically as follows:
[0062] First, based on the reading comprehension task dataset D2 obtained in step one, the question Q and context T of each sample are converted into a token sequence through a tokenizer, connected through the sentence separator [SEP] of the language model, and after passing through the pre-trained embedding layer, the embedded representation of the sample is obtained.
[0063] Next, the embedding representation of the word sequence after sample preprocessing is obtained through the pre-trained embedding layer, and then input into the pre-trained transformer encoder to obtain the hidden representation of the sample;
[0064] Then, using the hidden representation of the sample as input, the starting position pos_st′ and ending position pos_ed′ of the answer are predicted by the extractive reading comprehension classification head. The starting and ending positions of the true answer A of the sample, corresponding to the context, are pos_st and pos_ed, respectively. The loss is calculated using the cross-entropy loss function.
[0065] Finally, based on the obtained cross-entropy loss, the parameters of the embedding layer, transformer encoder, and extractive reading comprehension classifier in the extractive reading comprehension model are updated through gradient backpropagation to complete the training of the extractive reading comprehension model and obtain the trained extractive reading comprehension model M1.
[0066] Step 3: Based on the datasets D1 and D2 obtained in Step 1, train a large-scale language model M2 for the power industry.
[0067] First, based on the pre-training dataset D1 obtained in step one, the corresponding Chinese extended vocabulary is obtained through the SentencePiece method, and then fused with the original vocabulary of the large-scale language model to obtain a new vocabulary, namely the tokenizer.
[0068] Next, based on the pre-training dataset D1, a large-scale language model based on a transformer decoder architecture is pre-trained to enable the model to acquire professional knowledge in the power industry. In this embodiment, the LLaMA-7B model is selected as the base model for the large-scale language model.
[0069] Finally, based on the reading comprehension task dataset D2 and the pre-trained large-scale language model obtained above, the large-scale language model M2 is trained through instruction fine-tuning.
[0070] In an optional embodiment of the present invention, pre-training is performed using a large-scale language model based on a transformer decoder architecture, according to the pre-training dataset D1, specifically as follows:
[0071] First, the pre-trained dataset D1 and the newly obtained word list tokenizer are used to convert the text data in D1 into a token sequence, and every 4096 tokens are extracted as a word sequence sample.
[0072] Next, based on the obtained word sequence samples, the word sequence sample embedding vector is obtained through the embedding layer, and then the hidden representation of the word sequence samples is obtained through the transformer decoder;
[0073] Then, using the hidden representation of the word sequence samples as input, the causal language model prediction head predicts the token at the current position based on the token sequence preceding the current position, denoted as . Let y be the token at the current position in the original token sequence, and calculate the loss using the cross-entropy loss function.
[0074] Then, based on the obtained cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model are updated through gradient backpropagation, thus completing the pre-training of the large-scale language model.
[0075] Repeat the above steps for three complete training cycles to obtain a large-scale language model containing knowledge of the power field.
[0076] In an optional embodiment of the present invention, based on the reading comprehension task dataset D2 and the pre-trained large-scale language model obtained above, the large-scale language model M2 is trained by fine-tuning through instructions, specifically as follows:
[0077] First, based on the reading comprehension task dataset D2, the question Q and context T for each data point are constructed into the input of a large-scale language model in the form of instructions, i.e., prompts, and used as the input to the large-scale language model:
[0078] In this embodiment, Prompt: "Answer the question based on the given content.\n###Content: {T}\n\n###Question: {Q}\n\nAnswer:"
[0079] Next, the answer to question Q is generated in streaming mode by the pre-trained large-scale language model; the cross-entropy loss between the generated answer and the real answer is calculated using the real answer A of the sample as the label; the use of a large-scale language model based on a transformer decoder architecture to generate the answer to question in streaming mode is a well-known technique in this field, and its principle will not be elaborated here.
[0080] Finally, based on cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model are updated through gradient backpropagation to complete the training of the large-scale language model and obtain the trained large-scale language model M2.
[0081] Step 4: Divide the long text on the power sector to be read and understood into N text segments, i.e., T = {t1, t2, ..., t3}. i …t N If long texts in the power industry exist in PDF file format, it is necessary to parse the PDF file to obtain the text data. This can be achieved using the Fitz library in Python, which parses the PDF and divides the obtained text data into N segments based on natural text paragraphs.
[0082] Step 5: Based on the long text segmentation results T obtained in Step 4, and the relevant questions Q of the long text in the power field to be read and understood, the predicted answer P = {p1, p2, ... p1} for each text segment is obtained using the extractive reading comprehension model M1 based on the Tansformer encoder architecture. i …p N}
[0083] Step Six: Based on the extraction-based reading comprehension prediction result P obtained in Step Five, where r represents the number of answers that can be predicted in P, iterate through the prediction results p for each paragraph. i If p i If the start and end positions contained in the code can be successfully mapped to the corresponding paragraph, it is denoted as "the answer has been found", and r = r + 1. Otherwise, r remains unchanged, and the initial value of r is 0. If r = 1, the corresponding predicted answer is directly output as the final answer. If r > 1, proceed to step seven. If r = 0, proceed to step eight.
[0084] Step 7: Based on the prediction result P obtained in Step 5, select the corresponding text segments that can predict the answer to form a new text, and use the new text and question Q as input to the large-scale language model to generate the final answer through the large-scale language model M2.
[0085] Step 8: Based on question Q, match documents in the database, select the most relevant document and question to form the input of the large-scale language model, and generate the final answer through the large-scale language model M2.
[0086] In this step, the database is constructed based on the power-related documents obtained in step one. The question Q and the document database Ds are taken as input, and the most relevant paragraphs in the database are matched using the BM25 algorithm. The document containing the most relevant paragraph is then output as the most relevant document.
[0087] In steps seven and eight, the input to the large-scale language model is in the above-described prompt format.
[0088] In this embodiment, the power domain reading comprehension method based on a large-scale language model proposed in this invention is evaluated against a general Chinese domain model and a traditional extractive reading comprehension model on a constructed power domain reading comprehension dataset.
[0089] Five hundred manually constructed question-answer pairs, which were not used for training, were selected from the reading comprehension dataset in the power field as evaluation data. Each sample contains a question Q, context T, and reference answer A, i.e., {Q,T,A}.
[0090] Evaluation methods included manual assessment, ChatGPT assessment, and metric assessment. Manual assessment involved scoring the model-generated answer by a human, given the question, context, and a reference answer; the score range was [0, 10]. ChatGPT assessment involved calling the ChatGPT interface to evaluate the model's generated answer based on the context and question, and then providing a score. Metric assessment primarily used the perfect match rate (EM).
[0091] Table 1. Evaluation Results
[0092] Model Manual assessment ChatGPT Assessment Indicator Evaluation A general reading comprehension model based on BERT 35 39.60 37.2 Chinese-LLaMA-7B 78.3 73.91 38.3 This invention 90.2 87.25 39.5
[0093] As shown in Table 1, in the evaluation dataset, compared with traditional general reading comprehension models and the best large-scale language model with the same parameter level, this invention achieved better human evaluation and ChatGPT evaluation, and significantly outperformed the other two competitive models. This demonstrates that this invention can effectively complete reading comprehension tasks in the field of electrical knowledge.
[0094] This embodiment also provides a reading comprehension system for the power industry based on a large-scale language model. This system is used to implement the above embodiments, and details already described will not be repeated. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.
[0095] In this embodiment, a reading comprehension system for the power industry based on a large-scale language model includes:
[0096] The training data acquisition module is used to acquire text data related to the power industry and construct dataset D1 for model pre-training and dataset D2 for reading comprehension tasks.
[0097] The extractive reading comprehension model M1 consists of an embedding layer, a transformer encoder, and an extractive reading comprehension classification head. It takes the relevant questions and the segmentation results of the long text in the power field to be read and comprehension as input, and generates the answer prediction results for all text segments. The answer prediction results are the start and end positions of the corresponding answers for each text segment.
[0098] The extractive reading comprehension model M1 training module is used to perform two-stage training on the extractive reading comprehension model M1 using datasets D1 and D2. In the first stage of training, a masked language model prediction head is introduced to replace the extractive reading comprehension classification head, and the extractive reading comprehension model M1 is trained using the masked language model based on dataset D1. In the second stage of training, the pre-trained extractive reading comprehension model is further trained using dataset D2 to obtain the trained extractive reading comprehension model M1.
[0099] M2, a large-scale language model for the power industry, consists of an embedding layer, a transformer decoder, and a causal language model prediction head. It takes the relevant questions and their context of the long text in the power industry to be read and understood as input, and generates the answers to the questions.
[0100] The training module for the large-scale language model M2 in the power field is used to perform two-stage training on the large-scale language model M2 in the power field using datasets D1 and D2. In the first stage of training, the large-scale language model M2 in the power field is trained as a causal language model based on dataset D1. In the second stage of training, the pre-trained large-scale language model in the power field is trained again using dataset D2 to obtain the trained large-scale language model M2 in the power field.
[0101] The reading comprehension module is used to segment long texts in the power sector into paragraphs. The segmentation results and related questions from the long text are used as input. An extractive reading comprehension model M1 is used to obtain predicted answers for all text segments. The module then determines the number r of predicted answers. If r = 1, the corresponding predicted answer is directly output as the final answer. If r > 1, a new text is created by selecting text segments containing predicted answers from the predicted answers. This new text and the questions are then used as input to a large-scale language model M2, which generates the final answer. If r = 0, the module matches documents from a question-matching database, and the matched documents and questions are used as input to the large-scale language model M2, which generates the final answer.
[0102] The specific implementation process of the functions and roles of each module in the above system is detailed in the corresponding steps of the above method, and will not be repeated here. For the system embodiment, since it basically corresponds to the method embodiment, relevant parts can be referred to in the description of the method embodiment. The system embodiment described above is merely illustrative; the modules described as separate components may or may not be physically separated, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0103] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.
[0104] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A reading comprehension system for the power industry based on a large-scale language model, characterized in that, include: The training data acquisition module is used to acquire text data related to the power industry and construct dataset D1 for model pre-training and dataset D2 for reading comprehension tasks. The extractive reading comprehension model M1 consists of an embedding layer, a transformer encoder, and an extractive reading comprehension classification head. It takes the relevant questions and the segmentation results of the long text in the power field to be read and understood as input, and generates the answer prediction results for all text segments. The answer prediction results are the start and end positions of the corresponding answers for each text segment. The extractive reading comprehension model M1 training module is used to perform two-stage training on the extractive reading comprehension model M1 using datasets D1 and D2. In the first stage of training, a masked language model prediction head is introduced to replace the extractive reading comprehension classification head, and the extractive reading comprehension model M1 is pre-trained using the masked language model based on dataset D1. In the second stage of training, the pre-trained extractive reading comprehension model M1 is further trained using dataset D2 to obtain the trained extractive reading comprehension model M1. M2, a large-scale language model for the power industry, consists of an embedding layer, a transformer decoder, and a causal language model prediction head. It takes the relevant questions and their context of a long text in the power industry to be read and understood as input, and generates the answers to the questions. The training module for the large-scale language model M2 in the power field is used to perform two-stage training on the large-scale language model M2 in the power field using datasets D1 and D2. In the first stage of training, the large-scale language model M2 in the power field is pre-trained as a causal language model based on dataset D1. In the second stage of training, the pre-trained large-scale language model M2 in the power field is further trained using dataset D2 to obtain the trained large-scale language model M2 in the power field. The reading comprehension module is used to segment long texts in the power sector into paragraphs. The segmentation results and related questions from the long text are used as input. An extractive reading comprehension model M1 is used to obtain predicted answers for all text segments. The module then determines the number r of predicted answers. If r = 1, the corresponding predicted answer is directly output as the final answer. If r > 1, a new text is created by selecting text segments containing predicted answers from the predicted answers. This new text and the questions are then used as input to a large-scale language model M2, which generates the final answer. If r = 0, the module matches documents from a question-matching database, and the matched documents and questions are used as input to the large-scale language model M2, which generates the final answer.
2. A reading comprehension method for the power industry based on a large-scale language model, applied to the power industry reading comprehension system based on a large-scale language model as described in claim 1, characterized in that, Includes the following steps: 1) Obtain relevant text data in the power field and construct dataset D1 for model pre-training and dataset D2 for reading comprehension tasks; 2) Based on the datasets D1 and D2 obtained in step 1), train the extractive reading comprehension model M1 and the large-scale language model M2 for the power industry, respectively; 3) Divide the long texts in the field of electricity that are to be read and understood into paragraphs; 4) Take the relevant questions Q of the long text in the power field to be read and the long text segmentation results obtained in step 3) as input, and use the trained extractive reading comprehension model M1 to obtain the answer prediction results of all text segments; 5) Determine the number r of predicted answers in the predicted results of all text segments obtained in step 4). If r = 1, output the corresponding predicted answer as the final answer. If r > 1, proceed to step 6). If r = 0, proceed to step 7). 6) Select the text segments containing the predicted answer from the answer prediction results to form a new text, and use the new text and question Q as input to the large-scale language model M2 to generate the final answer; 7) Based on the document matching database of question Q, the matched documents and question Q are used as input to the large-scale language model M2, and the final answer is generated through the large-scale language model M2.
3. The reading comprehension method for the power sector based on a large-scale language model according to claim 2, characterized in that, Dataset D1 is plain text data in the field of electricity, and each sample in dataset D2 contains a question Q, context T, and answer A; The dataset D1 is used for masked language model pre-training of the extractive reading comprehension model M1 and causal language model pre-training of the large-scale language model M2 in the power field.
4. The reading comprehension method for the power sector based on a large-scale language model according to claim 2, characterized in that, The training process of the extractive reading comprehension model M1 includes: Using dataset D1, a masked language model is pre-trained on the extractive reading comprehension model M1 based on the transformer encoder architecture, enabling the extractive reading comprehension model M1 to incorporate professional knowledge in the power field. The pre-trained extractive reading comprehension model M1 is further trained using dataset D2 to obtain the trained extractive reading comprehension model M1.
5. The reading comprehension method for the power sector based on a large-scale language model according to claim 4, characterized in that, The aforementioned pre-training of a masked language model on an extractive reading comprehension model M1 based on a transformer encoder architecture using dataset D1 includes: Preprocessing of dataset D1: The original text is divided into several segments according to natural paragraphs, the text is converted into word sequences by a word segmenter, and the word sequences are randomly masked to obtain masked text segments; The embedded representation of the masked text paragraph is obtained through the embedding layer. The embedded representation of the masked text paragraph is then input into the transformer encoder to obtain the hidden representation of the masked text paragraph. Using the hidden representation of the masked text paragraph as input, the words at the masked positions are predicted by the masked language model prediction head, and the corresponding words in the original text paragraph are used as labels to calculate the cross-entropy loss. Based on cross-entropy loss, the parameters of the embedding layer, transformer encoder, and masked language model prediction head in the extractive reading comprehension model M1 are updated through gradient backpropagation, thus completing the pre-training of the extractive reading comprehension model M1.
6. The reading comprehension method for the power sector based on a large-scale language model according to claim 4, characterized in that, The process of using dataset D2 to further train the pre-trained extractive reading comprehension model M1 to obtain the trained extractive reading comprehension model M1 includes: Preprocessing of dataset D2: The context T and question Q of each sample in dataset D2 are converted into word sequences by a word segmenter and connected by the sentence separator of the language model; The embedded representation of the word sequence after sample preprocessing is obtained through the pre-trained embedding layer, and then input into the pre-trained transformer encoder to obtain the hidden representation of the sample. Using the hidden representation of the sample as input, the start and end positions of the answer are predicted by the extractive reading comprehension classification head. The start and end positions of the sample's true answer A, which correspond to the context, are used as labels to calculate the cross-entropy loss. Based on cross-entropy loss, the parameters of the embedding layer, transformer encoder, and extractive reading comprehension classification head in the extractive reading comprehension model M1 are updated through gradient backpropagation to complete the training of the extractive reading comprehension model M1 and obtain the trained extractive reading comprehension model M1.
7. The reading comprehension method for the power sector based on a large-scale language model according to claim 2, characterized in that, The training process of the large-scale language model M2 in the power field includes: Obtain the Chinese extended vocabulary corresponding to dataset D1, and merge it with the original vocabulary of the large-scale language model M2 to obtain a new vocabulary; Using dataset D1, a large-scale language model M2 based on a transformer decoder architecture is pre-trained, enabling the large-scale language model M2 to incorporate professional knowledge in the power field. The pre-trained large-scale language model M2 is trained using dataset D2, and the large-scale language model M2 is obtained through fine-tuning of instructions.
8. The reading comprehension method for the power sector based on a large-scale language model according to claim 7, characterized in that, The aforementioned pre-training of a large-scale language model M2 based on a transformer decoder architecture using dataset D1 includes: Preprocessing of dataset D1: Based on the new vocabulary obtained by fusing the Chinese extended vocabulary corresponding to dataset D1 and the original vocabulary of the large-scale language model M2, the text data in dataset D1 is converted into word sequences by a word segmenter, and each 4096 words are extracted as a word sequence sample. The embedding representation of word sequence samples is obtained through the embedding layer, and the embedding representation is input into the transformer decoder to obtain the hidden representation of word sequence samples; Using the hidden representation of the word sequence sample as input, the causal language model prediction head predicts the word at the current position based on the word sequence before the current position. The word at the current position in the original word sequence is used as the label, and the cross-entropy loss is calculated. Based on cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model M2 are updated through gradient backpropagation, thus completing the pre-training of the large-scale language model M2.
9. The reading comprehension method for the power sector based on a large-scale language model according to claim 7, characterized in that, The method of using dataset D2 to further train the pre-trained large-scale language model M2, and obtaining the large-scale language model M2 through instruction fine-tuning, includes: Preprocess the dataset D2: construct the question Q and context T of each sample in the dataset D2 into the input of a large-scale language model M2 with instructions; The answer to question Q is generated in streaming form by the pre-trained large-scale language model M2. Using the sample's true answer A as the label, calculate the cross-entropy loss between the generated answer and the true answer; Based on cross-entropy loss, the parameters of the embedding layer, transformer decoder, and causal language model prediction head in the large-scale language model M2 are updated through gradient backpropagation to complete the training of the large-scale language model M2 and obtain the trained large-scale language model M2.
10. The reading comprehension method for the power sector based on a large-scale language model according to claim 2, characterized in that, The number of predicted answers r contained in the prediction results of all text segments obtained in step 4) is specifically as follows: The prediction result for each paragraph in the answer prediction result P that iterates through all text segments. ,like If the start and end positions contained in the code can be successfully mapped to the corresponding paragraph, it is denoted as finding the answer, r = r + 1; otherwise, r remains unchanged, and the initial value of r is 0.
Citation Information
Patent Citations
Extraction type machine intelligent reading understanding question-answering system
CN111611361A
Data processing method and device for machine reading understanding, equipment and medium
CN114328883A
