A few-shot machine reading comprehension method for bridge detection text

Abstract

This invention relates to the field of bridge text information retrieval technology, specifically to a few-sample machine reading comprehension method for bridge detection text. In this method, even with a limited amount of sample data, each sample can be fully utilized. Simulated prediction data can be constructed using a question classification model and an answer prediction model, while actual training data can be obtained through manual annotation. Compared to existing technologies, this method fully develops and utilizes the collected text data. By using simulated training data and text data to be annotated, the answer extraction model undergoes two training sessions after pre-training (Pre-Training) and fine-tuning (Pre-Tuning). Compared to existing training methods that use pre-training + fine-tuning, this method significantly improves the training effect of the answer extraction model even when only identical text data is available.

Description

Technical Field

[0001] This invention relates to the field of bridge text information retrieval technology, specifically to a few-sample machine reading comprehension method for bridge detection text. Background Technology

[0002] As a crucial data source in bridge management, bridge inspection reports contain vast amounts of fine-grained data, allowing bridge managers to extract information such as the condition of bridge components, structural defects, location and causes of damage, and recommended remediation measures. However, bridge inspection reports are mostly in text format, with complex content, making it time-consuming and labor-intensive to retrieve relevant information using traditional manual query methods.

[0003] In recent years, with the development of deep learning and big data processing technologies, natural language processing (NLP) algorithms and models have been gradually applied to various fields. As a branch of NLP, machine reading comprehension can quickly and accurately locate the answer to a given question in a corpus, and is currently widely used in vertical industries such as search engines and intelligent customer service. However, due to the complexity and specialization of bridge detection text content, it is difficult to obtain a large amount of training data for training machine reading comprehension models. Existing public domain few-shot machine reading comprehension models have a large number of parameters and long algorithm training times, making them difficult to apply to vertical fields such as bridge detection. In few-shot cases, the traditional training method is pre-training followed by fine-tuning. Based on the model obtained from pre-training, fine-tuning is performed, targeting the training-related parts to achieve the training objective and meet the usage requirements. However, to achieve satisfactory results, this training method also requires a certain amount of training data in the fine-tuning stage. However, bridge detection text in this field is very difficult to obtain. Therefore, it is difficult to achieve good training results by using the pre-training + fine-tuning method.

[0004] Therefore, how to improve the training effect of the answer extraction model with only a small number of samples, thereby improving the accuracy of information extraction from bridge detection text, has become an urgent problem to be solved in the training of the answer extraction model. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a few-sample machine reading comprehension method for bridge detection text, which can improve the training effect of the answer extraction model with only a small number of samples, thereby improving the accuracy of information extraction from bridge detection text.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A few-shot machine reading comprehension method for bridge detection text includes the following steps:

[0008] S1. Collect a preset number of labeled text data and a preset number of unlabeled text data; the labels include manually set questions and answers;

[0009] S2. Input the unlabeled text data into the question classification model to obtain the predicted question type, and construct the unlabeled text data and the predicted question type into pseudo data for answer prediction;

[0010] S3. Input the pseudo-data used for answer prediction into the answer prediction model to obtain the predicted answer;

[0011] S4. Based on the unlabeled text data, the corresponding predicted question type, and the predicted answer, construct pseudo-question-answer pair data. The content of the pseudo-question-answer pair data includes the unlabeled text data, the predicted answer, and the corresponding question.

[0012] S5. Use pseudo-question answer pairs to fine-tune the pre-trained answer extraction model using data;

[0013] S6. Fine-tune the pre-tuned answer extraction model using labeled text data;

[0014] S7. Use the finely tuned answer extraction model to extract information from the bridge detection text.

[0015] Basic scheme principles and effects:

[0016] Using this method, the sample data can be reused multiple times after it is obtained. On the one hand, by setting annotations, the text data to be labeled for fine-tuning can be obtained; on the other hand, pseudo-question-answer pairs can be constructed through the processing of question classification models and answer prediction models.

[0017] Because question classification and answer prediction models cannot guarantee very high accuracy, pseudo-question-answer pairs may appear identical to manually labeled data, but their actual content differs to varying degrees. For example, pseudo-question-answer pairs may contain errors in answer placement or question type. Since the accuracy of pseudo-question-answer pairs is insufficient, they cannot be used for fine-tuning; otherwise, they would negatively impact the training during the fine-tuning phase. This method proposes adding a pre-tuning stage before fine-tuning: using the outputs of the question classification and answer prediction models to construct pre-tuning training data (i.e., text data + pseudo-question-answer pairs, which can be considered simulated training data), this data is used to train the answer extraction model to understand the semantic information and writing characteristics of the bridge detection text, as well as the format and methods of machine reading comprehension tasks. Then, the answer extraction model is fine-tuned using labeled data (i.e., actual training data). This is equivalent to using constructed simulated training data to coarsely optimize the answer extraction model, learn and understand the semantic information and writing characteristics of bridge detection text, and then using actual training data to finely optimize the answer extraction model.

[0018] In this way, even if the amount of sample data obtained is not large, each sample can be used to construct simulated prediction data through question classification models and answer prediction models, and at the same time, actual training data can be obtained through manual annotation. Each sample data can be fully utilized. Compared with existing technologies, this method makes full use of the collected text data. By simulating training data and text data to be labeled, the answer extraction model can undergo two training sessions after pre-training: pre-tuning and fine-tuning. Compared with the existing training method of using pre-training + fine-tuning, it can also significantly improve the training effect of the answer extraction model when only the same text data can be collected.

[0019] In summary, this method can improve the accuracy of information extraction from bridge detection text even with only a small number of samples.

[0020] Preferably, in S2, the problem classification model is a first base model trained with labeled text data; the underlying model of the first base model is the Chinese pre-trained language model MacBERT.

[0021] Preferably, the training process of the first base model includes:

[0022] h i =w MacBERT (s i );

[0023] p i =w MacBERTpooler (h i );

[0024] c i =softmax(linear(p) i ));

[0025] o p =argmax(c i );

[0026] loss = CrossEntropyLoss(o p ,o g );

[0027] Among them, s i Represents a text containing the answer, w MacBERT This represents a MacBERT encoding block. This represents the hidden layer word vectors obtained after MacBERT encoding; w MacBERTpooler This represents the encoded block of the MacBERTpooler, which is used to transform word vectors into sentence vectors. s i The sentence vectors are encoded; the encoded sentence vectors are input into a linear layer and normalized to obtain the probability of each question category. The argmax(·) function is used to obtain the marker position with the highest probability for each category. The corresponding question category is the question category corresponding to the answer contained in the sentence. The loss is the training loss value, which is calculated by the cross-entropy loss function. The backpropagation algorithm is used to gradually reduce the value of the loss value to achieve the training effect.

[0028] Beneficial effects: This approach can fully utilize the text data to be labeled set by staff. Furthermore, the question classification obtained through this question classification model can better train the answer extraction model to understand the semantic information and writing style of the bridge detection text, and to understand the form and method of the task during subsequent training.

[0029] Preferably, in S3, the answer prediction model is a second base model trained with labeled text data; the underlying model of the second base model is the Chinese pre-trained language model MacBERT.

[0030] Preferably, the training process of the second base model includes:

[0031] h i =w MacBERT (o p ,s′i );

[0032] p = softmax(linear(h) i ));

[0033] loss = CrossEntropyLoss(Start) g ,p[0])+CrossEntropyLoss(End g ,p[1]);

[0034] Among them, w MacBERT This represents a MacBERT encoding block. This represents the hidden layer word vectors obtained after MacBERT encoding; the encoded hidden layer vectors are input into a linear layer and normalized to obtain the probabilities of the start and end positions of the answer. p[0] represents the probability that the position is the start position of the answer, and p[1] represents the probability that the position is the end position of the answer. g Indicates the starting position of the true answer, End g This indicates the end position of the true answer; loss represents the loss value during the training of the second base model, calculated using the cross-entropy loss function. The backpropagation algorithm is used to update the parameters and gradually reduce the value of the loss to achieve the training effect.

[0035] Beneficial effects: This approach fully utilizes the text data to be labeled set by staff. Furthermore, the answers obtained through this answer prediction model can be used in subsequent training to better train the answer extraction model to understand the semantic information and writing style of the bridge detection text, and to comprehend the form and method of the task.

[0036] Preferably, in S4, the process of generating answers from the data for pseudo-question answering includes:

[0037] Start p =argmax(p[0]);

[0038] End p =argmax(p[1]);

[0039] Answer=s′ i [Start p End p ];

[0040] Here, Answer represents the answer in a pseudo-question-answer pair.

[0041] Preferably, in S1, based on the content of the text data of bridge detection, the problem types corresponding to the extracted information are summarized and analyzed to obtain a problem suffix tag table; the problem suffix tag table includes suffix tags for various problem types.

[0042] Preferably, in S4, the process of generating questions from the data for pseudo-question answering includes: assigning the suffix label p of the question type predicted by the question classification model to the question type. i A text segment span' preceding the answer segment in unmarked text. i By concatenating the parts, we obtain the corresponding question q. i =span′ i +p i .

[0043] Beneficial effects: This approach ensures the accuracy of both the question type and the question content, enabling the rapid and accurate generation of corresponding questions and guaranteeing the validity of the data used in the subsequent pre-tuning stage.

[0044] Preferably, in S1, the suffix markers for the problem type include: What is it?, How much is it?, Repair suggestion?, Where is it located?, Reason?, Problem?, and No obvious suffix.

[0045] Beneficial effects: These problem types, marked with intuitive and concise suffixes, comprehensively cover the problems that may occur when extracting information for bridge detection. This not only ensures the effectiveness of subsequent training but also guarantees the accuracy and efficiency of information extraction.

[0046] Preferably, the underlying model of the answer extraction model is the Chinese pre-trained language model MacBERT.

[0047] Beneficial effects: The answer extraction model and the answer prediction model use the same underlying model, which can ensure the effectiveness of learning the semantic information and writing characteristics of the bridge detection text, understanding the form and method of the task, and also ensure the training efficiency of the answer extraction model. Attached Figure Description

[0048] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0049] Figure 1 Flowchart for an embodiment;

[0050] Figure 2 This is a schematic diagram illustrating the process of generating training data for the problem extraction model in this embodiment;

[0051] Figure 3This is a schematic diagram illustrating the training process of the problem extraction model in the embodiment. Detailed Implementation

[0052] The following detailed explanation illustrates the specific implementation methods:

[0053] Example:

[0054] like Figure 1 As shown, this embodiment discloses a few-shot machine reading comprehension method for bridge detection text, including the following steps:

[0055] S1. Collect a preset number of labeled text data and a preset number of unlabeled text data; the labels include manually set questions and answers. Based on the content of the bridge inspection text data, summarize and analyze the question types corresponding to the extracted information to obtain a question suffix tag table; the question suffix tag table includes suffix tags for various question types. The suffix tags for the question types include: What?, How much?, Maintenance suggestion?, Where is it located?, Reason?, Problem?, and No obvious suffix. For ease of understanding, the suffixes for each question type are shown in Table 1:

[0056] Table 1 Problem Suffix Marking Table

[0057]

[0058] These problem types, labeled with intuitive and concise suffixes, comprehensively cover the problems that may arise during information extraction for bridge detection. This not only ensures the effectiveness of subsequent training but also guarantees the accuracy and efficiency of information extraction.

[0059] S2. Input the unlabeled text data into the question classification model to obtain the predicted question type, and construct pseudo-data for answer prediction using the unlabeled text data and the predicted question type. Specifically, the question classification model is a first base model trained with labeled text data; the underlying model of the first base model is the Chinese pre-trained language model MacBERT. The training process of the first base model includes:

[0060] h i =w MacBERT (s i );

[0061] p i =w MacBERTpooler (h i );

[0062] c i =softmax(linear(p) i ));

[0063] o p =argmax(c i );

[0064] loss = CrossEntropyLoss(o p ,o g );

[0065] Among them, s i Represents a text containing the answer, w MacBERT This represents a MacBERT encoding block. This represents the hidden layer word vectors obtained after MacBERT encoding; w MacBERTpooler This represents the encoded block of the MacBERTpooler, which is used to transform word vectors into sentence vectors. s i The sentence vectors are encoded; the encoded sentence vectors are input into a linear layer and normalized to obtain the probability of each question category. The `argmax()` function is used to obtain the marker position with the highest probability for each category; the corresponding question category is the question category of the answer contained in the sentence. The `loss` is the training loss value, calculated using the cross-entropy loss function. Backpropagation is used to gradually reduce the value of the loss value to achieve the desired training effect. In practical applications, the underlying MacBERT model of the question classification model can be replaced with any other pre-trained Chinese language model, including Chinese-Bert-WWM, SpanBERT, RoBerta, etc. The final bridge detection text information extraction performance will decrease slightly, but the impact is minimal.

[0066] The unlabeled bridge detection text is segmented by periods to obtain s′. i . s′ i The questions are sequentially input into a question classification model to obtain their types, and pseudo-data suitable for answer prediction is constructed based on this. <o p ,s′ i >

[0067] S3. Input the pseudo-data used for answer prediction into the answer prediction model to obtain the predicted answer. Specifically, the answer prediction model is trained using labeled text data on a second base model; the underlying model of the second base model is the Chinese pre-trained language model MacBERT. The training process of the second base model includes:

[0068] h i =w MacBERT (o p ,s′ i );

[0069] p = softmax(linear(h) i ));

[0070] loss = CrossEntropyLoss(Start) g ,p[0])+CrossEntropyLoss(End g ,p[1]);

[0071] Among them, w MacBERT This represents a MacBERT encoding block. This represents the hidden layer word vectors obtained after MacBERT encoding; the encoded hidden layer vectors are input into a linear layer and normalized to obtain the probabilities of the start and end positions of the answer. p[0] represents the probability that the position is the start position of the answer, and p[1] represents the probability that the position is the end position of the answer. g Indicates the starting position of the true answer, End g This indicates the end position of the true answer; loss represents the loss value during the training of the second base model, calculated using the cross-entropy loss function. The backpropagation algorithm is used to update the parameters and gradually reduce the value of the loss to achieve the training effect.

[0072] S4. Based on the unlabeled text data, the corresponding predicted question type, and the predicted answer, construct pseudo-question-answer pair data. The content of the pseudo-question-answer pair data includes the unlabeled text data, the predicted answer, and the corresponding question, such as... Figure 2 As shown.

[0073] In practice, the process of generating answers from the data using pseudo-question answering includes:

[0074] Start p =argmax(p[0]);

[0075] End p =argmax(p[1]);

[0076] Answer=s′ i [Start p End p ];

[0077] Here, Answer represents the answer in a pseudo-question-answer pair.

[0078] The process of generating questions in the pseudo-question answering dataset includes: taking the suffix label p of the question type predicted by the question classification model. i A text segment span' preceding the answer segment in unmarked text.i By concatenating the parts, we obtain the corresponding question q. i =span′ i +p i .

[0079] This approach ensures accuracy in both question type and question content, allowing for quick and accurate question generation and guaranteeing the validity of data used in subsequent pre-tuning. To illustrate, consider a simple example: If the answer to the text "The bridge is 1800 meters long" is "1800 meters," then the preceding text is "The bridge is long," and the question type is "How much?", so the question would be "How long is the bridge?".

[0080] S5. Use pseudo-question-answering pairs to fine-tune the pre-trained answer extraction model. It should be noted that in this embodiment, the base model of the large extraction model is the same as the base model of the answer prediction model—both are the pre-trained Chinese language model MacBERT. This ensures the effectiveness of learning the semantic information and writing characteristics of the bridge detection text, understanding the form and method of the task, and also guarantees the training efficiency of the answer extraction model.

[0081] In addition, during actual training, the answer prediction model can be directly used as the answer extraction model for pre-tuning training. However, since the training objectives and input sentence lengths of the answer prediction model and the answer extraction model are different, the training time will increase. At the same time, it will affect the semantic expression of word vectors of the pre-trained language model for bridge detection text, resulting in poor information extraction effect of the final bridge detection text.

[0082] S6. Fine-tune the pre-tuned answer extraction model using labeled text data. For example... Figure 3 As shown.

[0083] S7. Use the finely tuned answer extraction model to extract information from the bridge detection text.

[0084] The traditional pre-training + fine-tuning training method requires a large amount of annotated text data for the fine-tuning stage. If the amount of text data used for training is small, the information extraction performance of the trained answer extraction model will be extremely poor. Table 2 shows some of the extraction results when the sample size is 16.

[0085] Table 2 Information extraction results of the answer extraction model trained in Pre-Training+Fine-Tuning

[0086]

[0087]

[0088] In cases with few samples, traditional pre-training and fine-tuning paradigms suffer from weak semantic understanding of bridge detection texts due to insufficient training samples. They also fail to learn the form and methods of machine reading comprehension tasks, resulting in information extraction models outputting nothing on some test data (Example 1) or an incorrect result (Example 2).

[0089] Using this method, the sample data can be reused multiple times after it is obtained. On the one hand, by setting annotations, the text data to be labeled for fine-tuning can be obtained; on the other hand, pseudo-question-answer pairs can be constructed through the processing of question classification models and answer prediction models.

[0090] Because question classification and answer prediction models cannot guarantee very high accuracy, pseudo-question-answer pairs may appear identical to manually labeled data, but their actual content differs to varying degrees. For example, pseudo-question-answer pairs may contain errors in answer placement or question type. Since the accuracy of pseudo-question-answer pairs is insufficient, they cannot be used for fine-tuning; otherwise, they would negatively impact the training during the fine-tuning phase. This method proposes adding a pre-tuning stage before fine-tuning: using the outputs of the question classification and answer prediction models to construct pre-tuning training data (i.e., text data + pseudo-question-answer pairs, which can be considered simulated training data), this data is used to train the answer extraction model to understand the semantic information and writing characteristics of the bridge detection text, as well as the format and methods of machine reading comprehension tasks. Then, the answer extraction model is fine-tuned using labeled data (i.e., actual training data). This is equivalent to using constructed simulated training data to coarsely optimize the answer extraction model, learn and understand the semantic information and writing characteristics of bridge detection text, and then using actual training data to finely optimize the answer extraction model.

[0091] In this way, even if the amount of sample data obtained is not large, each sample can be used to construct simulated prediction data through question classification models and answer prediction models, and at the same time, actual training data can be obtained through manual annotation. Each sample data can be fully utilized. Compared with existing technologies, this method makes full use of the collected text data. By simulating training data and text data to be labeled, the answer extraction model can undergo two training sessions after pre-training: pre-tuning and fine-tuning. Compared with the existing training method of using pre-training + fine-tuning, it can also significantly improve the training effect of the answer extraction model when only the same text data can be collected.

[0092] Similarly, with a sample size of 16, the partial extraction results after pre-tuning are shown in Table 3.

[0093] Table 3 Information extraction results of the answer extraction model trained by Pre-Training+Pre-Tuning+Fine-Tuning

[0094]

[0095] Table 3 clearly shows that after adding pre-tuning, the model's information extraction performance in bridge detection text is significantly improved, demonstrating the importance of pre-tuning in few-sample information extraction for bridge detection text using machine reading comprehension methods.

[0096] This method can improve the accuracy of information extraction from bridge detection text when there are only a few samples.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A few-shot machine reading comprehension method for bridge detection text, characterized in that, The method comprises the following steps: S1, collecting a preset number of text data with labels and a preset number of text data without labels; the labels include questions and answers set by humans; S2, inputting the unlabeled text data into a question classification model to obtain a predicted question type, and constructing the unlabeled text data and the predicted question type as pseudo data for answer prediction; S3, inputting the pseudo data for answer prediction into an answer prediction model to obtain a predicted answer; S4, constructing pseudo question and answer pair data according to the unlabeled text data, the corresponding predicted question type and the predicted answer, the content of the pseudo question and answer pair data including the unlabeled text data, the predicted answer and the corresponding question; S5, using the pseudo question and answer pair data to pre-fine-tune the pre-trained answer extraction model; S6, using the labeled text data to fine-tune the pre-fine-tuned answer extraction model; S7, using the fine-tuned answer extraction model to perform information extraction on the bridge detection text; In S2, the question classification model is a first base model trained by using the labeled text data; The bottom model of the first base model is a Chinese pre-trained language model MacBERT; In S3, the answer prediction model is a second base model trained by using the labeled text data; the bottom model of the second base model is a Chinese pre-trained language model MacBERT; The bottom model of the answer extraction model is a Chinese pre-trained language model MacBERT.

2. The method of claim 1, wherein the method is a few-shot machine reading comprehension method for bridge detection text. The training process of the first base model comprises: wherein, represents a text containing an answer, represents an encoding block of MacBERT, represents a hidden layer word vector obtained after MacBERT encoding; represents an encoding block of MacBERTpooler, and the MacBERTpooler is used to convert the word vector into a sentence vector, represents a sentence vector of ; the encoded sentence vector is input into a linear layer and normalized for post-processing to obtain the probability of each question category ; the argmax(·) function is used to obtain the label position with the maximum probability of each category, and the corresponding question category is the question category corresponding to the answer contained in the sentence; is a loss value of training, which is calculated by a cross-entropy loss function, and the value of the loss value is gradually reduced by using a back propagation algorithm, so as to achieve the training effect.

3. The method of claim 2, wherein the method is a few-shot machine reading comprehension method for bridge detection text. The training process of the second base model comprises: wherein, represents the encoding block of MacBERT, represents the hidden layer word vector obtained after MacBERT encoding; the hidden layer vector obtained by encoding is input into a linear layer and normalized for post-processing to obtain the answer start position and end position probability ; represents the probability of all positions belonging to the answer start position, represents the probability of all positions belonging to the answer end position, represents the start position of the true answer, represents the end position of the true answer; represents the loss value during training of the second base model, which is calculated by a cross-entropy loss function, and the parameter is updated using a back propagation algorithm to gradually reduce the numerical value of the loss value, achieving the training effect.

4. The few-shot machine reading comprehension method for bridge detection text of claim 3, wherein: In S4, the generation process of the answer in the pseudo question and answer pair data comprises: Wherein, Answer represents the answer in the pseudo question and answer pair.

5. The few-shot machine reading comprehension method for bridge detection text according to claim 4, wherein: In S1, the question suffix label table is obtained by summarizing and analyzing the question types corresponding to the extracted information according to the content of the bridge detection text data; the question suffix label table includes suffix labels of various question types.

6. The few-shot machine reading comprehension method for bridge detection text of claim 5, wherein: In S4, the generation process of the question in the pseudo question-answer pair data includes: splicing the suffix label of the question type predicted by the question classification model , and a text segment before the answer segment in the unmarked text to obtain the corresponding question .

7. The method of claim 6, wherein the method is a few-shot machine reading comprehension method for bridge detection text. In S1, the suffix labels of the question types include what, how much, repair suggestion, where, reason, question, and no obvious suffix.

Images