Reading comprehension method, system, electronic device and storage medium based on fragment extraction

Abstract

The present application relates to a method, system, electronic device and storage medium for reading comprehension based on fragment extraction, wherein the method includes: a prediction model training step, using training data to train a pre-trained model and obtain a first prediction model; a prediction model fine-tuning step, using fine-tuning training data to fine-tune the first prediction model and the linear layer, update parameters and obtain a second prediction model; an answer acquisition step, splicing the question to be predicted and the context text to obtain a predicted input text, and inputting it into the second prediction model, obtaining the start position probability, end position probability and fragment position probability of the answer corresponding to the question to be predicted, and obtaining at least one answer fragment text. By improving the prediction model fine-tuning step and the answer acquisition step, increasing the fragment position probability, at least one answer fragment text can be obtained, and the scenarios of single answer and multiple answers can be used at the same time, which increases the applicability of the method.

Description

Technical Field

[0001] The present application relates to the field of computer information technology, and in particular to a reading comprehension method, system, electronic device and storage medium based on segment extraction. Background Art

[0002] In recent years, with the rapid development of deep learning technology, machine reading comprehension (MRC) tasks have become a research hotspot in the field of natural language processing. Machine reading comprehension refers to answering a specific question based on a given context text. There are four main types of machine reading comprehension tasks: cloze, multiple choice, fragment extraction, and free answer. Among them, fragment extraction refers to extracting a continuous sentence or phrase from the context text as the answer to a specific question given the context text and the specific question text. This task faces a larger search space and is more challenging, so fragment extraction reading comprehension is also a very important technology in the field of information extraction.

[0003] Currently, there are three technical methods for fragment extraction reading comprehension: rule-based reading comprehension, interactive model-based reading comprehension, and pre-trained model-based reading comprehension. However, the existing technology has the following shortcomings: lack of sensitivity to specific fields, and only single fragment extraction reading comprehension, which is not suitable for multiple answer scenarios in practical applications.

[0004] Currently, no effective solution has been proposed for the technical problem that existing technologies are not applicable to multi-answer scenarios. Summary of the invention

[0005] The embodiments of the present application provide a reading comprehension method, system, electronic device and storage medium based on fragment extraction, so as to at least solve the problem that the related art is not applicable to the multi-answer scenario.

[0006] In a first aspect, an embodiment of the present application provides a reading comprehension method based on fragment extraction, comprising:

[0007] A prediction model training step, training the pre-trained model using the training data, and obtaining a first prediction model;

[0008] A prediction model fine-tuning step, using the fine-tuning training data to fine-tune the first prediction model and the linear layer, update the parameters and obtain the second prediction model;

[0009] In the answer acquisition step, the predicted question text and the context text are concatenated to obtain a predicted input text, and the predicted input text is input into a second prediction model to obtain the starting position probability, the ending position probability and the fragment position probability of the answer corresponding to the predicted question text, and to obtain at least one answer fragment text.

[0010] In some embodiments, the training data includes general corpus and secondary training data, the secondary training data is training data of a set domain, and the prediction model training step further includes:

[0011] The model is trained once, using common corpus to train the preset network structure and obtain the pre-trained model;

[0012] The second model training step uses the second training data to train the pre-trained model until the model loss value drops to a preset threshold value, thereby obtaining a first prediction model.

[0013] In some embodiments, the fine-tuning training data includes question text and context text, the fine-tuning training data is pre-labeled with a start position label, an end position label, and a segment position label, and the prediction model fine-tuning step further includes:

[0014] Input text acquisition step, concatenating the question text and the context text to obtain the training input text;

[0015] A word-granularity semantic representation vector acquisition step, inputting the training input text into the first prediction model, the encoding layer of the first prediction model encodes the training input text, and outputs the word-granularity semantic representation vector of the training input text, the dimension of the word-granularity semantic representation vector is [L, H], where L corresponds to the length of the training input text, and H corresponds to the hidden layer size of the first prediction model;

[0016] The starting position probability acquisition step inputs the word-granularity semantic representation vector into the starting linear layer and the ending linear layer respectively to obtain the starting position probability and the ending position probability;

[0017] The fragment position probability acquisition step is to perform dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and input them into the fragment linear layer to output the fragment position probability;

[0018] The model loss acquisition step is to calculate the start position loss, end position loss and fragment position loss respectively using the cross entropy loss function according to the marked start position label, end position label, fragment position label and start position probability, end position probability and fragment position probability, and then calculate the total loss accordingly;

[0019] The model fine-tuning training step uses the annotated training input text to train the first prediction model, adjusts the parameters of the encoding layer, the start linear layer, the end linear layer, and the fragment linear layer until the total loss meets the preset conditions, and obtains the second prediction model.

[0020] In some embodiments, the step of obtaining the segment position probability further includes:

[0021] The word-granularity semantic representation vector increases one dimension in different directions and sets its size to 1 to obtain the first tensor and the second tensor;

[0022] The first tensor and the second tensor are respectively copied in the increased dimension, and converted into a first extended vector and a second extended vector;

[0023] The first extended vector and the second extended vector are processed to obtain the input tensor and input it into the fragment linear layer to obtain the fragment position probability.

[0024] In some embodiments, obtaining at least one answer segment text comprises:

[0025] According to the obtained starting position probability and ending position probability, multiple fragment texts are obtained, and the formula is used to filter according to the fragment position probability to obtain at least one answer fragment text. The specific formula is:

[0026]

[0027] in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

[0028] In a second aspect, the present application embodiment provides a reading comprehension system based on segment extraction, including:

[0029] A prediction model training module, which trains the pre-trained model using the training data and obtains a first prediction model;

[0030] A prediction model fine-tuning module, which uses the fine-tuning training data to fine-tune the first prediction model and the linear layer, updates the parameters and obtains a second prediction model;

[0031] The position prediction module concatenates the question text to be predicted and the context text to obtain a predicted input text, and inputs it into the second prediction model to obtain the starting position probability, ending position probability and fragment position probability of the answer corresponding to the question text to be predicted, and obtains at least one answer fragment text.

[0032] In some embodiments, the training data includes general corpus and secondary training data, the secondary training data is training data of a set domain, and the prediction model training model further includes:

[0033] The model is trained once, using common corpus to train the preset network structure and obtain the pre-trained model;

[0034] The model secondary training unit uses the secondary training data to train the pre-trained model until the model loss value drops to a preset threshold value, thereby obtaining a first prediction model.

[0035] In some embodiments, the fine-tuning training data includes question text and context text, the fine-tuning training data is pre-labeled with a start position label, an end position label, and a segment position label, and the prediction model fine-tuning module further includes:

[0036] An input text acquisition unit concatenates the question text and the context text to obtain a training input text;

[0037] A word-granularity semantic representation vector acquisition unit is used to input the training input text into the first prediction model, and the encoding layer of the first prediction model encodes the training input text and outputs the word-granularity semantic representation vector of the training input text, wherein the dimension of the word-granularity semantic representation vector is [L, H], wherein L corresponds to the length of the training input text, and H corresponds to the hidden layer size of the first prediction model;

[0038] The starting position probability acquisition unit inputs the word-granularity semantic representation vector into the starting linear layer and the ending linear layer respectively to obtain the starting position probability and the ending position probability;

[0039] The segment position probability acquisition unit performs dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and inputs them into the segment linear layer to output the segment position probability;

[0040] The model loss acquisition unit calculates the start position loss, end position loss and fragment position loss respectively using the cross entropy loss function according to the marked start position label, end position label, fragment position label and start position probability, end position probability and fragment position probability, and calculates the total loss accordingly;

[0041] The model fine-tuning training unit uses the annotated training input text to train the first prediction model, adjusts the parameters of the encoding layer, the start linear layer, the end linear layer and the fragment linear layer until the total loss meets the preset conditions, and obtains the second prediction model.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the reading comprehension method based on fragment extraction as described in the first aspect above is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the reading comprehension method based on fragment extraction as described in the first aspect above.

[0044] Compared with the related art, the reading comprehension method, system, electronic device and storage medium based on segment extraction provided by the embodiments of the present application can be applied to the field of deep learning technology and the field of natural language processing technology, and has the following advantages or beneficial effects:

[0045] Using secondary training on the basis of the pre-trained model to enhance the pre-trained model's ability to understand data in a specific field can greatly improve the accuracy of the extractive reading comprehension model. At the same time, this method can obtain at least one answer fragment text by improving the fine-tuning stage and the prediction stage, and introduce the probability of the fragment position. It can use both single-answer and multi-answer scenarios at the same time, which increases the applicability of the extractive reading comprehension model; at the same time, it increases the accuracy of the extraction. The details of one or more embodiments of the present application are presented in the following drawings and descriptions to make other features, purposes, and advantages of the present application more concise and easy to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0047] In the attached picture:

[0048] Figure 1 is a flow chart of a reading comprehension method based on segment extraction according to an embodiment of the present application;

[0049] Figure 2 is a flowchart of the prediction model training steps according to an embodiment of the present application;

[0050] Figure 3 is a flowchart of the steps of fine-tuning the prediction model according to an embodiment of the present application;

[0051] Figure 4 is a flow chart of a reading comprehension method based on segment extraction according to a preferred embodiment of the present application;

[0052] Figure 5is a flow chart of a fine-tuning process according to an embodiment of the present application;

[0053] Figure 6 is a structural block diagram of a reading comprehension system based on segment extraction according to an embodiment of the present application;

[0054] Figure 7 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Description of the drawings:

[0056] Prediction model training module 1; prediction model fine-tuning module 2; location prediction module 3;

[0057] Model primary training unit 11; model secondary training unit 12; input text acquisition unit 21;

[0058] Word granularity semantic representation vector acquisition unit 22; start position probability acquisition unit 23;

[0059] Segment position probability acquisition unit 24; Model loss acquisition unit 25;

[0060] Model fine-tuning training unit 26; processor 81; memory 82; communication interface 83; bus 80. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0062] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0063] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0064] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0065] In the prior art, methods for reading comprehension of fragment extraction include rule-based reading comprehension, interactive model-based reading comprehension, and pre-trained model-based reading comprehension. The following is a brief description of the above technologies, where:

[0066] Rule-based reading comprehension observes large amounts of data, manually summarizes and formulates complex rules for extraction.

[0067] Based on the reading comprehension of the interactive model, the encoding layer uses CNN or RNN to encode the context text and question text respectively, the interaction layer uses the attention mechanism to interact between the two, the decoding layer obtains the position of the predicted answer in the context text, and the output layer outputs the answer fragment. (CN112527999A)

[0068] Reading comprehension based on pre-trained models: Use the pre-trained model to encode the context text and questions, fine-tune the training of downstream tasks, obtain the position of the predicted answer in the context text, and output the answer fragment text.

[0069] In the reading comprehension of the pre-trained model, since the pre-trained model is trained on a large-scale general corpus, it lacks sensitivity to the field to which the target dataset belongs, which will have a certain impact on the accuracy of the model; at the same time, fragment extraction reading comprehension can be divided into single fragment extraction reading comprehension and multi-fragment extraction reading comprehension according to the task objectives. In the existing technology, most of them assume that there is only one answer fragment in the context text, so it is only applicable to single fragment extraction reading comprehension tasks. In most cases, there is more than one answer in the text, so it is not applicable to multi-fragment extraction reading comprehension.

[0070] Based on the above problems, this embodiment provides a reading comprehension method based on fragment extraction. Figure 1 is a flowchart of a reading comprehension method based on segment extraction according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0071] Prediction model training step S1, training the pre-trained model using training data and obtaining a first prediction model;

[0072] Prediction model fine-tuning step S2, using the fine-tuning training data to fine-tune the first prediction model and the linear layer, update the parameters and obtain the second prediction model;

[0073] In the answer acquisition step S3, the question text to be predicted and the context text are concatenated to obtain a predicted input text, and the predicted input text is input into the second prediction model to obtain the start position probability, end position probability and fragment position probability of the answer corresponding to the question text to be predicted, and obtain at least one answer fragment text.

[0074] Through the above steps, by improving the prediction model fine-tuning step S2 and the answer acquisition step S3, and introducing the fragment position probability, the second prediction model can perform multi-fragment information extraction tasks, thereby increasing the applicability of the extractive reading comprehension model.

[0075] Figure 2 is a flowchart of the prediction model training steps according to an embodiment of the present application, such as Figure 2 As shown, in some embodiments, the training data includes general corpus and secondary training data, the secondary training data is training data of a set field, and the prediction model training step S1 further includes:

[0076] A model training step S11, using common corpus to train a preset network structure and obtain a pre-trained model;

[0077] In the model secondary training step S12, the pre-trained model is trained using the secondary training data until the model loss value drops to a preset threshold value, thereby obtaining a first prediction model.

[0078] For example, the set field may be the power grid field, the education industry, the construction field, etc.

[0079] Through the above steps, the accuracy of the extractive reading comprehension model can be greatly improved by using the method of continued training based on the pre-trained model to enhance the pre-trained model's ability to understand data in a specific field.

[0080] It should be noted that the pre-trained model is a model that has been trained with a data set. In an embodiment of the present application, the pre-trained model is first trained using a general corpus according to a preset network structure, and then the obtained pre-trained model is trained according to the secondary training data to obtain a first prediction model. In this process, the pre-trained model can be set to a variety of pre-trained language models such as BERT model, ERNIE, RoBERTa, etc. The structure of the specific preset network model is adjusted according to the type of pre-trained model for specific application.

[0081] The BERT model is used as an example. The BERT model is a multi-layer Transformer. The structure of the BERT model is an existing technology and will not be elaborated on here.

[0082] It should be noted that the secondary training data is used for secondary training of the pre-trained model that has been trained using general corpus. The data can be set as training data for a specific field, and each piece of data is a piece of context text.

[0083] Through the model secondary training step, the encoding ability of the pre-trained model for the target domain data is effectively enhanced for the subsequent prediction model fine-tuning step S2. When the loss value in the secondary training drops to a certain threshold, the training can be stopped and the model can be saved in the prediction model fine-tuning step S2 for standby.

[0084] Figure 3 is a flowchart of the steps of fine-tuning the prediction model according to an embodiment of the present application, such as Figure 3As shown, in some embodiments, the fine-tuning training data includes question text and context text, and the fine-tuning training data is pre-labeled with a start position label, an end position label and a fragment position label, wherein the start position label is a one-dimensional array consisting of 0 and 1, and its length is the length of the input text, wherein 0 represents that the text corresponding to the position is not the start position, and 1 represents that the text corresponding to the position is the start position; the meaning of the end position label is the same as the start position label; the fragment position label is a two-dimensional upper triangular matrix consisting of 0 and 1, wherein the i-th row and the j-th column is 0, representing that the text fragment [i, j] of the input text is not the answer, and 1 represents that the text fragment [i, j] of the input text is the answer, and the dimension size of each dimension is the length of the input text.

[0085] The prediction model fine-tuning step S2 further includes:

[0086] Input text acquisition step S21, concatenating the question text and the context text to obtain a training input text;

[0087] It should be noted that the concatenation method of the question text and the context text in the input text acquisition step S21 is: [CLS] + question text + [SEP] + context text + [SEP].

[0088] The word-granularity semantic representation vector acquisition step S22 is to input the training input text into the first prediction model, and the encoding layer of the first prediction model encodes the training input text and outputs the word-granularity semantic representation vector of the training input text. The dimension of the word-granularity semantic representation vector is [L, H], where L corresponds to the length of the training input text and H corresponds to the hidden layer size of the first prediction model.

[0089] It should be noted that the word-granularity semantic representation vector is a vector representing the features of the input text obtained by transforming the input text through the pre-trained model.

[0090] The starting position probability acquisition step S23, inputs the word granularity semantic representation vector to the starting linear layer and the ending linear layer respectively to obtain the starting position probability and the ending position probability;

[0091] It should be noted that the processing of the above-mentioned linear layer at least includes matrix conversion and related calculation of preset weights. The working principle of the linear layer is the existing technology and will not be elaborated here.

[0092] The segment position probability acquisition step S24 is to perform dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and input them into the segment linear layer to output the segment position probability;

[0093] Model loss acquisition step S25, according to the marked start position label, end position label, segment position label and start position probability, end position probability, segment position probability, use the cross entropy loss function to calculate the start position loss, end position loss and segment position loss respectively, and calculate the total loss accordingly;

[0094] Taking the calculation of the starting position loss as an example, the loss calculation process is explained. The formula of the cross entropy loss function is as follows:

[0095]

[0096] Among them, i represents the text position, represents the starting position label of position i, Represents the starting position probability of position i.

[0097] The calculation process of the end position loss and the fragment position loss is the same as the above principle and will not be repeated here.

[0098] It should be noted that after calculating using the cross entropy loss function, sigmoid can also be used to scale the loss value to adjust the loss value and further adjust the parameters of the encoding layer and the linear layer.

[0099] The total loss in the above steps is the sum of the start position loss, the end position loss and the segment position loss.

[0100] The model fine-tuning training step S26 uses the annotated training input text to train the first prediction model, adjusts the parameters of the encoding layer, the start linear layer, the end linear layer and the segment linear layer until the total loss meets the preset conditions, and obtains the second prediction model.

[0101] Through the above steps, fine-tuning mainly refers to making minor adjustments to the parameters of the pre-trained model and linear layer. Compared with the continued training process of the pre-trained model, fine-tuning can allow the pre-trained model to further deepen its understanding of the training data in a specific field, so that the encoding of the pre-trained model can adapt to downstream reading comprehension tasks.

[0102] It should be noted that fine-tuning training data is a process for fine-tuning the first prediction model and the downstream network layer (e.g., linear layer). Each piece of data needs to provide question text, context text, start position label, end position label, and span position label.

[0103] For the single-segment extraction reading comprehension task in the prior art, such as in patent CN 113268571 A, the model will only predict two probabilities: the start position probability and the end position probability. And it will be assumed that there is only one answer segment, so it is only necessary to find a pair of start position and end position that meet specific conditions, so that the start position probability and the end position probability at the start position and the end position are maximized. In order to adapt to the segment extraction problem in the multi-answer scenario, the embodiment of the present application also proposes the span position probability (segment position probability).

[0104] In some embodiments, the segment position probability obtaining step S24 further includes:

[0105] The word-granularity semantic representation vector increases one dimension in different directions and sets its size to 1 to obtain the first tensor and the second tensor;

[0106] The first tensor and the second tensor are respectively copied in the increased dimension, and converted into a first extended vector and a second extended vector;

[0107] The first extended vector and the second extended vector are processed to obtain the input tensor and input it into the fragment linear layer to obtain the fragment position probability.

[0108] In practical applications, the word-granularity semantic representation vector can add one dimension in two different directions and set its size to 1 to obtain the first three-dimensional tensor and the second three-dimensional tensor. The first three-dimensional tensor and the second three-dimensional tensor are [L, 1, H] and [1, L, H] respectively.

[0109] The first three-dimensional tensor and the second three-dimensional tensor are respectively copied in the increased dimensions to convert them into a first extended vector and a second extended vector, and the dimensions of the first extended vector and the second extended vector are both [L, L, H];

[0110] The first extended vector and the second extended vector are processed to obtain the input three-dimensional tensor and input it into the fragment linear layer to obtain the fragment position probability, and the dimension of the fragment position probability is [L,L,H*2].

[0111] It should be noted that the above processing of the first extended vector and the second extended vector may be set to be a simple concatenation or an addition and averaging method.

[0112] In some embodiments, obtaining at least one answer segment text further comprises:

[0113] According to the obtained starting position probability and ending position probability, multiple fragment texts are obtained, and the formula is used to filter according to the fragment position probability to obtain at least one answer fragment text. The specific formula is:

[0114]

[0115] in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

[0116] Through the above steps, in the answer acquisition step, the second prediction model can predict three probabilities: start position probability (start position probability), end position probability (end position probability), span position probability (segment position probability). First, multiple fragment texts are obtained according to the start position probability and the end position probability, and then the above formula is used to filter according to the fragment position probability, and finally at least one answer fragment text is obtained.

[0117] It should be noted that the method of obtaining the answer fragment text in the above technical solution is not unique. In this embodiment, the start position probability, end position probability, and span position probability can be combined to calculate the answer fragment text, or only the span position probability can be used to obtain the answer fragment text, which will not be elaborated here.

[0118] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0119] Figure 4 It is a flowchart of a reading comprehension method based on fragment extraction according to a preferred embodiment of the present application.

[0120] S401, data preparation stage.

[0121] 1. Continue training data: data used for continuing training of the pre-trained model. Each piece of data is a piece of context text, and this process does not require labeling of each piece of data.

[0122] 2. Fine-tuning training data: used to fine-tune the pre-trained model and downstream network layers after continuing training. Each piece of data needs to provide question text, context text, start position label, end position label, and span position label. The question text and context text are concatenated as the input text for the fine-tuning stage. The start position label is a one-dimensional array composed of 0 and 1, with a length equal to the length of the input text, where 0 represents that the text corresponding to this position is not the start position, and 1 represents that the text corresponding to this position is the start position; the meaning of the end position label is the same as the start position label; the span position label is a two-dimensional upper triangular matrix composed of 0 and 1, where the i-th row and j-th column is 0, indicating that the text fragment [i, j] of the input text is not the answer, and 1 represents that the text fragment [i, j] of the input text is the answer, and the dimension size of each dimension is the length of the input text.

[0123] S402, the pre-trained model continues the training phase.

[0124] Using the data from the data preparation phase, the pre-trained model BERT is used alone for further training.

[0125] The pre-trained model BERT is trained on a large-scale general corpus. The model already has a high level of cognitive ability for the laws of the language itself (such as structural information, format information, etc.).

[0126] The purpose of continuing the training process is to enhance the pre-trained model's ability to encode the target domain data for subsequent fine-tuning. The training can be stopped as long as the loss value drops to a certain threshold, and the model can be saved for use in the fine-tuning stage.

[0127] S403, fine-tuning stage.

[0128] As above Figure 5 As shown in the figure, fine-tuning mainly refers to making minor adjustments to the parameters of the pre-trained model and the linear layer. Compared with the continued training stage of the pre-trained model, fine-tuning can allow the pre-trained model to further deepen its understanding of the training data in a specific field, so that the encoding of the pre-trained model can adapt to the downstream reading comprehension tasks. Specifically:

[0129] 1. Get input text: The input text is obtained by concatenating the question text and the context text using specific concatenation symbols. The concatenation method is: [CLS] + question text + [SEP] + context text + [SEP].

[0130] 2. Obtain word-level semantic representation vector: Input the input text into the pre-trained model after further training, encode the input text, and obtain the word-level semantic representation vector sequence_out of the input text, whose dimension is [L, H]. The first dimension L corresponds to the length of the input text, and the second dimension H corresponds to the hidden layer size of the pre-trained model.

[0131] 3. Calculate position probability: In the existing single-segment extraction reading comprehension task, the position probability needs to calculate the start position probability and the end position probability. The start position probability corresponds to the probability that each character in the input text is the start position; the end position probability corresponds to the probability that each character in the input text is the end position. Input the word-granularity semantic representation vector into the start linear layer and the end linear layer respectively, and the corresponding position probability can be obtained:

[0132]

[0133]

[0134] In the embodiment of the present application, the calculation of span position probability needs to be introduced. The span position probability is a two-dimensional tensor whose dimensions in both row and column directions are the length of the input text. Assume that the tensor is S, and each value S[i, j] in S represents the probability that the text segment [i, j] of the input text is the answer.

[0135] The span position probability can be obtained by expanding the dimension of the word-granularity semantic representation vector and calculating it through the span linear layer. The specific method is:

[0136] a) The two-dimensional sequence_out tensor is converted into two three-dimensional tensors by adding a dimension of size 1 in both directions , , with dimensions of [L, 1, H] and [1, L, H] respectively.

[0137] b) respectively , Copy it in the dimension of 1 and convert it into a three-dimensional tensor with two dimensions of [L, L, H] , .

[0138] c) , Concatenate to get a three-dimensional tensor of dimension [L, L, H*2] .

[0139] d) Connect the tensor T to the span linear layer and calculate the span position probability:

[0140]

[0141] 4. Calculate model loss: Use the start position label, end position label, span position label and start position probability, end position probability, span position probability of the labeled data, and use the cross entropy loss function to calculate the start position loss, end position loss, and span position loss respectively. And calculate the total loss of the model:

[0142]

[0143] 5. Model training: Use labeled data to train the model, continuously minimize the total loss of the model, use the error back propagation algorithm to continuously update the parameters of the encoding layer and the linear layer until the loss value meets the predetermined conditions, stop training, save the model, and complete the fine-tuning training process.

[0144] S404, prediction stage.

[0145] For the data to be predicted, the question text and the context text are concatenated to obtain the input text, which is then input into the saved model to obtain the position probability predicted by the model.

[0146] In the embodiment of the present application, the prediction model can predict three probabilities: start position probability, end position probability, and span position probability. Multiple fragment texts are obtained based on the start position probability and the end position probability, and then the above fragment texts are filtered by the span position probability to obtain the answer fragment text.

[0147] Specifically, here we take the predicted span position probability S, where S[i, j] is greater than 0, indicating that the text segment [i, j] of the input text is the answer segment. Since the answer segment must exist in the context text and its start position must be smaller than the end position, assuming that the question text length is q and the context text length is c, the position of the answer segment in the input text [i, j] is:

[0148]

[0149] in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

[0150] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0151] This embodiment also provides a reading comprehension system based on fragment extraction, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0152] Figure 6 is a structural block diagram of a reading comprehension system based on segment extraction according to an embodiment of the present application, such as Figure 6 As shown, the system includes:

[0153] Prediction model training module 1, training the pre-trained model using training data and obtaining a first prediction model;

[0154] Prediction model fine-tuning module 2, fine-tuning the first prediction model using fine-tuning training data, updating parameters of the coding layer and the linear layer, and obtaining a second prediction model;

[0155] Position prediction module 3 concatenates the question text to be predicted and the context text to obtain a predicted input text, and inputs it into a second prediction model to obtain the starting position probability, ending position probability and fragment position probability of the answer corresponding to the question text to be predicted, and obtains at least one answer fragment text.

[0156] In some embodiments, the training data includes general corpus and secondary training data, the secondary training data is training data of a set domain, and the prediction model training model 1 further includes:

[0157] A model one-time training unit 11 uses general corpus to train and obtain a pre-trained model;

[0158] The model secondary training unit 12 uses the secondary training data to train the pre-trained model until the model loss value drops to a preset threshold value, thereby obtaining a first prediction model.

[0159] In some embodiments, the fine-tuning training data includes question text and context text, the fine-tuning training data is pre-labeled with a start position label, an end position label, and a segment position label, and the prediction model fine-tuning module 2 further includes:

[0160] An input text acquisition unit 21 concatenates the question text and the context text using a specific concatenation symbol to obtain a training input text;

[0161] The word-granularity semantic representation vector acquisition unit 22 inputs the training input text into the first prediction model, and the encoding layer of the first prediction model encodes the training input text and outputs the word-granularity semantic representation vector of the training input text;

[0162] The starting position probability acquisition unit 23 inputs the word granularity semantic representation vector into the starting linear layer and the ending linear layer respectively to obtain the starting position probability and the ending position probability;

[0163] The segment position probability acquisition unit 24 performs dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and inputs them into the segment linear layer to output the segment position probability;

[0164] The model loss acquisition unit 25 calculates the start position loss, the end position loss and the segment position loss respectively using the cross entropy loss function according to the marked start position label, the end position label, the segment position label and the start position probability, the end position probability and the segment position probability, and calculates the total loss accordingly;

[0165] The model fine-tuning training unit 26 uses the annotated training input text to train the first prediction model, adjusts the parameters of the encoding layer, the start linear layer, the end linear layer and the segment linear layer until the total loss meets the preset conditions, and obtains the second prediction model.

[0166] In some of the embodiments, the segment position probability obtaining unit 24 may further:

[0167] The word-granularity semantic representation vector is increased by one dimension in different directions and its size is set to 1 to obtain a first tensor and a second tensor;

[0168] The first tensor and the second tensor are respectively copied in the increased dimension, and converted into a first extended vector and a second extended vector;

[0169] The first extended vector and the second extended vector are processed to obtain an input tensor and input it into the fragment linear layer to obtain the fragment position probability.

[0170] The position of the answer fragment text in the predicted input text in the position prediction module 3 is:

[0171]

[0172] in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

[0173] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0174] In addition, combined Figure 1 The reading comprehension method based on fragment extraction described in the embodiment of the present application can be implemented by an electronic device. Figure 7 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0175] The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0176] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0177] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0178] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0179] The processor 81 implements any one of the reading comprehension methods based on segment extraction in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0180] In some of the embodiments, the electronic device may further include a communication interface 83 and a bus 80. Figure 7 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0181] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0182] The bus 80 includes hardware, software or both, and couples the components of the electronic device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0183] The electronic device can perform the prediction model training step and the prediction model fine-tuning step in the embodiment of the present application based on the acquired training data, thereby realizing the combination Figure 1 Describes a fragment extraction-based reading comprehension method.

[0184] In addition, in combination with the reading comprehension method based on segment extraction in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the reading comprehension methods based on segment extraction in the above embodiments is implemented.

[0185] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A reading comprehension method based on fragment extraction, characterized in that: include: A prediction model training step, training the pre-trained model using the training data, and obtaining a first prediction model; A prediction model fine-tuning step, using the fine-tuning training data to fine-tune the first prediction model and the linear layer, update the parameters and obtain a second prediction model; The answer acquisition step includes concatenating the question text to be predicted and the context text to obtain a predicted input text, and inputting the predicted input text into the second prediction model to obtain the start position probability, end position probability and fragment position probability of the answer corresponding to the question text to be predicted, and obtaining at least one answer fragment text; The fine-tuning training data includes question text and context text, and the fine-tuning training data is pre-labeled with a start position label, an end position label, and a segment position label. The prediction model fine-tuning step further includes: An input text acquisition step of concatenating the question text and the context text to obtain a training input text; A word-granularity semantic representation vector acquisition step, inputting the training input text into the first prediction model, wherein the encoding layer of the first prediction model encodes the training input text and outputs the word-granularity semantic representation vector of the training input text, wherein the dimension of the word-granularity semantic representation vector is [L, H], wherein L corresponds to the length of the training input text, and H corresponds to the hidden layer size of the first prediction model; A starting position probability acquisition step, inputting the word-granularity semantic representation vector into a starting linear layer and an ending linear layer respectively to obtain a starting position probability and an ending position probability; The fragment position probability acquisition step is to perform dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and input them into the fragment linear layer to output the fragment position probability; a model loss acquisition step, according to the pre-marked start position label, the end position label, the segment position label and the start position probability, the end position probability, and the segment position probability, using a cross entropy loss function to respectively calculate the start position loss, the end position loss, and the segment position loss, and calculate the total loss accordingly; A model fine-tuning training step, using the annotated training input text to train the first prediction model, adjusting the parameters of the encoding layer, the start linear layer, the end linear layer and the segment linear layer until the total loss meets a preset condition, and obtaining a second prediction model; Wherein, obtaining at least one answer fragment text comprises: A plurality of fragment texts are obtained according to the obtained starting position probability and the ending position probability, and a formula is used to filter according to the fragment position probability to obtain at least one of the answer fragment texts, and the formula is specifically: in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

2. The reading comprehension method based on segment extraction according to claim 1, characterized in that: The training data includes general corpus and secondary training data, wherein the secondary training data is training data of a set field, and the prediction model training step further includes: The model is trained once, using common corpus to train the preset network structure and obtain the pre-trained model; The second model training step uses the second training data to train the pre-trained model until the model loss value drops to a preset threshold value, thereby obtaining a first prediction model.

3. The reading comprehension method based on fragment extraction according to claim 1, characterized in that: The fragment position probability acquisition step further includes: The word-granularity semantic representation vector increases one dimension in different directions and sets its size to 1, thereby obtaining a first tensor and a second tensor; Respectively copying the first tensor and the second tensor in the increased dimension to convert them into a first extended vector and a second extended vector; The first extended vector and the second extended vector are processed to obtain an input tensor and input it to the fragment linear layer to obtain the fragment position probability.

4. A reading comprehension system based on fragment extraction, characterized in that: include: A prediction model training module, which trains the pre-trained model using the training data and obtains a first prediction model; A prediction model fine-tuning module, which uses the fine-tuning training data to fine-tune the first prediction model and the linear layer, updates the parameters and obtains a second prediction model; A position prediction module, which concatenates the question text to be predicted and the context text to obtain a predicted input text, and inputs the predicted input text into the second prediction model to obtain the start position probability, end position probability and fragment position probability of the answer corresponding to the question text to be predicted, and obtains at least one answer fragment text; The fine-tuning training data includes question text and context text, the fine-tuning training data is pre-labeled with a start position label, an end position label and a segment position label, and the prediction model fine-tuning module further includes: An input text acquisition unit concatenates the question text and the context text to obtain a training input text; A word-granularity semantic representation vector acquisition unit is configured to input the training input text into the first prediction model, wherein the encoding layer of the first prediction model encodes the training input text and outputs the word-granularity semantic representation vector of the training input text, wherein the dimension of the word-granularity semantic representation vector is [L, H], wherein L corresponds to the length of the training input text, and H corresponds to the hidden layer size of the first prediction model; A starting position probability acquisition unit inputs the word granularity semantic representation vector into a starting linear layer and an ending linear layer respectively to obtain a starting position probability and an ending position probability; A fragment position probability acquisition unit performs dimension expansion on the word-granularity semantic representation vector to obtain a first expansion vector and a second expansion vector, and inputs the first expansion vector and the second expansion vector into a fragment linear layer to output a fragment position probability; The model loss acquisition unit calculates the start position loss, end position loss and fragment position loss respectively using the cross entropy loss function according to the marked start position label, end position label, fragment position label and start position probability, end position probability and fragment position probability, and calculates the total loss accordingly; A model fine-tuning training unit is used to train the first prediction model using the annotated training input text, adjust the parameters of the encoding layer, the start linear layer, the end linear layer and the segment linear layer until the total loss meets a preset condition, and obtain a second prediction model; Wherein, obtaining at least one answer fragment text comprises: A plurality of fragment texts are obtained according to the obtained starting position probability and the ending position probability, and a formula is used to filter according to the fragment position probability to obtain at least one of the answer fragment texts, and the formula is specifically: in, , , are the starting position probability, ending position probability and fragment position probability respectively, i is the candidate starting position, j is the candidate ending position, q is the length of the question text, and c is the length of the context text.

5. The reading comprehension system based on segment extraction according to claim 4 is characterized in that: The training data includes general corpus and secondary training data, wherein the secondary training data is training data of a set field, and the prediction model training model further includes: The model is trained once, using common corpus to train the preset network structure and obtain the pre-trained model; The model secondary training unit uses the secondary training data to train the pre-trained model until the model loss value drops to a preset threshold value to obtain a first prediction model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the reading comprehension method based on segment extraction as claimed in any one of claims 1 to 3 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the reading comprehension method based on segment extraction as claimed in any one of claims 1 to 3 is implemented.

Images