Machine reading comprehension method and device thereof, electronic device, and storage medium

Abstract

The application discloses a machine reading comprehension method and device, an electronic device and a storage medium, and relates to the field of artificial intelligence, wherein the machine reading comprehension method comprises the following steps: processing received preset question text and preset paragraph text to obtain an input sequence; inputting the input sequence into a pre-training model to output a vector feature set; integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set; and determining an answer text matched with the preset question text from the preset paragraph text based on the target vector feature set. The application solves the technical problem that the semantic correlation between adjacent characters cannot be integrated in the related art, resulting in low machine reading comprehension accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a machine reading comprehension method and device, an electronic device and a storage medium. BACKGROUND

[0002] Machine reading comprehension is a challenging task in natural language processing (NLP) and has good application prospects in intelligent question answering, information retrieval and other fields. Machine reading comprehension aims to enable machines to automatically answer related questions based on relevant text content, and is an important foundation for natural language understanding (NLU) and strong artificial intelligence. At present, according to the type of answers, machine reading comprehension tasks can be divided into four types: cloze test, multiple choice, span extraction and free answer. Due to the characteristics of being more suitable for real application scenarios than cloze test and multiple choice mode, and being easier to evaluate than free answer type, span extraction machine reading comprehension has become the most popular choice. Given a paragraph and a related question, the machine needs to extract a continuous text interval in the paragraph as the answer. Not limited to a word or an entity, according to specific circumstances, the answer can be a very long sentence.

[0003] Pre-trained language models such as BERT can effectively utilize non-labeled data for model training, thereby learning rich word vector representations. For downstream tasks, there is no need to train a model from scratch, and the pre-trained model obtained from large-scale corpus can achieve very good results on new data sets through simple "fine-tuning". Pre-training methods have been continuously developed, and many excellent models have emerged, which have achieved success in the field of machine reading comprehension.

[0004] In related technologies, based on the pre-training method to complete the machine reading comprehension task, the network structure is generally to concatenate a fully connected layer after the pre-training model outputs the word vector representation, and output the probability distribution of each token (i.e. character) as the start and end position of the answer through softmax (normalized exponential function).

[0005] Figure 1 is a schematic diagram of an optional BERT-based span machine reading comprehension method according to related technologies, as shown in Figure 1As shown, the Question and Paragraph are input into the BERT model. Before inputting into the BERT model, the Question can be processed into Tok1…TokN, and the Paragraph can be processed into Tok1…TokM. The [CLS] symbol is added before Tok1…TokN, and the [SEP] symbol is added between Tok1…TokN and Tok1…TokM. The representation E of [CLS]Tok1…TokN[SEP]Tok1…TokM is then defined. CLS E1…E N E SEP E 1‘ …E M′ Input into the BERT model, output C T1…T N T SEP T 1‘ …T M′ Then, using softmax, the probability distribution of each token as the start and end positions of the answer is output. For example, T 1‘ The probability of T being the starting position (S) is Pstart1, and the probability of it being the ending position (E) is Pend1; M ‘ The probability of Pstart being S (the starting position) is Pstart. M The probability of being E (the termination position) is Pend. M .

[0006] However, machine reading comprehension based on pre-trained models such as BERT has the following drawbacks:

[0007] (1) Differences in training methods between Chinese and English pre-trained models. Currently, Chinese pre-trained models are trained based on individual characters, and their output vectors are, more accurately, character vectors. English pre-trained models, however, are word-based, which significantly impacts the acquisition of answer ranges. For example:

[0008] The / capital / city / of / China / is / Beijing;

[0009] The capital of China is Beijing.

[0010] (2) The semantic correlation between adjacent words is ignored, which is crucial for the answer interval positioning of machine reading comprehension. Whether a token belongs to the category of the answer starting word or the ending word depends on the semantic of its adjacent words to some extent, but the mode of directly outputting the start and end position probability through the full connection layer of the related technology lacks explicit modeling of the adjacent context. For example, if a token is followed by a punctuation mark, the probability of the token as the start position of the answer will be relatively low; if a word is followed by "located in", the probability of the word as the start position of the answer to a "location" type question will be relatively high.

[0011] For example:

[0012] The Palace Museum is located atBeijing,which is completed in 1420;

[0013] The Palace Museum is located atBeijing,which is completed in 1420;

[0014] (3) It is difficult to further improve the effect of the pre-training model in the downstream task of machine reading comprehension. Due to the strong feature modeling capability and large parameter quantity of the pre-training model, simply stacking various attention network layers after the pre-training model cannot significantly improve the actual effect. The current improvement method mainly comes from a larger scale of pre-training model or the introduction of external data knowledge, which is time-consuming and difficult to implement.

[0015] At present, there is no effective solution to the above problems. SUMMARY

[0016] The embodiments of the present application provide a machine reading comprehension method and device, electronic equipment and storage medium, to at least solve the technical problem that the semantic correlation between adjacent characters cannot be integrated in the related art, resulting in low accuracy of machine reading comprehension.

[0017] According to an aspect of an embodiment of the present application, a machine reading comprehension method is provided, including: processing a received preset question text and a preset paragraph text to obtain an input sequence, wherein the input sequence includes a plurality of characters; inputting the input sequence into a pre-training model to output a vector feature set, wherein the vector feature set includes a plurality of vector features, and each vector feature corresponds to a character; integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set; determining an answer text matching the preset question text from the preset paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the preset paragraph text.

[0018] Optionally, the step of processing the received preset question text and preset paragraph text to obtain an input sequence includes: determining a first symbol, a second symbol, and a third symbol, wherein the first symbol is a start symbol of the input sequence, the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and a preset supplementary text, and the preset supplementary text is an added blank text; dividing the preset question text into a plurality of question characters and dividing the preset paragraph text into a plurality of paragraph characters; and constructing the input sequence based on the first symbol, the second symbol, the third symbol, the question characters, and the paragraph characters.

[0019] Optionally, after obtaining the input sequence, the method further includes: determining an input structure of each character in the input sequence, wherein the input structure includes character semantic coding, character type coding, and position coding; and determining a number of coding layers of a preset feature extractor and a vector length of the vector feature.

[0020] Optionally, the step of inputting the input sequence into a pre-training model to output a vector feature set includes: converting a structure of each character in the input sequence into the input structure; inputting the converted input sequence into the pre-training model; and based on the pre-training model, using the preset feature extractor to code the input sequence for a continuous number of coding layers to obtain the vector feature set, wherein a length of each vector feature in the vector feature set is the vector length.

[0021] Optionally, the step of integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set includes: performing a dimension increasing operation on the vector features to obtain dimension-increased vector features; performing convolution on every two adjacent dimension-increased vector features based on a first preset convolution kernel, a first preset step, and a first preset supplementary type to obtain new vector features; determining a vector weight of the new vector features based on a second preset convolution kernel, a second preset step, and a second preset supplementary type; determining a target vector feature based on a product of the dimension-increased vector features, the vector weight, and the new vector features; and obtaining the target vector feature set based on all target vector features.

[0022] Optionally, the step of determining the answer text matching the preset question text from the preset paragraph text based on the target vector feature set comprises: performing dimension reduction on each target vector feature in the target vector feature set, and inputting the reduced target vector feature into a preset output layer; calculating a start position value and an end position value of the target vector feature by using the preset output layer; sorting all the start position values and all the end position values to obtain a sorting result; determining a maximum start position value and a maximum end position value satisfying a preset position condition based on the sorting result, wherein the preset position condition is a condition that the start position is located before the end position; and determining the answer text based on a target start position indicated by the maximum start position value and a target end position indicated by the maximum end position value.

[0023] Optionally, before the reduced target vector feature is input into the preset output layer, the method further comprises: determining a start cross-entropy loss based on a first loss parameter and a character output probability value corresponding to a real start position; determining an end cross-entropy loss based on a second loss parameter and a character output probability value corresponding to a real end position; constructing a loss function based on the start cross-entropy loss and the end cross-entropy loss; and training the preset output layer based on the loss function.

[0024] According to another aspect of the embodiments of the present application, a machine reading comprehension device is also provided, which comprises: a processing unit configured to process a received preset question text and a preset paragraph text to obtain an input sequence, wherein the input sequence comprises a plurality of characters; an output unit configured to input the input sequence into a pre-trained model to output a vector feature set, wherein the vector feature set comprises a plurality of vector features, and each vector feature corresponds to a character; an integration unit configured to integrate every two adjacent vector features in the vector feature set to obtain a target vector feature set; and a determination unit configured to determine an answer text matching the preset question text from the preset paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the preset paragraph text.

[0025] Optionally, the processing unit comprises: a first determining module configured to determine a first symbol, a second symbol, and a third symbol, wherein the first symbol is a start symbol of the input sequence, the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and a preset supplementary text, and the preset supplementary text is an increased blank text; a first dividing module configured to divide the preset question text into a plurality of question characters and divide the preset paragraph text into a plurality of paragraph characters; and a first constructing module configured to construct the input sequence based on the first symbol, the second symbol, the third symbol, the question characters, and the paragraph characters.

[0026] Optionally, the machine reading comprehension device further comprises: a second determining module configured to determine an input structure of each character in the input sequence after obtaining the input sequence, wherein the input structure comprises: character semantic encoding, character type encoding, and position encoding; and a third determining module configured to determine a number of encoding layers of a preset feature extractor and a vector length of the vector feature.

[0027] Optionally, the output unit comprises: a first converting module configured to convert a structure of each character in the input sequence into the input structure; a first input module configured to input the converted input sequence into the pre-training model; and a first encoding module configured to perform, based on the pre-training model, continuous encoding of the input sequence for the number of encoding layers by using the preset feature extractor to obtain the set of vector features, wherein a length of each vector feature in the set of vector features is the vector length.

[0028] Optionally, the integration unit comprises: a first dimension upgrading module configured to perform dimension upgrading on the vector features to obtain upgraded vector features; a first convolution module configured to perform convolution on each two adjacent upgraded vector features based on a first preset convolution kernel, a first preset step, and a first preset supplementary type to obtain new vector features; a fourth determining module configured to determine a vector weight of the new vector features based on a second preset convolution kernel, a second preset step, and a second preset supplementary type; a fifth determining module configured to determine target vector features based on a product of the upgraded vector features, the vector weight, and the new vector features; and a first output module configured to obtain the set of target vector features based on all the target vector features.

[0029] Optionally, the determining unit comprises: a first dimension reduction module, configured to perform dimension reduction operation on each target vector feature in the target vector feature set, and input the dimension-reduced target vector feature into a preset output layer; a first calculation module, configured to calculate a start position value and an end position value of the target vector feature by using the preset output layer; a first sorting module, configured to sort all the start position values and all the end position values to obtain a sorting result; a sixth determination module, configured to determine a maximum start position value and a maximum end position value that satisfy a preset position condition based on the sorting result, wherein the preset position condition is a condition that the start position is located before the end position; and a seventh determination module, configured to determine the answer text based on a target start position indicated by the maximum start position value and a target end position indicated by the maximum end position value.

[0030] Optionally, the machine reading comprehension apparatus further comprises: an eighth determination module, configured to determine a start cross-entropy loss based on a first loss parameter and a character output probability value corresponding to a true start position before inputting the dimension-reduced target vector feature into the preset output layer; a ninth determination module, configured to determine an end cross-entropy loss based on a second loss parameter and a character output probability value corresponding to a true end position; a second construction module, configured to construct a loss function based on the start cross-entropy loss and the end cross-entropy loss; and a first training module, configured to train the preset output layer based on the loss function.

[0031] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to perform the machine reading comprehension method when the computer program is running.

[0032] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises one or more processors and a memory, and the memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the machine reading comprehension method.

[0033] In this disclosure, the received preset question text and preset paragraph text are processed to obtain an input sequence. This input sequence is then fed into a pre-trained model, which outputs a vector feature set. Every two adjacent vector features in the vector feature set are integrated to obtain a target vector feature set. Based on the target vector feature set, the answer text matching the preset question text is determined from the preset paragraph text. In this disclosure, the preset question text and preset paragraph text are first processed, and then the resulting input sequence is fed into a pre-trained model. Next, every two adjacent vector features in the output vector feature set are integrated to obtain a target vector feature set. Then, the answer text matching the preset question text is determined from the preset paragraph text based on the target vector feature set. By integrating the relevant information between adjacent characters, the machine reading comprehension level can be improved, resulting in more accurate answers. This solves the technical problem in related technologies where the semantic correlation between adjacent characters cannot be integrated, leading to low accuracy in machine reading comprehension. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0035] Figure 1 This is a schematic diagram of an optional BERT-based interval machine reading comprehension method based on related technologies;

[0036] Figure 2 This is a flowchart of an optional machine reading comprehension method according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of an optional improved machine reading comprehension structure according to an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of an optional machine reading comprehension device according to an embodiment of the present invention;

[0039] Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) for a machine reading comprehension method according to an embodiment of the present invention. Detailed Implementation

[0040] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0041] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0042] In order for those skilled in the art to better understand the present application, the following explains some terms or names involved in the embodiments of the present application:

[0043] Pre-training model, a dynamic word vector representation method, is widely used in various NLP tasks. The pre-training model can be pre-trained using unlabeled text corpus to obtain dynamic vector representation of words, and through a simple fully connected layer, better results can be achieved on most downstream tasks.

[0044] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model.

[0045] The interval extraction machine reading comprehension technology inputs a user question and a related text paragraph, and the model automatically extracts a continuous text interval from the paragraph as the answer output of the user question according to the calculation result.

[0046] It should be noted that the machine reading comprehension method and the device thereof in the present disclosure can be used in the field of artificial intelligence, and can also be used in any field other than the field of artificial intelligence under the condition of machine reading comprehension. The application field of the machine reading comprehension method and the device thereof in the present disclosure is not limited.

[0047] It should be noted that the related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties. For example, an interface is provided between the system and the related user or institution. Before obtaining the related information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the related information is obtained.

[0048] The following embodiments of the present application can be applied to various systems / applications / devices for machine reading comprehension. The present application proposes a machine reading comprehension enhancement method based on neighbor semantic integration, which effectively improves the machine reading comprehension effect by integrating the semantic relationship between adjacent words and phrases while introducing a small number of additional parameters. In addition, a new network structure called Neighbor Context Integration (NCI) layer is proposed, which can be easily embedded into various pre-training models.

[0049] The present application will be described in detail below in conjunction with various embodiments.

[0050] Embodiment one

[0051] According to an embodiment of the present application, an embodiment of a machine reading comprehension method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0052] Figure 2 is a flowchart of an optional machine reading comprehension method according to an embodiment of the present application, as shown in Figure 2 The method comprises the following steps:

[0053] Step S201, processing the received preset question text and preset paragraph text to obtain an input sequence, wherein the input sequence comprises a plurality of characters.

[0054] Step S202, inputting the input sequence into a pre-training model to output a vector feature set, wherein the vector feature set comprises a plurality of vector features, and each vector feature corresponds to a character.

[0055] Step S203, integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set.

[0056] Step S204, determining an answer text matching the preset question text from the preset paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the preset paragraph text.

[0057] Through the above steps, the received preset question text and the preset paragraph text can be processed to obtain an input sequence, the input sequence is input into a pre-training model, a vector feature set is output, each two adjacent vector features in the vector feature set are integrated to obtain a target vector feature set, and an answer text matching the preset question text is determined from the preset paragraph text based on the target vector feature set. In the embodiment of the present application, the preset question text and the preset paragraph text can be processed first, then the obtained input sequence is input into the pre-training model, and then each two adjacent vector features in the output vector feature set are integrated to obtain the target vector feature set, and the answer text matching the preset question text is determined from the preset paragraph text according to the target vector feature set. By integrating the relevant information between adjacent characters, the machine reading comprehension level can be improved, and more accurate answers can be obtained, thereby solving the technical problem that the semantic correlation between adjacent characters cannot be integrated in related technologies, resulting in low machine reading comprehension accuracy.

[0058] The embodiment of the present application will be described in detail below in combination with the above steps.

[0059] Step S201, processing the received preset question text and the preset paragraph text to obtain an input sequence, wherein the input sequence includes a plurality of characters.

[0060] Optionally, the step of processing the received preset question text and the preset paragraph text to obtain an input sequence includes: determining a first symbol, a second symbol and a third symbol, wherein the first symbol is a start symbol of the input sequence, the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and a preset supplementary text, and the preset supplementary text is an added blank text; dividing the preset question text into a plurality of question characters and dividing the preset paragraph text into a plurality of paragraph characters; and constructing the input sequence based on the first symbol, the second symbol, the third symbol, the question characters and the paragraph characters.

[0061] In the embodiment of the present application, the received preset question text and the preset paragraph text can be processed first to obtain an input sequence (i.e. tokens sequence) composed of a plurality of characters, specifically: the preset question text can be represented as , and the preset paragraph text can be represented as , wherein i represents the i-th character in the question Q, m represents the length of the question text, j represents the j-th character in the paragraph P, and n represents the length of the paragraph text. Three special characters<cls> 、 <sep> 、 <sep>(i.e., determining the first symbol, the second symbol, and the third symbol), wherein <cls>The symbol (i.e., the first symbol) is generally used in a classification task and is a start symbol of the input sequence, which is used to indicate that the characters after the symbol are the question and the passage of the current input; and the two <sep>The symbols are used to distinguish the question and paragraph interval, paragraph and padding (i.e. preset supplementary text, which is additional white space text, which can be supplemented with white space (e.g. 0) until the preset text input length is reached) interval (i.e. the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and the preset supplementary text). Then, the preset question text can be divided into multiple question characters, and the preset paragraph text can be divided into multiple paragraph characters, and based on the first symbol, the second symbol, the third symbol, the question characters and the paragraph characters, an input sequence is constructed, that is, it can be spliced in the following order to obtain a tokens sequence (i.e. input sequence) with a length of m+n+3:

[0062] S=[ <cls> ,Q, <sep> ,P, <sep>];

[0063] Optionally, after obtaining the input sequence, further comprising: determining an input structure of each character in the input sequence, wherein the input structure comprises: character semantic encoding, character type encoding, and position encoding; determining the number of encoding layers of the preset feature extractor and the vector length of the vector feature.

[0064] In the embodiments of the present application, for each character s in S i , the input structure of the initial input of the model can be determined first: (i.e., determining the input structure of each character in the input sequence, which comprises: character semantic encoding character type encoding position encoding ), wherein, denotes token semantic encoding, denotes character type encoding (i.e., question type encoding or paragraph type encoding, which can be represented by 0 and 1), denotes position encoding, and all can be learned through a network.

[0065] The number of encoding layers L of the preset feature extractor (e.g., a Transformer module, which is a feature extractor) and the vector length d of the vector feature can also be determined.

[0066] Step S202: inputting the input sequence into the pre-trained model to output a vector feature set, wherein the vector feature set comprises a plurality of vector features, and each vector feature corresponds to a character.

[0067] Optionally, the step of inputting the input sequence into the pre-trained model to output the vector feature set comprises: converting the structure of each character in the input sequence into an input structure; inputting the converted input sequence into the pre-trained model; based on the pre-trained model, using a preset feature extractor to perform continuous encoding layer encoding on the input sequence to obtain a vector feature set, wherein the length of each vector feature in the vector feature set is the vector length.

[0068] In the embodiment of the present application, a pre-training model such as BERT can be used as an input text encoding layer to obtain a vector representation (i.e., a vector feature) (i.e., the input sequence can be input into the pre-training model, and a vector feature set including multiple vector features is output, each vector feature corresponding to a character). Specifically, the structure of each character in the input sequence can be converted into an input structure first, and then the converted input sequence is input into the pre-training model. Based on the pre-training model, a preset feature extractor is used to encode the input sequence for a continuous number of encoding layers to obtain a vector feature set (each vector feature in the vector feature set has a vector length), that is, the input input sequence can be output through an L-layer (i.e., the number of encoding layers) continuous Transformer module (i.e., the preset feature extractor) to obtain the encoding layer result (i.e., the vector feature set), which is described as follows:

[0069]

[0070] In the embodiment of the present application, the output hidden state of the last encoding layer is where R d represents a vector dimension space with a vector length of d. For a small-version model, L=12 and d=256 can be set; for a base-version model, L=24 and d=768 can be set.

[0071] In step S203, each two adjacent vector features in the vector feature set are integrated to obtain a target vector feature set.

[0072] Optionally, the step of integrating each two adjacent vector features in the vector feature set to obtain a target vector feature set includes: performing a dimension increasing operation on the vector features to obtain dimension-increased vector features; performing convolution on each two adjacent dimension-increased vector features based on a first preset convolution kernel, a first preset step size, and a first preset padding type to obtain new vector features; determining a vector weight of the new vector features based on a second preset convolution kernel, a second preset step size, and a second preset padding type; determining a target vector feature based on the product of the dimension-increased vector features, the vector weight, and the new vector features; and obtaining the target vector feature set based on all target vector features.

[0073] In the embodiment of the application, after the pre-trained model coding, the vector feature set of the input sequence can be obtained, and the Shape (shape) size can be set as [batch_size, max_length, embedding_size], wherein batch_size represents the number of samples selected for training at one time, max_length represents the maximum length, and embedding_size represents the word vector size. Although in the multi-layer Transformer module, the vector representation between each two tokens in the question and the paragraph is modeled through the self-attention mechanism, there is no explicit information fusion between the words in the paragraph and their adjacent words. Especially in the interval extraction type MRC (Machine Reading Comprehension) task, usually according to the correlation between the question features and the paragraph features, the start and end position probability distribution representing the answer interval is output. In the coding process, the question features are integrated into the paragraph word vector representation through the attention mechanism, and finally the probability value is independently output according to the vector representation of each token in the paragraph. In this case, the adjacent context information is crucial to the accurate determination of whether the word belongs to the start or end position. For example, if the position immediately after a certain token is a punctuation mark, the probability of the token as the starting position will be relatively low; if a word is after "located in", the probability of the word as the starting position of a certain "place" type question will be relatively high.

[0074] Therefore, the embodiment proposes a neighbor semantic integration layer (Neighbor Context Integration) to realize the feature fusion function between the adjacent context. The whole NCI layer is very simple, and the input of the layer is X (i.e. the vector feature set), that is, the vector feature set can be integrated by the NCI layer to obtain the target vector feature set. Specifically,

[0075] The vector feature can be dimensioned by formula (1) first to obtain the dimensioned vector feature X, then the dimensioned vector features are convolved by formula (2) according to the first predetermined convolution kernel kernel1 (i.e. operation unit, which can be set according to actual conditions), the first predetermined stride strides (which can be set according to actual conditions) and the first predetermined padding type (which can be selected as the "SAME" type, i.e. can be padded with 0), to obtain the new vector feature, then the vector weight of the new vector feature is determined by formula (3) according to the second predetermined convolution kernel kernel2 (i.e. operation unit, which can be set according to actual conditions), the second predetermined stride strides (which can be set according to actual conditions) and the second predetermined padding type (which can be selected as the "SAME" type, i.e. can be padded with 0), then the target vector feature Y can be determined by formula (4) according to the product of the dimensioned vector feature X, the vector weight G(X) and the new vector feature H(X), and then the target vector feature set can be obtained according to all target vector features.

[0076] X = Expand_dims(X, -1) (1);

[0077] H(X) = Conv2d(X, kernel1, strides, "SAME") (2);

[0078] G(X) = σ(Conv2d(X, kernel2, strides, "SAME")) (3);

[0079] Y = G(X) · H(X) + X (4);

[0080] In formula (1), Expand_dims represents the dimensioning function, -1 represents the last position, the original dimension of the input vector feature X is three-dimensional, and the shape of the output dimensioned vector feature X is X.shape = [batch_size, max_length, embedding_size, 1] after increasing the channel dimension by formula (1).

[0081] In formula (2), the operation Conv2d of two-dimensional convolution is used to integrate the near neighbor context information, the shape of the first predetermined convolution kernel can be set as kernel.shape = [3, 1, 1, 1], the shape of the first predetermined stride can be set as strides = [1, 1, 1, 1], and the first predetermined padding type is padding = "SAME".

[0082] In formula (3), σ represents a gating unit for controlling the proportion of newly added feature weights, and the parameter type is consistent with formula (2).

[0083] In formula (4), the network structure of the residual error can be used to add the newly extracted features to the original features. By comparing the connection mode of the residual error with the connection mode of the Highway before the initial encoding input in the BIDAF (Bi-Directional Attention Flow, a machine reading comprehension model), since the pre-training model can well extract the feature capability, the original feature does not need to be weighted and inhibited, and thus the effect of the residual connection is more obvious.

[0084] In the embodiment, formula (2), (3), and (4) can be stacked multiple times, so as to expand the sensing area of the convolutional layer for the neighboring context, and different optimal stacking times N can be obtained according to different data sets and different backbone network models.

[0085] In step S204, the answer text matching the preset question text is determined from the preset paragraph text based on the target vector feature set, and the answer text is a continuous text in the preset paragraph text.

[0086] Optionally, the step of determining the answer text matching the preset question text from the preset paragraph text based on the target vector feature set includes: performing dimension reduction on each target vector feature in the target vector feature set, and inputting the dimension-reduced target vector feature into a preset output layer; calculating a start position value and an end position value of the target vector feature by using the preset output layer; sorting all start position values and all end position values to obtain a sorting result; determining a maximum start position value and a maximum end position value that satisfy a preset position condition based on the sorting result, wherein the preset position condition is a condition that the start position is located before the end position; and determining the answer text based on a target start position indicated by the maximum start position value and a target end position indicated by the maximum end position value.

[0087] In the embodiment of the application, the answer text matching the preset question text can be determined from the preset paragraph text based on the target vector feature set, and the answer text is a continuous text in the preset paragraph text. Specifically, dimension reduction can be performed on each target vector feature in the target vector feature set by using formula (5), and then the dimension-reduced target vector feature can be input into a preset output layer (that is, the original dimension of the vector feature can be restored by using formula (5), and the new vector feature representation can be continuously input into the network output layer module (i.e., the preset output layer)).

[0088] Y=Squeeze(Y,-1) (5);

[0089] wherein Squeeze represents a dimension reduction function.

[0090] Then, a preset output layer can be used to calculate the start position value and the end position value of the target vector feature (the preset output layer can determine the interval of the answer corresponding to the question in the text sequence, that is, the probability value of each token on the paragraph as the start and end positions can be output). The start position values and the end position values are sorted to obtain a sorting result, and according to the sorting result, the maximum start position value and the maximum end position value that meet the preset position condition (that is, the condition that the start position is located before the end position) are determined (that is, the start position value and the end position value with the maximum probability and the start position located before the end position are selected). According to the target start position indicated by the maximum start position value and the target end position indicated by the maximum end position value, the answer text is determined (that is, the combination with the maximum joint probability and meeting the condition that the start position is located before the end position is selected as the final prediction interval (that is, the answer text)).

[0091] In this embodiment, a full connection layer plus softmax can be used as the preset output layer structure to calculate the start position value and the end position value of the target vector feature. The formula is as follows:

[0092]

[0093]

[0094] wherein w1 and w2 represent preset weight parameters, yi represents the i-th target vector feature, and yj represents the j-th target vector feature in the sum of the j target vector features. i j

[0095] Optionally, before the reduced target vector feature is input into the preset output layer, the method further includes: determining a start cross-entropy loss based on a first loss parameter and a character output probability value corresponding to a real start position; determining a termination cross-entropy loss based on a second loss parameter and a character output probability value corresponding to a real termination position; constructing a loss function based on the start cross-entropy loss and the termination cross-entropy loss; and training the preset output layer based on the loss function.

[0096] In the embodiment of the application, the start cross-entropy loss Lossstart can be determined according to the first loss parameter γ and the character output probability value corresponding to the real start position by formula (8). start The termination cross-entropy loss Lossend can be determined according to the second loss parameter γ and the character output probability value corresponding to the real termination position by formula (9). end ​​​​​Then, the loss function Loss is constructed according to the start cross-entropy loss and the end cross-entropy loss through formula (10). The preset output layer can be trained in a joint training manner based on the loss function (i.e., the preset output layer is trained based on the loss function).

[0097]

[0098]

[0099]

[0100] wherein N represents the sample quantity, and formula (8) and (9) represent the cross-entropy loss of the start and end outputs respectively, and is the predicted probability value of the token output corresponding to the real start and end positions. The gamma parameter can refer to the focal loss (loss function) idea in the field of image target detection, and is used to weaken the influence of simple samples on the optimization direction, so that the model is more inclined to learn difficult samples. In the machine reading comprehension task, the short text answer class is usually easier to learn, and the interval positioning effect of the long answer sample is far lower than that of the short answer sample. Therefore, adding the gamma parameter in the machine reading comprehension task has a certain promoting effect on the model effect.

[0101] Figure 3 is a schematic diagram of an improved machine reading comprehension structure according to an embodiment of the present application, as shown in Figure 3 The semantic integration module is added to the original machine reading comprehension structure (i.e., the input question + paragraph is pre-trained and encoded, the output vector representation is calculated, and the start probability and the end probability of each vector representation are calculated), wherein the semantic integration module structure comprises: performing dimension increasing operation, then performing N times of convolution operation, weight calculation operation and connection operation, and then performing dimension reduction operation on the obtained vector representation to output the final vector representation.

[0102] In the embodiment of the present application, a machine reading comprehension method is provided, which can effectively utilize the related information between the neighboring words, obtain better feature representation on the basis of the original pre-training word and character amount, and further improve the machine reading comprehension level, and achieves the following beneficial effects:

[0103] (1) The semantic relationship between the characters is well integrated by the neighboring semantic integration module according to the difference between the pre-training methods in the Chinese and English fields, and better vector representation is obtained;

[0104] (2) Through the near-neighbor semantic integration mechanism, context information is effectively considered in the machine reading comprehension output answer interval prediction process, so that the model prediction effect is better;

[0105] (3) Compared with some current methods of expanding the scale of pre-training data, redesigning the pre-training structure, introducing external knowledge, etc. to improve the effect of machine reading comprehension, the embodiment can be more convenient, the amount of additional parameters introduced is small, and the pre-training encoding model can be replaced at any time. In addition, the effect of improving machine reading comprehension task on some small-scale parameter models is more obvious.

[0106] The following will be described in detail in conjunction with another embodiment.

[0107] Embodiment Two

[0108] The machine reading comprehension device provided in the embodiment includes a plurality of implementation units, each of which corresponds to each implementation step in the above embodiment one.

[0109] Figure 4 is a schematic diagram of an optional machine reading comprehension device according to an embodiment of the application, as shown in Figure 4 The machine reading comprehension device can include a processing unit 40, an output unit 41, an integration unit 42, and a determination unit 43, wherein

[0110] The processing unit 40 is configured to process the received pre-set question text and pre-set paragraph text to obtain an input sequence, wherein the input sequence includes a plurality of characters.

[0111] The output unit 41 is configured to input the input sequence into a pre-training model to output a vector feature set, wherein the vector feature set includes a plurality of vector features, and each vector feature corresponds to a character.

[0112] The integration unit 42 is configured to integrate every two adjacent vector features in the vector feature set to obtain a target vector feature set.

[0113] The determination unit 43 is configured to determine an answer text matching the pre-set question text from the pre-set paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the pre-set paragraph text.

[0114] The machine reading comprehension device can process the received preset question text and preset paragraph text through the processing unit 40 to obtain an input sequence, input the input sequence into the pre-trained model through the output unit 41, output a vector feature set, integrate each two adjacent vector features in the vector feature set through the integration unit 42 to obtain a target vector feature set, and determine the answer text matched with the preset question text from the preset paragraph text based on the target vector feature set through the determination unit 43. In the embodiment of the application, the preset question text and the preset paragraph text can be processed first, then the obtained input sequence is input into the pre-trained model, then each two adjacent vector features in the output vector feature set are integrated to obtain the target vector feature set, and then the answer text matched with the preset question text is determined from the preset paragraph text according to the target vector feature set. By integrating the relevant information between adjacent characters, the machine reading comprehension level can be improved, and more accurate answers can be obtained, thereby solving the technical problem that the semantic correlation between adjacent characters cannot be integrated in the related art, resulting in low machine reading comprehension accuracy.

[0115] Optionally, the processing unit comprises: a first determination module configured to determine a first symbol, a second symbol, and a third symbol, wherein the first symbol is a start symbol of the input sequence, the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and the preset supplementary text, and the preset supplementary text is an added blank text; a first division module configured to divide the preset question text into a plurality of question characters and divide the preset paragraph text into a plurality of paragraph characters; and a first construction module configured to construct the input sequence based on the first symbol, the second symbol, the third symbol, the question characters, and the paragraph characters.

[0116] Optionally, the machine reading comprehension device further comprises: a second determination module configured to determine an input structure of each character in the input sequence after the input sequence is obtained, wherein the input structure comprises: character semantic encoding, character type encoding, and position encoding; and a third determination module configured to determine a number of encoding layers of the preset feature extractor and a vector length of the vector feature.

[0117] Optionally, the output unit comprises: a first conversion module configured to convert the structure of each character in the input sequence into the input structure; a first input module configured to input the converted input sequence into the pre-trained model; and a first encoding module configured to encode the input sequence using the preset feature extractor for a continuous number of encoding layers based on the pre-trained model to obtain the vector feature set, wherein the length of each vector feature in the vector feature set is the vector length.

[0118] Optionally, the integrating unit comprises: a first dimension increasing module, configured to perform dimension increasing operation on the vector features to obtain dimension-increased vector features; a first convolution module, configured to perform convolution on each two adjacent dimension-increased vector features based on a first preset convolution kernel, a first preset step length and a first preset supplement type to obtain new vector features; a fourth determining module, configured to determine vector weights of the new vector features based on a second preset convolution kernel, a second preset step length and a second preset supplement type; a fifth determining module, configured to determine target vector features based on the product of the dimension-increased vector features, the vector weights and the new vector features; and a first output module, configured to obtain a target vector feature set based on all the target vector features.

[0119] Optionally, the determining unit comprises: a first dimension decreasing module, configured to perform dimension decreasing operation on each target vector feature in the target vector feature set and input the dimension-decreased target vector feature into a preset output layer; a first calculating module, configured to calculate a start position value and an end position value of the target vector feature by using the preset output layer; a first sorting module, configured to sort all the start position values and all the end position values to obtain a sorting result; a sixth determining module, configured to determine a maximum start position value and a maximum end position value satisfying a preset position condition based on the sorting result, wherein the preset position condition is a condition that the start position is located before the end position; and a seventh determining module, configured to determine an answer text based on a target start position indicated by the maximum start position value and a target end position indicated by the maximum end position value.

[0120] Optionally, the machine reading comprehension device further comprises: an eighth determining module, configured to determine a start cross-entropy loss based on a first loss parameter and a character output probability value corresponding to a real start position before the dimension-decreased target vector feature is input into the preset output layer; a ninth determining module, configured to determine an end cross-entropy loss based on a second loss parameter and a character output probability value corresponding to a real end position; a second constructing module, configured to construct a loss function based on the start cross-entropy loss and the end cross-entropy loss; and a first training module, configured to train the preset output layer based on the loss function.

[0121] The machine reading comprehension device described above can further comprise a processor and a memory, and the processing unit 40, the output unit 41, the integrating unit 42, the determining unit 43 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0122] The processor described above comprises a core, and the core retrieves the corresponding program units from the memory. The core can be set to one or more, and the core parameters are adjusted to determine the answer text matching the preset question text from the preset paragraph text based on the target vector feature set.

[0123] The above-mentioned memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0124] The application also provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: processing the received preset question text and preset paragraph text to obtain an input sequence, inputting the input sequence into a pre-trained model to output a vector feature set, integrating each two adjacent vector features in the vector feature set to obtain a target vector feature set, and determining an answer text matching the preset question text from the preset paragraph text based on the target vector feature set.

[0125] According to another aspect of the embodiments of the application, a computer readable storage medium is also provided, which includes a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the machine reading comprehension method described above when the computer program is running.

[0126] According to another aspect of the embodiments of the application, an electronic device is also provided, which includes one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the machine reading comprehension method described above.

[0127] Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) for a machine reading comprehension method according to an embodiment of the application. As shown in Figure 5 , the electronic device can include one or more (as shown in Figure 5 502a, 502b, …, 502n) processors 502 (the processor 502 can include but is not limited to a machine reading comprehension device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 504 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 5 the structure shown is only schematic, and it does not limit the structure of the above-mentioned electronic device. For example, the electronic device can include more or fewer components than those shown in Figure 5 , or have a different configuration from that shown in Figure 5 .

[0128] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0129] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0131] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0132] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0133] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions which essentially contribute to the prior art can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The above-mentioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0134] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.< / sep> < / sep> < / cls> < / sep> < / cls> < / sep> < / sep> < / cls>

Claims

1. A machine reading comprehension method, characterized in that, The method comprises the following steps: processing the received preset question text and preset paragraph text to obtain an input sequence, wherein the input sequence comprises a plurality of characters; inputting the input sequence into a BERT pre-training model to output a vector feature set, wherein the vector feature set comprises a plurality of vector features, and each vector feature corresponds to a character; integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set; determining an answer text matching the preset question text from the preset paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the preset paragraph text; wherein the step of integrating every two adjacent vector features in the vector feature set to obtain a target vector feature set comprises: performing dimensionality increasing operation on the vector features to obtain dimensionality increased vector features; performing convolution on every two adjacent dimensionality increased vector features based on a first preset convolution kernel, a first preset step and a first preset supplementary type to obtain new vector features; determining vector weights of the new vector features based on a second preset convolution kernel, a second preset step and a second preset supplementary type; determining target vector features based on the product of the dimensionality increased vector features, the vector weights and the new vector features; and obtaining the target vector feature set based on all the target vector features; wherein the step of determining an answer text matching the preset question text from the preset paragraph text based on the target vector feature set comprises: performing dimensionality reduction operation on each target vector feature in the target vector feature set and inputting the dimensionally reduced target vector features into a preset output layer; calculating starting position values and ending position values of the target vector features using the preset output layer; sorting all the starting position values and all the ending position values to obtain a sorting result; determining a maximum starting position value and a maximum ending position value satisfying a preset position condition based on the sorting result, wherein the preset position condition is a condition that the starting position is located before the ending position; and determining the answer text based on a target starting position indicated by the maximum starting position value and a target ending position indicated by the maximum ending position value.

2. The machine reading comprehension method of claim 1, wherein, The step of processing the received preset question text and preset paragraph text to obtain an input sequence comprises: determining a first symbol, a second symbol and a third symbol, wherein the first symbol is a start symbol of the input sequence, the second symbol is used to distinguish the preset question text and the preset paragraph text, and the third symbol is used to distinguish the preset paragraph text and a preset supplementary text, which is an added blank text; dividing the preset question text into a plurality of question characters and dividing the preset paragraph text into a plurality of paragraph characters; constructing the input sequence based on the first symbol, the second symbol, the third symbol, the question characters and the paragraph characters.

3. The machine reading comprehension method of claim 1, wherein, After obtaining the input sequence, the method further comprises: determine an input structure of each of the characters in the input sequence, wherein the input structure comprises a character semantic code, a character type code, and a position code; determine a number of encoding layers of a preset feature extractor and a vector length of the vector feature.

4. The machine reading comprehension method of claim 3, wherein, The step of inputting the input sequence into a BERT pre-training model to output a vector feature set comprises: converting the structure of each of the characters in the input sequence into the input structure; inputting the converted input sequence into the BERT pre-training model; based on the BERT pre-training model, using the preset feature extractor to encode the input sequence for a continuous number of encoding layers to obtain the vector feature set, wherein the length of each of the vector features in the vector feature set is the vector length.

5. The machine reading comprehension method of claim 1, wherein, Before inputting the target vector feature after dimension reduction into a preset output layer, further comprising: determining a start cross-entropy loss based on a first loss parameter and a character output probability value corresponding to a real start position; determining a termination cross-entropy loss based on a second loss parameter and a character output probability value corresponding to a real termination position; constructing a loss function based on the start cross-entropy loss and the termination cross-entropy loss; training the preset output layer based on the loss function. 6.A machine reading comprehension device employing the machine reading comprehension method according to claim 1, characterized in that, comprise: a processing unit configured to process a received preset question text and a preset paragraph text to obtain an input sequence, wherein the input sequence comprises a plurality of characters; an output unit configured to input the input sequence into a BERT pre-training model to output a vector feature set, wherein the vector feature set comprises a plurality of vector features, and each of the vector features corresponds to one of the characters; an integration unit configured to integrate every two adjacent vector features in the vector feature set to obtain a target vector feature set; a determination unit configured to determine an answer text matching the preset question text from the preset paragraph text based on the target vector feature set, wherein the answer text is a continuous text in the preset paragraph text.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein the computer program controls the device where the computer-readable storage medium is located to perform the machine reading comprehension method of any one of claims 1 to 5 when the computer program is running.

8. An electronic device, comprising: comprise one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the machine reading comprehension method of any one of claims 1 to 5.

Images