Text reading comprehension method and device based on article difference perception representation

Abstract

The application discloses a text reading comprehension method and device based on article difference perception representation, a storage medium and an electronic equipment, belongs to the field of natural language processing and artificial intelligence, and aims to solve the technical problems of how to effectively utilize article information to improve the accuracy of answer selection and how to realize effective matching between a question and options, thereby improving the prediction accuracy of a text reading comprehension system, and adopts the technical scheme that: ① a text reading comprehension method based on article difference perception representation comprises the following modules: a pre-training embedding representation module, a feature filtering module, an article difference perception representation interaction module and a label prediction module. ② a text reading comprehension device based on article difference perception representation comprises a text reading comprehension data set acquisition unit, a text reading comprehension model construction unit and a text reading comprehension model training unit.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and specifically to a text reading comprehension method and apparatus based on text difference perception representation. Background Technology

[0002] Machine text reading comprehension refers to the task of a computer answering related questions based on given text information. It falls under the category of Natural Language Processing (NLP) and is one of its newest and most popular research areas. It can be divided into four sub-tasks: fill-in-the-blank text reading comprehension, extractive text reading comprehension, text reading comprehension, and generative text reading comprehension. In text reading comprehension tasks, questions and options generally do not appear directly in the text. Therefore, to select the correct answer, the computer needs strong reasoning and summarizing abilities. For text reading comprehension tasks, a question and its four corresponding options are each related to a specific part of the text, and the content pointed to by the correct answer must be consistent with the content pointed to by the question. Therefore, how to find the corresponding parts of the text for each question and each option, and how to compare the similarity between the corresponding parts of the text for each question and each option, is a problem that urgently needs to be solved. To date, existing methods have not substantially solved this problem. Therefore, text reading comprehension is a very challenging task.

[0003] To effectively utilize the information in a text, most existing methods integrate this information into questions or answer choices in various ways, then perform an interactive matching process between the questions and choices, ultimately selecting the option with the highest match to the question as the correct answer. While this approach is simple and easy to implement, the problem lies in the fact that questions and answer choices are short sentences containing limited information. Integrating text information into these sentences in a fusion manner leads to the underutilization of a significant amount of textual information, and may even result in information loss. This undoubtedly reduces the prediction accuracy of text reading comprehension models.

[0004] To address the shortcomings of existing methods, this invention proposes a text reading comprehension method and apparatus based on text difference-aware representation. This method can effectively utilize text information to improve the accuracy of answer selection and can effectively prevent the loss of text information. By achieving effective matching between questions and options, it improves the prediction accuracy of the text reading comprehension system. Summary of the Invention

[0005] The technical objective of this invention is to provide a text reading comprehension method, apparatus, storage medium, and electronic device based on article difference-aware representation, in order to address how to effectively utilize article information to improve the accuracy of answer selection and how to achieve effective matching between questions and options, thereby improving the prediction accuracy of the text reading comprehension system.

[0006] The technical objective of this invention is achieved as follows: a text reading comprehension method based on article difference-aware representation, comprising the following steps:

[0007] S1. Obtain a text reading comprehension dataset: Download a publicly available text reading comprehension dataset from the internet or create your own dataset;

[0008] S2. Constructing a text reading comprehension model: Constructing a text reading comprehension model based on article difference perception representation;

[0009] S3. Train the text reading comprehension model: Train the text reading comprehension model constructed in step S2 on the text reading comprehension training dataset obtained in step S1.

[0010] Preferably, the specific steps for constructing the text reading comprehension model in step S2 are as follows:

[0011] S201, Constructing the Input Module

[0012] For each data point in the dataset, the text sequence is denoted as context; the question sequence as query; and all candidate options as response. Based on the correct answer, the label for each data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. These three text sequences and the label together constitute one input data point.

[0013] S202, Constructing pre-trained embedded representation modules

[0014] The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. The specific implementation is shown in the following formula:

[0015]

[0016]

[0017]

[0018] In this equation (1.1), the pre-trained language model BERT is used to embed and encode the input article sequence context, and the subscript n represents the number of network layers within BERT. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates an embedded representation of the options;

[0019] S203, Constructing the Feature Filtering Module

[0020] Embedded representations of articles, questions, and options using adaptive average pooling operations. and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: and

[0021] S204. Construct an article difference perception representation interaction module.

[0022] The module structure is as follows: Figure 5 As shown; its received feature filtering module unit outputs the article pooling feature representation Question Pooling Feature Representation and option pooling feature representation Then, two different article difference representations are obtained using two methods, and finally concatenated to obtain the final article difference representation, which is denoted as . And then pass it to the label prediction module.

[0023] S205, Constructing a Label Prediction Module

[0024] The article difference representation obtained in step S204 will be used as the input of this module. It will be processed by a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option as the correct answer. This module will predict the candidate option with the highest probability as the correct answer.

[0025] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S205 predicts which of the options is the correct answer.

[0026] Preferably, the specific steps for constructing the article difference-aware representation interaction module in step S204 are as follows:

[0027] S20401. Obtain two article representations: Use two methods to obtain two different article difference representations;

[0028] S2040101. Obtaining the Question-Aware Article Representation: The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the question pooling feature representation to obtain the question-aware article representation, i.e. The specific implementation is shown in the following formula:

[0029]

[0030]

[0031]

[0032] Formula (2.1) represents the interactive computation between the article pooling feature representation and the question pooling feature representation achieved through the dot product multiplication operation. and These represent the pooling feature representations for articles and questions, respectively. The dot product operation is represented; Formula (2.2) represents the attention weights obtained through normalization, where i and i' represent the indices of the corresponding elements in the input tensor, and I kl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (2.1); formula (2.3) indicates that the attention weights obtained by formula (2.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the question-aware article representation; I represents and the number of elements in α;

[0033] S2040102. Obtaining the Option-Aware Article Representation: The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the option pooling feature representation to obtain the option-aware article representation, i.e. The specific implementation is shown in the following formula:

[0034]

[0035]

[0036]

[0037] Formula (3.1) represents the interactive computation between the article pooling feature representation and the option pooling feature representation through the dot product multiplication operation. and These represent the article pooling feature representation and the option pooling feature representation, respectively. The dot product operation is represented; Formula (3.2) represents the attention weights obtained through normalization, where i and i' represent the indices of the corresponding elements in the input tensor, and I rl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (3.1); formula (3.3) indicates that the attention weights obtained by formula (3.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the option-aware article representation; I represents and the number of elements in α;

[0038] S20402. Obtain two difference representations: In order to capture the difference between the two article representations, two difference feature capture methods are used respectively;

[0039] S2040201. Obtaining the difference representation 1: Add a dimension to each of the two article representations, and then perform a join operation on the two article representations on the newly added dimension to obtain the question-and-choice-aware article representation, i.e. The specific implementation is shown in the following formula:

[0040]

[0041]

[0042]

[0043] Wherein, formula (4.1) represents the article representation for question awareness using the Reshape operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Equation (4.2) represents the option-aware article representation using the Reshape operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Equation (4.3) indicates that the connection operation is used to... and By making connections on the newly added dimensions, we obtain the article representation for question selection perception. Its shape is (batch_size, time_steps, output_dimension, 2);

[0044] Then, a 2D CNN is used to extract its features, followed by mapping it through a 1-dimensional fully connected network, and finally, its dimensions are reshaped to obtain the difference representation 1, i.e. The specific implementation is shown in the following formula:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] In this equation (5.1), the result of convolving the f-th convolutional kernel with a specific region of the question-and-answer document representation and then mapping it using the ReLU function is given, where [x1, y1] represents the size of the convolutional kernel. This represents the weight matrix of the f-th convolutional kernel, where i and j represent the x and y coordinates of the convolutional region, and m... l and m h Let i:i+x1-1, j:j+y1-1 represent the length and height of the text representation in the question selection perception, and let j:j+y1-1 represent the convolutional region. This represents the bias matrix of the f-th convolutional kernel. Let represent the convolution result of the f-th convolutional kernel in the region i:i+x1-1, j:j+y1-1; Formula (5.2) represents the integration of the convolution results of the f-th convolutional kernel in each region to obtain the final convolution result of the f-th convolutional kernel, where s x1 and s y1 Indicates the horizontal convolution stride and the vertical convolution stride. Let f represent the final convolution result of the f-th convolution kernel; Formula (5.3) represents combining the final convolution results of n convolution kernels to obtain the final convolution result of the network for the question-and-answer perception article representation, with shape (batch_size, time_steps, output_dimension, n); Formula (5.4) represents using a fully connected network of dimension 1 to map the final convolution result of the question-and-answer perception article representation to obtain the mapping result, with shape (batch_size, time_steps, output_dimension, 1); Formula (5.5) represents reshaping the shape of the mapping result, with the reshaping mapping result having shape (batch_size, time_steps, output_dimension).

[0051] S2040202. Obtaining Difference Representation 2: Perform a difference operation on the two article representations and take their absolute values ​​to obtain Difference Representation 2, i.e. The specific implementation is shown in the following formula:

[0052]

[0053] S20403. Obtain the perceptual representation of text differences: Connect the two difference representations to obtain the perceptual representation of text differences, i.e. This information is then passed to the label prediction module; see the following formula for details:

[0054]

[0055] As a preferred embodiment, the specific steps for training the text reading comprehension model in step S3 are as follows:

[0056] S301. Constructing the loss function

[0057] This invention uses cross-entropy as the loss function;

[0058] S302, Constructing the optimization function

[0059] After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to their default values ​​in PyTorch.

[0060] When a text reading comprehension model has not yet been trained, it needs to be further trained to optimize its parameters; once the text reading comprehension model has been trained, it can predict which of the candidate options is the correct one.

[0061] A text reading comprehension device based on text difference-aware representation, the device comprising:

[0062] The text reading comprehension dataset acquisition unit is used to download publicly available text reading comprehension datasets from the internet.

[0063] The text reading comprehension model building unit is used to construct a pre-trained embedding representation module, a feature filtering module, an article difference-aware representation interaction module, and a label prediction module, thereby constructing a text reading comprehension model.

[0064] The text reading comprehension model training unit is used to construct the loss function and optimization function to complete the answer selection.

[0065] Preferably, the text reading comprehension model building unit includes:

[0066] The input module unit is responsible for preprocessing the raw dataset to construct the input data.

[0067] The pre-trained embedding representation module is responsible for using a pre-trained language model to perform embedding encoding operations on the input data, thereby obtaining the embedding representations of articles, questions, and options in the input data.

[0068] The feature filtering module is responsible for performing feature filtering operations on the embedded representations of articles, questions, and options using adaptive average pooling to obtain the corresponding feature pooling representations.

[0069] The article difference perception representation interaction module unit is responsible for receiving the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module unit. Then, it uses two methods to obtain two different article difference representations, and finally concatenates them to obtain the final article difference representation.

[0070] The label prediction module is responsible for determining which candidate option is the correct one based on the article's difference representation.

[0071] The text reading comprehension model training unit also includes:

[0072] The loss function unit is responsible for calculating the error between the prediction result and the actual data using the cross-entropy loss function.

[0073] The optimization function unit is responsible for training and adjusting the parameters during model training to reduce prediction error.

[0074] A storage medium storing a plurality of instructions, which are loaded by a processor to execute the steps of the text reading comprehension method based on article difference-aware representation described above.

[0075] An electronic device, the electronic device comprising:

[0076] The aforementioned storage medium; and

[0077] A processor for executing instructions in the storage medium.

[0078] The text reading comprehension method and apparatus based on article difference perception representation of the present invention have the following advantages:

[0079] (1) This invention can capture and utilize the semantic information contained in the text by pre-training the embedded representation module, making the captured semantic features richer and more accurate;

[0080] (2) The present invention can effectively filter out useless information and reduce sequence length through the feature filtering module, thereby reducing the cost of training the model and improving the efficiency of training.

[0081] (3) The present invention uses an article difference perception and interaction module to effectively utilize article information to improve the accuracy of answer selection;

[0082] (4) This invention uses an article difference perception representation interaction module to achieve effective matching between questions and options, thereby improving the prediction accuracy of the text reading comprehension system;

[0083] (5) The method and apparatus proposed in this invention, combined with text difference perception representation, can effectively improve the accuracy of sequence interaction in text reading comprehension. Attached Figure Description

[0084] The invention will be further described below with reference to the accompanying drawings.

[0085] Figure 1 A flowchart of a text reading comprehension method based on article difference-aware representation.

[0086] Figure 2 Flowchart for building a text reading comprehension model

[0087] Figure 3 Flowchart for training a text reading comprehension model

[0088] Figure 4 A flowchart of a text reading comprehension device based on text difference-aware representation.

[0089] Figure 5 A structural diagram of the article difference perception interaction module.

[0090] Figure 6 This is a schematic diagram of the framework of a text reading comprehension model based on article difference-aware representation. Detailed Implementation

[0091] The following detailed description of the text reading comprehension method, apparatus, storage medium, and electronic device based on article difference perception representation of the present invention, with reference to the accompanying drawings and specific embodiments, is provided.

[0092] Example 1: A model framework for text reading comprehension based on article difference-aware representation.

[0093] The overall model framework structure of this invention is as follows: Figure 6 As shown. By Figure 6 As can be seen, the main framework of this invention includes a pre-trained embedding representation module, a feature filtering module, an article difference-aware representation interaction module, and a label prediction module. The pre-trained embedding representation module performs embedding encoding on the input article sequence, question sequence, and option sequence to obtain their respective embedded encoding representations, which are then passed to the model's feature filtering module. The feature filtering module performs average pooling on the three embedded encoding representations to obtain filtered feature representations, namely, article pooling feature representation, question pooling feature representation, and option pooling feature representation, which are then passed to the article difference-aware representation interaction module. The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation with the question pooling feature representation and the option pooling feature representation, respectively, to obtain question-aware article representations and option-aware article representations. To capture the differences between the two article representations, two difference feature capture methods are used: First, a two-dimensional representation of the two article representations is obtained through a concatenation operation, and then a 2D CNN is used to extract features from them, resulting in difference representation 1; second, a difference operation is performed on the two article representations, and their absolute values ​​are taken, resulting in difference representation 2. Finally, the two difference representations are concatenated to obtain the article difference-aware representation, which is then passed to the label prediction module. The label prediction module maps the article difference-aware representation to a floating-point value in a specified range, using it as the match degree between the option and the question; then it compares the match degrees of different options and selects the option with the highest match degree as the correct answer.

[0094] Example 2: Article Difference Perception Representation Interaction Module.

[0095] The article difference perception representation interaction module receives the article pooling feature representation, question pooling feature representation, and option pooling feature table output by the feature filtering module unit. Then, it uses two methods to obtain two different article difference representations and finally concatenates them to obtain the final article difference representation.

[0096] Specifically, the implementation process of this module is as follows:

[0097] The first step is to obtain two types of article representations: The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the question pooling feature representation to obtain the question-aware article representation, i.e. The specific implementation is shown in the following formula:

[0098]

[0099]

[0100]

[0101] Formula (1.1) represents the interactive computation between the article pooling feature representation and the question pooling feature representation through the dot product multiplication operation. and These represent the pooling feature representations for articles and questions, respectively. The dot product operation is represented; Formula (1.2) represents the attention weights obtained through normalization, where i and i' represent the indices of the corresponding elements in the input tensor, and I kl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (1.1); formula (1.3) indicates that the attention weights obtained by formula (1.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the question-aware article representation; I represents and the number of elements in α;

[0102] For example, in PyTorch, the code implementation described above would be as follows:

[0103] q_p_dot=tf.expand_dims(context_pool,axis=1)*tf.expand_dims(query_pool,axis=2)

[0104] sd1=tf.multiply(tf.tanh(K.dot(q_p_dot,self.Wd)),self.vd)

[0105] sd2=tf.squeeze(sd1,axis=-1)

[0106] ad = tf.nn.softmax(sd2)

[0107] qdq=K.batch_dot(ad,context_pool)

[0108] Where q_p_dot represents the click results of the article pooling feature representation and the question pooling feature representation; ad represents the attention weight obtained through normalization operation; and qd represents the article representation for question perception.

[0109] The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the option pooling feature representation to obtain the option-aware article representation, i.e. The specific implementation is shown in the following formula:

[0110]

[0111]

[0112]

[0113] Formula (2.1) represents the interactive computation between the article pooling feature representation and the option pooling feature representation through the dot product multiplication operation. and These represent the article pooling feature representation and the option pooling feature representation, respectively. The dot product operation is represented; Formula (2.2) represents the attention weights obtained through normalization, where i and i' represent the indices of the corresponding elements in the input tensor, and I rl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (2.1); formula (2.3) indicates that the attention weights obtained by formula (2.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the option-aware article representation; I represents and the number of elements in α;

[0114] For example, in PyTorch, the code implementation described above would be as follows:

[0115] q_p_dot=tf.expand_dims(context_pool,axis=1)*tf.expand_dims(response_pool,axis=2)

[0116] sd1=tf.multiply(tf.tanh(K.dot(q_p_dot,self.Wd)),self.vd)

[0117] sd2=tf.squeeze(sd1,axis=-1)

[0118] ad = tf.nn.softmax(sd2)

[0119] qdr=K.batch_dot(ad,context_pool)

[0120] Where q_p_dot represents the click result of the article pooling feature representation and the option pooling feature representation; ad represents the attention weight obtained through normalization operation; and qd represents the option-aware article representation.

[0121] The second step is to obtain two difference representations: In order to capture the difference between the two article representations, two different difference feature capture methods are used respectively.

[0122] Obtaining the difference representation 1: Add a dimension to each of the two article representations, and then perform a join operation on the two article representations along the newly added dimension to obtain the question-selection-aware article representation, i.e. The specific implementation is shown in the following formula:

[0123]

[0124]

[0125]

[0126] Wherein, formula (3.1) represents the article representation for question awareness using the Reshape operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Equation (3.2) represents the option-aware article representation using the Reshape operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Equation (3.3) indicates that the connection operation is used to... and By making connections on the newly added dimensions, we obtain the article representation for question selection perception. Its shape is (batch_size, time_steps, output_dimension, 2);

[0127] For example, in PyTorch, the code implementation described above would be as follows:

[0128] cq=torch.reshape(qdq,(qdq.view(0),qdq.view(1),qdq.view(2),1))

[0129] cr=torch.reshape(qdr,(qdr.view(0),qdr.view(1),qdr.view(2),1))

[0130] cqr=torch.cat((cq,cr),dim=-1)

[0131] Wherein, cq represents the article representation with added dimensions for question perception; cr represents the article representation with added dimensions for option perception; and cqr represents the article representation with added dimensions for question-option perception.

[0132] Then, a 2D CNN is used to extract its features, followed by mapping it through a 1-dimensional fully connected network, and finally, its dimensions are reshaped to obtain the difference representation 1, i.e. The specific implementation is shown in the following formula:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] Wherein, formula (4.1) represents the result of the f-th convolutional kernel convolving a specific region of the question-and-answer document representation and then mapping it using the ReLU function, where [x1, y1] represents the size of the convolutional kernel. This represents the weight matrix of the f-th convolutional kernel, where i and j represent the x and y coordinates of the convolutional region, and m... l and m h Let i:i+x1-1, j:j+y1-1 represent the length and height of the text representation in the question selection perception, and let j:j+y1-1 represent the convolutional region. This represents the bias matrix of the f-th convolutional kernel. Let represent the convolution result of the f-th convolutional kernel in the region i:i+x1-1, j:j+y1-1; Formula (4.2) represents the integration of the convolution results of the f-th convolutional kernel in each region to obtain the final convolution result of the f-th convolutional kernel, where s x1 and s y1 Indicates the horizontal convolution stride and the vertical convolution stride. Let f represent the final convolution result of the f-th convolution kernel; Formula (4.3) represents combining the final convolution results of n convolution kernels to obtain the final convolution result of the network layer for the question-and-answer perception article representation, with shape (batch_size, time_steps, output_dimension, n); Formula (4.4) represents using a fully connected network of dimension 1 to map the final convolution result of the question-and-answer perception article representation to obtain the mapping result, with shape (batch_size, time_steps, output_dimension, 1); Formula (4.5) represents reshaping the shape of the mapping result, with the reshaping mapping result having shape (batch_size, time_steps, output_dimension).

[0139] For example, in PyTorch, the code implementation described above would be as follows:

[0140] self.conv=Conv2d(in_channels=2, out_channels=16, kernel_size=1, stride=1)

[0141] cqr_conv = self.conv(cqr)

[0142] self.classifier=nn.Linear(config.hidden_size*16,1)

[0143] cqr_conv_re=torch.reshape(cqr_conv,(cqr_conv.view(0),cqr_conv.view(1),cqr_conv.view(2)))

[0144] Where cqr_conv represents the text representation after the question-and-selective perception is processed by a two-dimensional convolution operation; cqr_conv_re represents the difference representation 1; 2 represents the input channel dimension, which is the last dimension in the question-and-selective perception text representation; and 16 represents the number of convolution kernels.

[0145] Obtaining Difference Representation 2: Perform a difference operation on the two article representations and take their absolute values ​​to obtain Difference Representation 2, i.e. The specific implementation is shown in the following formula:

[0146]

[0147] For example, in PyTorch, the code implementation described above would be as follows:

[0148] cqr2 = torch.abs(qdq - qdr)

[0149] Where cqr2 represents difference representation 2.

[0150] Step 3: Obtain the text difference perception representation: Connect the two difference representations to obtain the text difference perception representation, i.e. This information is then passed to the label prediction module; see the following formula for details:

[0151]

[0152] For example, in PyTorch, the code implementation described above would be as follows:

[0153] cqr_z=torch.cat((cqr,cqr2),dim=-1)

[0154] Here, cqr_z is the article difference perception representation.

[0155] Example 3: Specific steps of the invention.

[0156] As attached Figure 1 As shown, the present invention provides a text reading comprehension method based on article difference-aware representation, which includes the following steps:

[0157] S1. Obtain a text reading comprehension dataset: Download a publicly available text reading comprehension dataset from the internet;

[0158] S2. Constructing a text reading comprehension model: Constructing a text reading comprehension model using text difference perception representation;

[0159] S3. Train the text reading comprehension model: Train the text reading comprehension model constructed in step S2 on the text reading comprehension training dataset obtained in step S1.

[0160] S1. Obtain the text reading comprehension dataset.

[0161] Download publicly available text reading comprehension datasets from the internet or create your own dataset.

[0162] For example, there are many publicly available text reading comprehension datasets online, such as RACE. The data format in this dataset is as follows:

[0163]

[0164]

[0165] S2. Constructing a text reading comprehension model

[0166] The process of building a text reading comprehension model is as follows: Figure 2 As shown, the main operations are to construct an input module, a pre-trained embedding representation module, a feature filtering module, an article difference-aware representation interaction module, and a label prediction module.

[0167] S201, Constructing the Input Module

[0168] For each data point in the dataset, the text sequence is denoted as context; the question sequence is denoted as query; and all candidate options are denoted as response. Based on the correct answer, the label for that data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. The three text sequences and the label together constitute one input data point.

[0169] For example, using the data shown in S1 as a sample, we can construct one input data entry. The result is shown below:

[0170] (context:They like using the Internet.They have lots of pocket moneyto spend.And they spend a higher proportion of it online than the rest ofus.Teenagers are just the sort of people an online seller is interested in,and the things they want to buy-games,CDs and clothing-are easily sold onthe Web.But paying online is a tricky business for consumers who are tooyoung to own credit cards.Most have to use a parent’s card.They want afacility that allows them to spend money.That may come sooner than theythink:new ways to take pocket money into cyber space are coming out rapidlyon both sides of the Atlantic.If successful,these products can stimulateonline sales.In general,teenagers spend huge amounts:$153bn(billion)in the USlast year and￡20bn annually in the UK.Most teenagers have access to theInternet at home or at school-88percent in the US,69percent in theUK.According to the Jupiter Research,one in eight of those with Internetaccess has bought something online-mainly CDs and books.In most cases,parents pay for these purchases with credit cards,an arrangement that isoften unsatisfactory for them and their children.Pressing parents to spendonline is less productive than pressing on the high street.They are morelikely to ask“Why?”if you ask to spend some money online.One way to helpteenagers change notes and coins into cybercash is through prepaid cards suchas InternetCash in the US and Smart cards in the UK.Similar to those for pay-as-you-go mobile telephones,they are sold in amounts such as￡20or$50with aconcealed 14-digit number that can be used to load the cash into an onlineaccount.,query:According to the passage,which of the following statements isTRUE?,response:More than half of the teenagers in the US and the UK haveInternet access.Teenagers pay for goods online with their own creditcards.Most teenagers in the US and the UK have bought somethingonline.Teenagers found it easier to persuade parents to buy online than inashop.,1000).

[0171] S202, Constructing pre-trained embedded representation modules

[0172] The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. The specific implementation is shown in the following formula:

[0173]

[0174]

[0175]

[0176] In this equation (1.1), the pre-trained language model BERT is used to embed and encode the input article sequence context, and the subscript n represents the number of network layers within BERT. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates the embedded representation of the option.

[0177] For example, when this invention is implemented on the RACE dataset, n is set to 12 and t is set to 1. In PyTorch, the code implementation described above is as follows:

[0178] context_embed,_=BERT_n(context)

[0179] query_embed, _ = BERT_t(query)

[0180] response_embed,_=BERT_t(response)

[0181] Here, context_embed is the embedded representation of the article, query_embed is the embedded representation of the question, and response_embed is the embedded representation of the option.

[0182] S203, Constructing the Feature Filtering Module

[0183] Embedded representations of articles, questions, and options using adaptive average pooling operations. and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: and

[0184] For example, in PyTorch, the code implementation described above would be as follows:

[0185] self.context_pool_adapt=AdaptiveAvgPool1d(output_size=64)

[0186] context_pool=torch.transpose(self.context_pool_adapt(torch.transpose(context_embed,dim0=1,dim1=2)),dim0=1,dim1=2)

[0187] self.qr_pool_adapt=AdaptiveAvgPool1d(output_size=15)

[0188] query_pool=torch.transpose(self.qr_pool_adapt(torch.transpose(query_embed,dim0=1,dim1=2)),dim0=1,dim1=2)

[0189] response_pool=torch.transpose(self.qr_pool_adapt(torch.transpose(response_embed,dim0=1,dim1=2)),dim0=1,dim1=2)

[0190] In this code, `AdaptiveAvgPool1d()` represents the adaptive average pooling operation, `output_size` represents the output dimension of the input data after processing by this operation, and in this invention, the output dimension for the article representation is set to 64, and the output dimension for the question and option is set to 15. `torch.transpose` represents the transpose operation, `context_pool` represents the article pooled feature representation obtained after adaptive average pooling of the article's embedding representation, `query_pool` represents the question pooled feature representation obtained after adaptive average pooling of the question's embedding representation, and `response_pool` represents the option pooled feature representation obtained after adaptive average pooling of the option's embedding representation.

[0191] S204. Construct an article difference perception representation interaction module.

[0192] The module structure is as follows: Figure 5 As shown; its received feature filtering module unit outputs the article pooling feature representation Question Pooling Feature Representation and option pooling feature representation Then, two different article difference representations are obtained using two methods, and finally concatenated to obtain the final article difference representation, which is denoted as .

[0193] S205, Constructing a Label Prediction Module

[0194] The article difference representation obtained in step S204 will be used as input to this module. It will be processed by a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option being the correct answer. This module will predict the candidate option with the highest probability as the correct answer.

[0195] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S205 predicts which of the options is the correct answer.

[0196] S3, Training the Text Reading Comprehension Model

[0197] The text reading comprehension model constructed in step S2 is trained on the text reading comprehension training dataset obtained in step S1. The process is as follows: Figure 3 As shown.

[0198] S301. Constructing the loss function

[0199] This invention uses cross-entropy as the loss function.

[0200] For example, in PyTorch, the code implementation described above would be as follows:

[0201] loss_fct = CrossEntropyLoss()

[0202] loss=loss_fct(logits.view(-1,self.num_labels),labels.view(-1))

[0203] Here, labels are the true labels, and logits are the correct probabilities of the model output.

[0204] S302, Constructing the optimization function

[0205] After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to their default values ​​in PyTorch.

[0206] For example, in PyTorch, the code implementation described above would be as follows:

[0207] optimizer=BertAdam(optimizer_grouped_parameters,lr=2e-5)

[0208] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in this model.

[0209] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S205 predicts which of the options is the correct answer.

[0210] Example 4:

[0211] Based on a storage medium of Embodiment 1, a plurality of instructions are stored therein, which are loaded by a processor to execute the steps of the text reading comprehension method based on article difference-aware representation of Embodiment 1.

[0212] Example 5:

[0213] An electronic device based on Embodiment 4 includes: a storage medium of Embodiment 4; and a processor for executing instructions in the storage medium.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; the present invention has been described in detail with reference to the foregoing embodiments. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text reading comprehension method based on text difference perception representation, characterized in that, The method includes the following steps: S1. Obtain a text reading comprehension dataset: Download a publicly available text reading comprehension dataset from the internet or create your own dataset; S2. Constructing a text reading comprehension model: Constructing a text reading comprehension model based on article difference perception representation; S3. Training the text reading comprehension model: Train the text reading comprehension model constructed in step S2 on the text reading comprehension training dataset obtained in step S1; The specific steps for constructing the text reading comprehension model in step S2 are as follows: S201, Constructing the Input Module For each data point in the dataset, the text sequence is denoted as context; the question sequence as query; and all candidate options as response. Based on the correct answer, the label for each data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. These three text sequences and the label together constitute one input data point. S202, Constructing pre-trained embedded representation modules The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: , and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. See the following formula for specific implementation: （1.1） （1.2） （1.3） In this context, formula (1.1) represents the embedding encoding of the input article sequence context using the pre-trained language model BERT, and the subscript n represents the number of network layers within BERT used. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates an embedded representation of the options; S203, Constructing the Feature Filtering Module Embedded representations of articles, questions, and options using adaptive average pooling operations. , and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: , and ; S204. Construct an article difference perception representation interaction module. Receive the article pooling feature representation output by the feature filtering module Question pooling feature representation and option pooling feature representation Then, two different article difference representations are obtained using two strategies, and finally concatenated to obtain the final article difference representation, which is denoted as... ; S205, Constructing a Label Prediction Module The article difference representation obtained in step S204 will be used as the input of this module. It will be processed by a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option as the correct answer. This module will predict the candidate option with the highest probability as the correct answer. If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S205 predicts which of the options is the correct answer.

2. The text reading comprehension method based on article difference-aware representation according to claim 1, characterized in that, The construction process of the article difference perception representation interaction module in step S204 is as follows: S20401. Obtain two article representations: Use two strategies to obtain two different article representations; S2040101. Obtaining the Question-Aware Article Representation: The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the question pooling feature representation to obtain the question-aware article representation, i.e. The specific implementation is shown in the following formula: （2.1） ,i=1,2,…, I kl (2.2) （2.3） Formula (2.1) represents the interactive computation between the article pooling feature representation and the question pooling feature representation achieved through dot product multiplication. and These represent the pooling feature representations for articles and questions, respectively. The formula (2.2) represents the dot product multiplication operation; it represents the attention weights obtained through normalization, where i and i' represent the indices of the elements in the corresponding input tensors. kl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (2.1); formula (2.3) indicates that the attention weights obtained by formula (2.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the question-aware article representation; I represents and The number of elements in; S2040102. Obtaining the Option-Aware Article Representation: The article difference-aware representation interaction module performs attention calculations on the article pooling feature representation and the option pooling feature representation to obtain the option-aware article representation, i.e. The specific implementation is shown in the following formula: （3.1） ,i=1,2,…,I rl (3.2) （3.3） Formula (3.1) represents the interactive computation between the article pooling feature representation and the option pooling feature representation through the dot product multiplication operation. and These represent the article pooling feature representation and the option pooling feature representation, respectively. The dot product operation is represented; formula (3.2) represents the attention weights obtained through normalization, where i and i' represent the indices of the corresponding elements in the input tensor, and I rl Indicates the input tensor The number of elements in the formula, and the meanings of other symbols are the same as in formula (3.1); formula (3.3) indicates that the attention weights obtained by formula (3.2) are used to complete the feature selection of the article pooling feature representation, thereby obtaining the option-aware article representation; I represents and The number of elements in; S20402. Obtain two article difference representations: In order to capture the difference between the two article representations, two difference feature capture methods are used respectively; S2040201. Obtaining the difference representation 1: Add a dimension to each of the two article representations, and then perform a join operation on the two article representations on the newly added dimension to obtain the question-and-choice-aware article representation, i.e. The specific implementation is shown in the following formula: （4.1） （4.2） （4.3） Wherein, formula (4.1) represents the article representation for question awareness using the Reshape dimension transformation operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Equation (4.2) represents the option-aware article representation using the Reshape operation. Adding a dimension, the result of adding the dimension The shape is (batch_size, time_steps, output_dimension, 1); Formula (4.3) indicates that the connection operation is used to... and By making connections on the newly added dimensions, we obtain the article representation for question selection perception. , its shape is (batch_size, time_steps, output_dimension, 2); Then, a 2D CNN is used to extract its features, followed by mapping through a 1-dimensional fully connected network, and finally, its dimensions are reshaped to obtain the difference representation 1, i.e., The specific implementation is shown in the following formula: （5.1） （5.2） （5.3） （5.4） （5.5） In this context, formula (5.1) represents the result of the f-th convolutional kernel convolving a specific region of the question-and-answer document representation and then mapping it using the ReLU function, where [x1, y1] represents the size of the convolutional kernel. This represents the weight matrix of the f-th convolutional kernel, where i and j represent the x and y coordinates of the convolutional region. and Let i:i+x1-1, j:j+y1-1 represent the length and height of the text representation in the question selection perception, and let j:j+y1-1 represent the convolutional region. This represents the bias matrix of the f-th convolutional kernel. Let represent the convolution result of the f-th convolution kernel in the region i:i+x1-1, j:j+y1-1; Formula (5.2) represents the integration of the convolution results of the f-th convolution kernel in each region to obtain the final convolution result of the f-th convolution kernel, where and Indicates the horizontal convolution stride and the vertical convolution stride. Let f represent the final convolution result of the f-th convolutional kernel; Formula (5.3) represents combining the final convolution results of n convolutional kernels to obtain the final convolution result of the network layer for the question-and-answer perception article representation, with a shape of (batch_size, time_steps, output_dimension, n); Formula (5.4) represents using a fully connected network of dimension 1 to map the final convolution result of the question-and-answer perception article representation to obtain the mapping result, with a shape of (batch_size, time_steps, output_dimension, 1); Formula (5.5) represents reshaping the shape of the mapping result, with the reshaping mapping result having a shape of (batch_size, time_steps, output_dimension). S2040202. Obtaining Difference Representation 2: Perform a difference operation on the two article representations and take their absolute values ​​to obtain Difference Representation 2, i.e. The specific implementation is shown in the following formula: （6） S20403. Obtain the perceptual representation of text differences: Connect the two difference representations to obtain the perceptual representation of text differences, i.e. It is then passed to the label prediction module; the specific implementation is shown in the following formula: （7）。 3. The text reading comprehension method based on article difference-aware representation according to claim 1, characterized in that, The specific steps for training the text reading comprehension model in step S3 are as follows: S301. Constructing the loss function Cross-entropy is used as the loss function; S302, Constructing the optimization function After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to the default values ​​in PyTorch. If the model has not yet been trained, step S3 needs to be performed to train it and optimize the model parameters; once the model has been trained, step S2 predicts which option is the correct answer.

4. A text reading comprehension device based on text difference-aware representation, used to implement the text reading comprehension method based on text difference-aware representation as described in any one of claims 1 to 3, characterized in that, The device includes: The text reading comprehension dataset acquisition unit is used to download publicly available text reading comprehension datasets from the internet or to construct datasets independently. The text reading comprehension model building unit is used to construct a pre-trained embedding representation module, a feature filtering module, an article difference-aware representation interaction module, and a label prediction module, thereby constructing a text reading comprehension model; The text reading comprehension model training unit is used to construct the loss function and optimization function to complete the answer selection.

5. The text reading comprehension device based on article difference perception representation according to claim 4, characterized in that, The text reading comprehension model building unit includes: The input module is responsible for preprocessing the raw dataset to construct the input data; The pre-trained embedding representation module is responsible for using a pre-trained language model to perform embedding encoding operations on the input data, thereby obtaining the embedding representations of articles, questions, and options in the input data; The feature filtering module is responsible for performing feature filtering operations on the embedded representations of articles, questions, and options using adaptive average pooling to obtain the corresponding pooled feature representations. The article difference-aware representation interaction module is responsible for receiving the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module. Then, it uses two methods to obtain two different article difference representations, and finally concatenates them to obtain the final article difference representation. The label prediction module is responsible for determining which candidate option is the correct option based on the article's difference representation. The text reading comprehension model training unit also includes: The loss function is responsible for calculating the error between the prediction result and the actual data using the cross-entropy loss function. The optimization function is responsible for training and adjusting the parameters during model training to reduce prediction error.

6. A storage medium storing a plurality of instructions, characterized in that, The instructions are loaded by the processor and execute the text reading comprehension method based on article difference-aware representation as described in any one of claims 1 to 3.

7. An electronic device, characterized in that, The electronic device includes: The storage medium as claimed in claim 6; and A processor for executing instructions in the storage medium.

Images