A selective reading comprehension method and device based on the Dot Triple-Attention mechanism
By implementing the Dot Triple-Attention mechanism to achieve direct interaction between the text, questions, and options, the problem of insufficient sequential interaction in existing methods is solved, thereby improving the accuracy and efficiency of selective reading comprehension.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2022-08-15
- Publication Date
- 2026-07-03
AI Technical Summary
Existing selective reading comprehension methods cannot effectively realize the direct interaction between the three text sequences of article, question, and option, resulting in insufficient model prediction accuracy.
A selection-based reading comprehension method based on the Dot Triple-Attention mechanism is adopted. By constructing a selection-based reading comprehension model, a pre-trained embedding representation module, a feature filtering module, a Dot Triple-Attention interaction module, and a feature aggregation module are used to realize direct interaction between three text sequences: article, question, and option.
It improves the prediction accuracy of selective reading comprehension systems by capturing rich semantic features and efficient sequence interactions, thereby enhancing the computational and training efficiency of the model and reducing training costs.
Smart Images
Figure CN115345172B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to a selective reading comprehension method and apparatus based on the Dot Triple-Attention mechanism. Background Technology
[0002] Machine reading comprehension is a task that uses algorithms to enable computers to understand the semantics of articles and answer related questions. Machine reading comprehension is a popular task in the field of natural language processing, which can be subdivided into four categories: fill-in-the-blank reading comprehension, extractive reading comprehension, selection reading comprehension, and generative reading comprehension. Among them, the candidate answer options in the selection reading comprehension task are proposed by human experts, and the content of the options generally does not appear directly in the article. In this task, choosing the correct answer requires strong summarization and reasoning abilities. Selection reading comprehension requires relying on a given article and selecting the correct answer from multiple options for the question posed in the question. The correct answer option must meet two requirements: (1) The answer option must conform to the semantic description of the article. As mentioned above, since the content of the option does not appear directly in the original text, in order to determine whether the option conforms to the semantics of the article, the reading comprehension system needs to be able to find the part in the text related to the option and make analogical reasoning with the option to determine whether the option conforms to the semantics of the article. (2) The answer option must be related to the corresponding question. Some options may be worded correctly and fit the description in the text, but they might be irrelevant to the question and not the answer to it. Therefore, selective reading comprehension is a very challenging task, and so far, existing methods have not substantially solved this problem.
[0003] To effectively utilize information from texts, most existing methods integrate this information into questions or options in various ways, then perform interactive matching between the questions and options, ultimately selecting the option with the highest match to the question as the correct answer. This approach is simple and easy to implement, but the problem lies in the fact that choice-based reading comprehension is essentially a task of determining the relationship between three text sequences (text, question, and options). Using a fusion approach to utilize text information, besides information loss during the fusion process, inevitably leads to insufficient interaction between sequences, thus reducing the accuracy of the reading comprehension model. Therefore, to completely solve this problem, a mechanism that enables direct interaction between the three text sequences is urgently needed. However, most current inter-sequence interaction mechanisms used in natural language processing are attention mechanisms. Existing attention mechanisms can only achieve interaction between two sequences, and cannot capture more interactions between sequences. For choice-based reading comprehension tasks, if the text, question, and options cannot interact directly, the interaction process will be insufficient, thus affecting the overall performance of the model.
[0004] To address the shortcomings of existing methods, this invention proposes a selection-based reading comprehension method and apparatus based on the Dot Triple-Attention mechanism. This method enables direct interaction between three text sequences: article, question, and option, thereby improving the prediction accuracy of the selection-based reading comprehension system. Summary of the Invention
[0005] The technical objective of this invention is to provide a method, apparatus, storage medium, and electronic device for selective reading comprehension based on the Dot Triple-Attention mechanism, in order to solve the problem of how to enable direct interaction between three text sequences: article, question, and option, thereby improving the prediction accuracy of the selective reading comprehension system.
[0006] The technical objective of this invention is achieved as follows: a selective reading comprehension method based on the Dot Triple-Attention mechanism, comprising the following steps:
[0007] S1. Obtain a selective reading comprehension dataset: Download a publicly available selective reading comprehension dataset from the internet or create your own dataset;
[0008] S2. Constructing a selective reading comprehension model: Constructing a selective reading comprehension model based on the Dot Triple-Attention mechanism;
[0009] S3. Training the selective reading comprehension model: Train the selective reading comprehension model constructed in step S2 on the selective reading comprehension training dataset obtained in step S1.
[0010] As a preferred embodiment, the specific steps for constructing the selective reading comprehension model in step S2 are as follows:
[0011] S201, Constructing the Input Module
[0012] For each data point in the dataset, the text sequence is denoted as context; the question sequence as query; and all candidate options as response. Based on the correct answer, the label for each data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. These three text sequences and the label together constitute one input data point.
[0013] S202, Constructing pre-trained embedded representation modules
[0014] The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. The specific implementation is shown in the following formula:
[0015]
[0016]
[0017]
[0018] In this equation (1.1), the pre-trained language model BERT is used to embed and encode the input article sequence context, and the subscript n represents the number of network layers within BERT. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates an embedded representation of the options;
[0019] S203, Constructing the Feature Filtering Module
[0020] Embedded representations of articles, questions, and options using adaptive average pooling operations. and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: and
[0021] S204. Construct the Dot Triple-Attention interaction module
[0022] This module is a stackable module, and its structure is as follows: Figure 4 As shown, it receives the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module as input. The first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, which are then passed to the second-layer interaction structure. At the same time, a fully connected network is used to map the article pooling feature representation to obtain the first-layer mapping representation of the article, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer question interaction result and the first-layer option interaction result based on the first-layer mapping representation of the article to obtain the second-layer question interaction result and the second-layer option interaction result, which are then passed to the third-layer interaction structure. At the same time, a fully connected network is used to map the first-layer mapping representation of the article to obtain the second-layer mapping representation of the article, which is then passed to the third-layer interaction structure. This process can be repeated multiple times to generate the depth-level question interaction result, the depth-level option interaction result, and the depth-level mapping result of the article. Finally, these results are passed to the feature aggregation module.
[0023] Specifically, the first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, denoted as... and The article pooling feature representation is mapped through a fully connected network to obtain the first-layer mapping representation of the article, denoted as . See the formula below for specific implementation details.
[0024]
[0025]
[0026]
[0027] in, and These represent the article pooling feature representation, question pooling feature representation, and option pooling feature representation obtained after processing by the feature filtering module, respectively; Formula (2.1) indicates the use of... Interactive operations are performed on the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation; Formula (2.2) represents the use of Interactive operations are performed on the pooling feature representations of options and questions based on the article pooling feature representation; Equation (2.3) indicates the use of the fully connected network Dense 1 A mapping operation is performed on the pooled feature representation of the article.
[0028] Furthermore, the first-level question interaction results, the first-level option interaction results, and the first-level mapping representation of the article are represented as follows: and This is passed to the second-level interaction structure; the second-level interaction structure, based on the first-level mapping representation of the article, interacts with the first-level question interaction result and the first-level option interaction result to obtain the second-level question interaction result and the second-level option interaction result, denoted as... and The first-layer mapping representation of the article is then mapped through a fully connected network to obtain the second-layer mapping representation of the article, denoted as . See the formula below for specific implementation details.
[0029]
[0030]
[0031]
[0032] Wherein, formula (3.1) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level question interaction result and the first-level option interaction result; Formula (3.2) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level option interaction results and the first-level question interaction results; Formula (3.3) represents the use of a fully connected network Dense 2 Perform mapping operations on the second-level mapping representation of the article.
[0033] Furthermore, the second-level question interaction results, the second-level option interaction results, and the second-level mapping representation of the article are represented as follows: and The interaction is passed to the third layer of the interaction structure; and so on, the interaction can be repeated multiple times to generate the question interaction representation, the option interaction representation, and the mapping representation of the article at the depth layer; for the depth layer, the specific implementation is shown in the following formula.
[0034]
[0035]
[0036]
[0037] Wherein, formula (4.1) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the question interaction result and the option interaction result at the depth-1 level; Formula (4.2) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the interaction results of the options at the depth-1 level and the interaction results of the questions at the depth-1 level; Equation (4.3) represents the use of a fully connected network Dense depth Perform a mapping operation on the (depth-1)th layer mapping representation of the article.
[0038] Furthermore, the formulas used in formulas (2.1) to (4.3) All are based on the Dot Triple-Attention mechanism. The Dot Triple-Attention algorithm consists of three steps: the first step is to calculate the relevance scores of the article representation, question representation, and option representation; the second step is to perform feature aggregation operation on the article representation and option representation; and the third step is to perform a dot product operation on the relevance score obtained in the first step and the feature aggregation representation obtained in the second step to obtain the interaction result of the corresponding representation.
[0039] S205, Constructing the Feature Aggregation Module
[0040] Adaptive average pooling is used to filter features from the question interaction results, option interaction results, and the article's mapping representation at the depth level. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, denoted as […].
[0041] S206, Constructing a Label Prediction Module
[0042] The final aggregated feature representation obtained in step S205 will be used as the input of this module. It will be processed by a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option as the correct answer. This module will predict the candidate option with the highest probability as the correct answer.
[0043] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S206 predicts which of the options is the correct answer.
[0044] Preferably, the Dot Triple-Attention mechanism construction process in step S204 is as follows:
[0045] S20401. Calculate the relevance score: This step first calculates the relevance score using three input tensors: query, key, and context, denoted as... A dot product operation is performed to achieve interactive computation among the three; then, a normalization operation is performed to obtain the correlation score among the three, which is denoted as α. The specific implementation is shown in the following formula:
[0046]
[0047]
[0048] In this context, formula (5.1) represents the interactive computation between query, key, and context through dot product operation; query represents the question information in reading comprehension; key represents the option information in reading comprehension; and context represents the joint context representation in reading comprehension. and These represent three input tensors, where d represents the input tensor. In the input tensor q, the element indices are: D represents the number of elements in the input tensor q, and i, i', j, and j' represent the input tensor. and The element index, I kl and J cl Representing the input tensors respectively and The number of elements in the tensor; Γ is a third-order tensor with shape D×D×D, where the element at position (d,d,d) has a value of 1 (d=1,2,…,D), and the element at other positions has a value of 0;
[0049] ×1, ×2, and ×3 represent modular multiplication (1, 2, and 3) respectively. These are custom tensor operators defined for easier formula writing and expression, and their operation process is as follows: Given two tensors... and but express and Modular multiplication of n, where Similarly, X and column vectors The modular multiplication rule of n is defined as in,
[0050] Formula (5.2) represents the correlation score of the three obtained through normalization operation, and the symbol has the same meaning as in Formula (5.1).
[0051] S20402, Feature Aggregation: This step mainly uses element-wise multiplication to perform feature aggregation on the two input tensors, value and context, denoted as... and Perform the aggregation operation, as detailed in the formula below.
[0052]
[0053] Here, 'value' represents the option information in the reading comprehension, which is the same as the input tensor 'key'; formula (5.3) indicates that the process is completed by element-wise multiplication. and The aggregation operation, where i and j represent the input tensors. and The element index, I vl and J cl Representing the input tensors respectively and The number of elements in the data.
[0054] S20403. Dot Product Operation: Perform a dot product operation between the relevance score tensor obtained in S20401 and the feature aggregation representation obtained in S20402 to obtain the context-enhanced attention embedding representation. See the following formula for the specific implementation:
[0055]
[0056] Where, α ij v' is the relevance score obtained from S20401. (i,j) Obtained for S20402 and The aggregation representation; i and j represent the element indices of α and v'; I and J represent the lengths of the two dimensions in the input tensors α and v'.
[0057] Dot Triple-Attention is a stackable structure that can dynamically adjust the number of layers according to the specific task, denoted as . depth indicates the specific number of layers set for a particular task; This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. Its and Same as above, omitted here.
[0058] As a preferred embodiment, the specific steps for training the selective reading comprehension model in step S3 are as follows:
[0059] S301. Constructing the loss function
[0060] Cross-entropy is used as the loss function;
[0061] S302, Constructing the optimization function
[0062] After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to their default values in PyTorch.
[0063] When the selective reading comprehension model has not yet been trained, it needs to be further trained to optimize its parameters; when the selective reading comprehension model has been trained, it can predict which of the candidate options is the correct option.
[0064] A selective reading comprehension device based on the Dot Triple-Attention mechanism, the device comprising:
[0065] The selective reading comprehension dataset acquisition unit is used to download publicly available selective reading comprehension datasets from the internet.
[0066] The selective reading comprehension model building unit is used to construct a pre-trained embedding representation module, a feature filtering module, a Dot Triple-Attention interaction module, a feature aggregation module, and a label prediction module, thereby constructing a selective reading comprehension model.
[0067] The selective reading comprehension model training unit is used to construct the loss function and optimization function to complete the answer selection.
[0068] Preferably, the selective reading comprehension model building unit includes:
[0069] The input module unit is responsible for preprocessing the raw dataset to construct the input data.
[0070] The pre-trained embedding representation module is responsible for using a pre-trained language model to perform embedding encoding operations on the input data, thereby obtaining the embedding representations of articles, questions, and options in the input data.
[0071] The feature filtering module is responsible for performing feature filtering operations on the embedded representations of articles, questions, and options using adaptive average pooling to obtain the corresponding feature pooling representations.
[0072] The Dot Triple-Attention interaction module unit is responsible for receiving the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module unit, and then repeatedly interacting to generate the question interaction result, option interaction result, and article mapping result at the depth layer.
[0073] The feature aggregation module is responsible for using adaptive average pooling to perform feature filtering on the question interaction results, option interaction results, and the article's mapping representation at the depth layer. It then concatenates the three filtered feature representations to obtain the final aggregated feature representation.
[0074] The label prediction module is responsible for determining which candidate option is the correct option based on the final aggregated feature representation.
[0075] The selective reading comprehension model training unit also includes:
[0076] The loss function unit is responsible for calculating the error between the prediction result and the actual data using the cross-entropy loss function.
[0077] The optimization function unit is responsible for training and adjusting the parameters during model training to reduce prediction error.
[0078] A storage medium storing multiple instructions, which are loaded by a processor to execute the steps of the above-described selective reading comprehension method based on the Dot Triple-Attention mechanism.
[0079] An electronic device, the electronic device comprising:
[0080] The aforementioned storage medium; and
[0081] A processor for executing instructions in the storage medium.
[0082] The selective reading comprehension method and apparatus based on the Dot Triple-Attention mechanism of the present invention have the following advantages:
[0083] (1) This invention can capture and utilize the semantic information contained in the text by pre-training the embedded representation module, making the captured semantic features richer and more accurate;
[0084] (2) The present invention can effectively filter out useless information and reduce sequence length through the feature filtering module, thereby reducing the cost of training the model and improving the efficiency of training.
[0085] (3) This invention enables direct interaction between the three sequences of text, question and option through the Dot Triple-Attention interaction module, making the sequence interaction more complete;
[0086] (4) This invention utilizes the high efficiency of the Dot product calculation method through the Dot Triple-Attention interaction module, thereby improving the computation and training efficiency of the model.
[0087] (5) The present invention can aggregate and utilize multiple features through the feature aggregation module, thereby improving the accuracy of selective reading comprehension;
[0088] (6) The method and apparatus proposed in this invention, combined with the Dot Triple-Attention mechanism, can effectively improve the accuracy of sequence interaction in selective reading comprehension. Attached Figure Description
[0089] The invention will be further described below with reference to the accompanying drawings.
[0090] Figure 1 A flowchart of a selective reading comprehension method based on the Dot Triple-Attention mechanism.
[0091] Figure 2 Flowchart for constructing a selective reading comprehension model
[0092] Figure 3 Flowchart for training a selective reading comprehension model
[0093] Figure 4 A flowchart of a selective reading comprehension device based on the Dot Triple-Attention mechanism.
[0094] Figure 5 A schematic diagram of the Dot Triple-Attention interaction module.
[0095] Figure 6 This is a schematic diagram of the framework of a selection-based reading comprehension model based on the Dot Triple-Attention mechanism. Detailed Implementation
[0096] The following detailed description of the selective reading comprehension method, apparatus, storage medium, and electronic device based on the Dot Triple-Attention mechanism of the present invention, with reference to the accompanying drawings and specific embodiments, is provided.
[0097] Example 1: A model framework for selective reading comprehension based on the Dot Triple-Attention mechanism.
[0098] The overall model framework structure of this invention is as follows: Figure 6 As shown. By Figure 6 As can be seen, the main framework of this invention includes a pre-trained embedding representation module, a feature filtering module, a Dot Triple-Attention interaction module, a feature aggregation module, and a label prediction module. The pre-trained embedding representation module performs embedding encoding on the input article sequence, question sequence, and option sequence to obtain their respective embedded encoding representations, which are then passed to the model's feature filtering module. The feature filtering module performs average pooling on the three embedded encoding representations to obtain filtered feature representations, namely, article pooling feature representations, question pooling feature representations, and option pooling feature representations, which are then passed to the Dot Triple-Attention interaction module. The Dot Triple-Attention interaction module is a stackable module with controllable layers. It receives the pooled feature representations output by the feature filtering module, performs interaction operations among the article pooling feature representations, question pooling feature representations, and option pooling feature representations through the Dot Triple-Attention mechanism, and passes the results to the feature aggregation module. The feature aggregation module performs average pooling on the interacted question and option representations and concatenates them with the article representation, which has also undergone average pooling, to obtain the final feature representation, which is then passed to the label prediction module. The label prediction module maps the feature representation to a floating-point number in a specified range, and uses it as the matching degree between the option and the question; then it compares the matching degrees of different options and predicts the option with the highest matching degree as the correct answer.
[0099] Example 2: Dot Triple-Attention mechanism.
[0100] Dot Triple-Attention is an attention mechanism based on the Dot Product Attention algorithm. Dot Product Attention first receives three input tensors: query, key, and value. Then, it calculates the attention weight between query and key, and finally calculates the dot product of the attention weight and value to obtain the final attention representation. The difference between Dot Triple-Attention and Dot Product Attention is that the latter receives three inputs (query, key, and value) and can only calculate the attention weight between query and key, while Dot Triple-Attention can receive four inputs (query, key, value, and context) and calculate the attention weight between query, key, and context, where context represents the contextual information related to query and key. The implementation process of the Dot Triple-Attention mechanism includes three steps: the first step is to calculate the relevance score between query, key, and context; the second step is to perform feature aggregation operation on context and value; and the third step is to perform a dot product operation between the relevance score obtained in the first step and the feature aggregation representation obtained in the second step to obtain the context-enhanced attention embedding representation. It should be noted that in Dot Triple-Attention and Dot Product Attention, the input value and key are the same.
[0101] Specifically, the implementation process of this mechanism is as follows:
[0102] The first step is to calculate the relevance score: This step begins by processing three input tensors, query, key, and context, denoted as... A dot product operation is performed to achieve interactive computation among the three; then, a normalization operation is performed to obtain the correlation score among the three, which is denoted as α. The specific implementation is shown in the following formula:
[0103]
[0104]
[0105] In this context, formula (5.1) represents the interactive computation between query, key, and context through dot product operation; query represents the question information in reading comprehension; key represents the option information in reading comprehension; and context represents the joint context representation in reading comprehension. and These represent three input tensors, where d represents the input tensor. In the index of the element, D represents the input tensor. The number of elements in the input tensor, where i, i', j, and j' represent the number of elements in the input tensor. and The element index, I kl and J cl Representing the input tensors respectively and The number of elements in; Γ is a third-order tensor with shape D×D×D, where the element at position (d,d,d) has a value of 1 (d=1,2,…,D), and the element at other positions has a value of 0; ×1, ×2 and ×3 represent 1-modular multiplication, 2-modular multiplication and 3-modular multiplication respectively; Formula (5.2) represents the correlation score of the three obtained through normalization operation, and the symbol meaning is the same as that of Formula (5.1).
[0106] For example:
[0107] In PyTorch, the code implementation for the above operation is as follows:
[0108] query_layer=self.query(sen1)
[0109] key_layer = self.key(sen2)
[0110] context_layer=self.context(context_sens)
[0111] key_context=unsqueeze(key_layer,dim=2)*unsqueeze(context_layer,dim=3)
[0112] key_context=key_context.view(key_context.size(0),key_context.size(1),key_context.size(2)*key_context.size(3),key_context.size(4))
[0113] attention_scores=torch.matmul(query_layer,key_context.transpose(-1,-2))
[0114] attention_probs=nn.Softmax(dim=-1)(attention_scores)
[0115] Where sen1 represents the formula sen2 represents the formula context_sens represents the formula attention_scores represents the formula in... attention_probs represents the relevance score in the formula, which is α.
[0116] The second step, feature aggregation: This step mainly involves element-wise multiplication of the two input tensors, value and context, denoted as... and Perform the aggregation operation, as detailed in the formula below.
[0117]
[0118] Here, 'value' represents the option information in the reading comprehension, which is the same as the input tensor 'key'; formula (5.3) indicates that the process is completed by element-wise multiplication. and The aggregation operation, where i and j represent the input tensors. and The element index, I vl and J cl Representing the input tensors respectively and The number of elements in the data.
[0119] For example:
[0120] In PyTorch, the code described above is implemented as follows:
[0121] value_context=unsqueeze(key_layer,dim=2)*unsqueeze(context_layer,dim=3)
[0122] value_context=value_context.view(value_context.size(0),value_context.size(1),value_context.size(2)*value_context.size(3),value_context.size(4))
[0123] Here, value_context represents the aggregated result in the formula, which is v'.
[0124] The third step is the dot product operation: the relevance score tensor obtained in the first step is multiplied by the feature aggregation representation obtained in the second step to obtain the context-enhanced attention embedding representation. The specific implementation is shown in the following formula:
[0125]
[0126] Where, α ij v' is the relevance score obtained in the first step. (i,j) For the second step and The aggregation representation; i and j represent the element indices of α and v'; I and J represent the lengths of the two dimensions in the input tensors α and v'.
[0127] For example, in PyTorch, the code implementation described above would be as follows:
[0128] query_new_layer=torch.matmul(attention_probs,value_context)
[0129] Where query_new_layer represents the interaction result in the formula, which is q new .
[0130] Dot Triple-Attention is a stackable structure that can dynamically adjust the number of layers according to the specific task, denoted as . depth indicates the specific number of layers set for a particular task; This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. Its and Same as above, omitted here.
[0131] Example 3: Specific steps of the invention.
[0132] As attached Figure 1 As shown, the present invention provides a selective reading comprehension method based on the Dot Triple-Attention mechanism, which includes the following steps:
[0133] S1. Obtain the multiple-choice reading comprehension dataset: Download a publicly available multiple-choice reading comprehension dataset from the internet;
[0134] S2. Constructing a selective reading comprehension model: Utilizing the Dot Triple-Attention mechanism to construct a selective reading comprehension model;
[0135] S3. Training the selective reading comprehension model: Train the selective reading comprehension model constructed in step S2 on the selective reading comprehension training dataset obtained in step S1.
[0136] S1. Obtain the selective reading comprehension dataset.
[0137] Download publicly available choice-based reading comprehension datasets from the internet or create your own dataset.
[0138] For example, there are many publicly available selective reading comprehension datasets online, such as RACE. The data format in this dataset is as follows:
[0139]
[0140]
[0141] S2. Constructing a selective reading comprehension model
[0142] The process of building a selective reading comprehension model is as follows: Figure 2 As shown, the main operations are to construct an input module, a pre-trained embedding representation module, a feature filtering module, a Dot Triple-Attention interaction module, a feature aggregation module, and a label prediction module.
[0143] S201, Constructing the Input Module
[0144] For each data point in the dataset, the text sequence is denoted as context; the question sequence is denoted as query; and all candidate options are denoted as response. Based on the correct answer, the label for that data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. The three text sequences and the label together constitute one input data point.
[0145] For example, using the data shown in S1 as a sample, we can construct one input data entry. The result is shown below:
[0146] (context:Are you carrying too much on your back at school?You're notalone.Back experts in the United States are worried that young students arehaving back and neck problems as a result of carrying too much in theirbackpacks(schoolbags)."It hurts my back when I run,"said Oberlin Reyes,astudent in Virginia."It's hard to get up the stairs with my backpack,becauseit's too heavy."Oberlin is among students whose backpacks were weighed for aweek in a recent study.They had regular backpacks with two straps to carrythem,but a number of students with heavy loads had switched to rollingbackpacks(which have wheels and can roll on the ground).Shirley Park,whosebackpack weighed 10kilograms,said she changed to a rolling backpack becauseshe was starting to have back pain.How much is too much?Experts say studentsshould carry no more than 10 to 15 percent of their own body weight.A fewstudents had one suggestion to lighten the load:less homework.P.S.Doctor'ssuggestion:①Lighten the load.Clean out binders and take home only the books you need that night. ②Wide straps are better. They send out the weight over your shoulders more evenly. And be sure to wear both straps rather than hanging the pack over one shoulder. ③Pack smart. isabout__., response: the problems made by rolling backpacks the advantage of backpacks the best backpacks for students how to lighten students' backpacks, 0001).
[0147] S202, Constructing pre-trained embedded representation modules
[0148] The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. The specific implementation is shown in the following formula:
[0149]
[0150]
[0151]
[0152] In this equation (1.1), the pre-trained language model BERT is used to embed and encode the input article sequence context, and the subscript n represents the number of network layers within BERT. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates the embedded representation of the option.
[0153] For example, when this invention is implemented on the RACE dataset, n is set to 12 and t is set to 1. In PyTorch, the code implementation described above is as follows:
[0154] context_embed,_=BERT_n(context)
[0155] query_embed, _ = BERT_t(query)
[0156] response_embed,_=BERT_t(response)
[0157] Here, context_embed is the embedded representation of the article, query_embed is the embedded representation of the question, and response_embed is the embedded representation of the option.
[0158] S203, Constructing the Feature Filtering Module
[0159] Embedded representations of articles, questions, and options using adaptive average pooling operations. and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: and
[0160] For example, in PyTorch, the code implementation described above would be as follows:
[0161] self.context_pool_adapt=AdaptiveAvgPool1d(output_size=64)
[0162] context_pool=torch.transpose(self.context_pool_adapt(torch.transpose(context_embed,dim0=1,dim1=2)),dim0=1,dim1=2)
[0163] self.qr_pool_adapt=AdaptiveAvgPool1d(output_size=15)
[0164] query_pool=torch.transpose(self.qr_pool_adapt(torch.transpose(query_embed,dim0=1,dim1=2)),dim0=1,dim1=2)
[0165] response_pool=torch.transpose(self.qr_pool_adapt(torch.transpose(response_embed,dim0=1,dim1=2)),dim0=1,dim1=2)
[0166] In this code, `AdaptiveAvgPool1d()` represents the adaptive average pooling operation, `output_size` represents the output dimension of the input data after processing by this operation, and in this invention, the output dimension for the article representation is set to 64, and the output dimension for the question and option is set to 15. `torch.transpose` represents the transpose operation, `context_pool` represents the article pooled feature representation obtained after adaptive average pooling of the article's embedding representation, `query_pool` represents the question pooled feature representation obtained after adaptive average pooling of the question's embedding representation, and `response_pool` represents the option pooled feature representation obtained after adaptive average pooling of the option's embedding representation.
[0167] S204. Construct the Dot Triple-Attention interaction module
[0168] This module is a stackable module, and its structure is as follows: Figure 5As shown, it receives the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module as input. The first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, which are then passed to the second-layer interaction structure. At the same time, a fully connected network is used to map the article pooling feature representation to obtain the first-layer mapping representation of the article, which is then passed to the second-layer interaction structure. The second-layer interaction structure interacts with the first-layer question interaction result and the first-layer option interaction result based on the first-layer mapping representation of the article to obtain the second-layer question interaction result and the second-layer option interaction result, which are then passed to the third-layer interaction structure. At the same time, a fully connected network is used to map the first-layer mapping representation of the article to obtain the second-layer mapping representation of the article, which is then passed to the third-layer interaction structure. This process can be repeated multiple times to generate the depth-level question interaction result, the depth-level option interaction result, and the depth-level mapping result of the article. Finally, these results are passed to the feature aggregation module.
[0169] Specifically, the first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, denoted as... and The article pooling feature representation is mapped through a fully connected network to obtain the first-layer mapping representation of the article, denoted as . See the formula below for specific implementation details.
[0170]
[0171]
[0172]
[0173] in, and These represent the article pooling feature representation, question pooling feature representation, and option pooling feature representation obtained after processing by the feature filtering module, respectively; Formula (2.1) indicates the use of... Interactive operations are performed on the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation; Formula (2.2) represents the use of Interactive operations are performed on the pooling feature representations of options and questions based on the article pooling feature representation; Equation (2.3) indicates the use of the fully connected network Dense 1 A mapping operation is performed on the pooled feature representation of the article.
[0174] Furthermore, the first-level question interaction results, the first-level option interaction results, and the first-level mapping representation of the article are represented as follows: and This is passed to the second-level interaction structure; the second-level interaction structure, based on the first-level mapping representation of the article, interacts with the first-level question interaction result and the first-level option interaction result to obtain the second-level question interaction result and the second-level option interaction result, denoted as... and The first-layer mapping representation of the article is then mapped through a fully connected network to obtain the second-layer mapping representation of the article, denoted as . See the formula below for specific implementation details.
[0175]
[0176]
[0177]
[0178] Wherein, formula (3.1) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level question interaction result and the first-level option interaction result; Formula (3.2) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level option interaction results and the first-level question interaction results; Formula (3.3) represents the use of a fully connected network Dense 2 Perform mapping operations on the second-level mapping representation of the article.
[0179] Furthermore, the second-level question interaction results, the second-level option interaction results, and the second-level mapping representation of the article are represented as follows: and The interaction is passed to the third layer of the interaction structure; and so on, the interaction can be repeated multiple times to generate the question interaction representation, the option interaction representation, and the mapping representation of the article at the depth layer; for the depth layer, the specific implementation is shown in the following formula.
[0180]
[0181]
[0182]
[0183] Wherein, formula (4.1) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the question interaction result and the option interaction result at the depth-1 level; Formula (4.2) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the interaction results of the options at the depth-1 level and the interaction results of the questions at the depth-1 level; Equation (4.3) represents the use of a fully connected network Dense depth Perform a mapping operation on the (depth-1)th layer mapping representation of the article.
[0184] For example, in PyTorch, the code implementation described above would be as follows:
[0185]
[0186] Where query_inter is the query interaction result at level i, response_inter is the option interaction result at level i, context_dense is the mapping representation of the article at level i, and depth is the level at which the structure achieves the best results on the dataset. In this invention, depth is set to 3.
[0187] Furthermore, the formulas used in formulas (2.1) to (4.3) All are implemented based on the Dot Triple-Attention mechanism. The Dot Triple-Attention algorithm consists of three steps: the first step is to calculate the relevance scores of the article representation, question representation, and option representation; the second step is to perform feature aggregation operation on the article representation and option representation; and the third step is to perform a dot product operation on the relevance score obtained in the first step and the feature aggregation representation obtained in the second step to obtain the interaction result. The specific implementation is described in Example 2. The article embedding representation, question embedding representation, and option embedding representation correspond to the context, key, and query in Example 2, respectively.
[0188] S205, Constructing the Feature Aggregation Module
[0189] Adaptive average pooling is used to filter features from the question interaction results, option interaction results, and the article's mapping representation at the depth level. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, denoted as […].
[0190] For example, in PyTorch, the code implementation described above would be as follows:
[0191] self.pool_qo=AdaptiveAvgPool1d(output_size=64)
[0192] query_out=torch.squeeze(self.pool_qo(torch.transpose(query_inter,dim0=1,dim1=2)),dim=-1)
[0193] response_out=torch.squeeze(self.pool_qo(torch.transpose(response_inter,dim0=1,dim1=2)),dim=-1)
[0194] context_out=torch.squeeze(self.pool_qo(torch.transpose(context_dense,dim0=1,dim1=2)),dim=-1)
[0195] feature=torch.cat([query_out,response_out,context_out],dim=-1)
[0196] Among them, query_out is the question feature representation after average pooling of the question interaction results at the depth layer, response_inter is the option feature representation after average pooling of the option interaction results at the depth layer, context_dense is the article feature representation after average pooling of the article's mapping representation at the depth layer, and feature is the final aggregated feature representation.
[0197] S206, Constructing a Label Prediction Module
[0198] The final aggregated feature representation obtained in step S205 will serve as the input to this module. It will be processed through a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option being the correct answer. This module will predict the candidate option with the highest probability as the correct answer.
[0199] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S206 predicts which of the options is the correct answer.
[0200] S3, Training a Selective Reading Comprehension Model
[0201] The selection-based reading comprehension model constructed in step S2 is trained on the selection-based reading comprehension training dataset obtained in step S1. The process is as follows: Figure 3 As shown.
[0202] S301. Constructing the loss function
[0203] Cross-entropy is used as the loss function.
[0204] For example, in PyTorch, the code implementation described above would be as follows:
[0205] loss_fct = CrossEntropyLoss()
[0206] loss=loss_fct(logits.view(-1,self.num_labels),labels.view(-1))
[0207] Here, labels are the true labels, and logits are the correct probabilities of the model output.
[0208] S302, Constructing the optimization function
[0209] After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to their default values in PyTorch.
[0210] For example, in PyTorch, the code implementation described above would be as follows:
[0211] optimizer=BertAdam(optimizer_grouped_parameters,lr=2e-5)
[0212] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in this model.
[0213] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S206 predicts which of the options is the correct answer.
[0214] Example 4:
[0215] Based on a storage medium of Embodiment 1, which stores multiple instructions, the instructions are loaded by a processor to execute the steps of the selective reading comprehension method based on the Dot Triple-Attention mechanism of Embodiment 1.
[0216] Example 5:
[0217] An electronic device based on Embodiment 4 includes: a storage medium of Embodiment 4; and a processor for executing instructions in the storage medium.
[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; the present invention has been described in detail with reference to the foregoing embodiments. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A selective reading comprehension method based on the Dot Triple-Attention mechanism, characterized in that, The method includes the following steps: S1. Obtain a selective reading comprehension dataset: Download a publicly available selective reading comprehension dataset from the internet or create your own dataset; S2. Constructing a selective reading comprehension model: Constructing a selective reading comprehension model based on the Dot Triple-Attention mechanism; S3. Training the selective reading comprehension model: Train the selective reading comprehension model constructed in step S2 on the selective reading comprehension training dataset obtained in step S1; The specific steps for constructing the selective reading comprehension model in step S2 are as follows: S201, Constructing the Input Module For each data point in the dataset, the text sequence is denoted as context; the question sequence is denoted as query; and all candidate options are denoted as response. Based on the correct answer, the label for each data point is determined: if the correct answer is A, it is denoted as 1000; if the correct answer is B, it is denoted as 0100; if the correct answer is C, it is denoted as 0010; and if the correct answer is D, it is denoted as 0001. These three text sequences and the label together constitute one input data point. S202, Constructing pre-trained embedded representation modules The pre-trained embedding representation module uses a pre-trained language model to perform embedding encoding operations on the input data constructed in S201, thereby obtaining the embedding representations of the text, questions, and options in the input data, denoted as follows: , and The input constructed in S201 contains three text sequences. The article sequence uses a separate encoding module, while the question and option sequences share a common encoding module. Since the pre-trained language model itself contains a multi-layer encoding network, the output of different layers can be selected as its embedding representation based on different input objects. See the following formula for specific implementation: （1.1） （1.2） （1.3） In this context, formula (1.1) represents the embedding encoding of the input article sequence context using the pre-trained language model BERT, and the subscript n represents the number of network layers within BERT used. The text represents the embedding representation of the article; Formula (1.2) indicates the embedding encoding of the input question sequence query using the pre-trained language model BERT, where the subscript t indicates the number of network layers within BERT used. This represents the embedded representation of a question; formula (1.3) is basically the same as formula (1.2), except that the encoding object is the option response. This indicates an embedded representation of the options; S203, Construct the feature filtering module; S204. Construct the Dot Triple-Attention interaction module; S205, Constructing the feature aggregation module: Adaptive average pooling is used to filter features from the question interaction results, option interaction results, and the article's mapping representation at the depth level. The three filtered feature representations are then concatenated to obtain the final aggregated feature representation, denoted as […]. ; S206. Construct the label prediction module: The final aggregated feature representation obtained in step S205 will be used as input, and it will be processed by a fully connected network with a dimension of 4 and an activation function of softmax to obtain the probability of each candidate option as the correct answer. This module will predict the candidate option with the highest probability as the correct answer. If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; when the model has been trained, step S206 predicts which of the options is the correct answer.
2. The selective reading comprehension method based on the Dot Triple-Attention mechanism according to claim 1, characterized in that, The construction of the feature filtering module in step S203 is as follows: Embedded representations of articles, questions, and options using adaptive average pooling operations. , and Feature filtering is performed to obtain the corresponding pooling feature representations, namely, the article pooling feature representation, the question pooling feature representation, and the option pooling feature representation, which are denoted as follows: , and .
3. The selective reading comprehension method based on the Dot Triple-Attention mechanism according to claim 1, characterized in that, The construction of the Dot Triple-Attention interaction module in step S204 is as follows: This module is a stackable module; it receives the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module as input; the first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, and then passes them to the second-layer interaction structure. At the same time, a fully connected network is used to map the article pooling feature representation to obtain the first-layer mapped representation of the article, and then passes it to the second-layer interaction structure; the second-layer interaction structure interacts with the first-layer question interaction result and the first-layer option interaction result based on the first-layer mapped representation of the article to obtain the second-layer question interaction result and the second-layer option interaction result, and then passes them to the third-layer interaction structure. At the same time, a fully connected network is used to map the first-layer mapped representation of the article to obtain the second-layer mapped representation of the article, and then passes it to the third-layer interaction structure; This process is repeated multiple times to generate the question interaction results at the depth level, the option interaction results at the depth level, and the mapping results at the depth level of the article; finally, these results are fed into the feature aggregation module. Specifically, the first-layer interaction structure interacts with the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation to obtain the first-layer question interaction result and the first-layer option interaction result, denoted as... and The article pooling feature representation is mapped through a fully connected network to obtain the first-layer mapping representation of the article, denoted as . The specific implementation is shown in the following formula: （2.1） （2.2） （2.3） in, , and These represent the article pooling feature representation, question pooling feature representation, and option pooling feature representation obtained after processing by the feature filtering module, respectively; Formula (2.1) indicates the use of... Interactive operations are performed on the question pooling feature representation and the option pooling feature representation based on the article pooling feature representation; Formula (2.2) represents the use of Interactive operations are performed on the pooling feature representations of options and questions based on the article pooling feature representation; Equation (2.3) indicates the use of a fully connected network. Perform a mapping operation on the article pooling feature representation; Furthermore, the first-level question interaction results, the first-level option interaction results, and the first-level mapping representation of the article are represented as follows: , and This is passed to the second-level interaction structure; the second-level interaction structure, based on the first-level mapping representation of the article, interacts with the first-level question interaction result and the first-level option interaction result to obtain the second-level question interaction result and the second-level option interaction result, denoted as... and The first-level mapping representation of the article is then mapped through a fully connected network to obtain the second-level mapping representation of the article, denoted as . The specific implementation is shown in the following formula: （3.1） （3.2） （3.3） Wherein, formula (3.1) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level question interaction result and the first-level option interaction result; formula (3.2) represents the use of The first-level mapping based on the article represents the interaction operation between the first-level option interaction result and the first-level question interaction result; Formula (3.3) represents the use of a fully connected network. Perform mapping operations on the second-level mapping representation of the article; Furthermore, the second-level question interaction results, the second-level option interaction results, and the second-level mapping representation of the article are represented as follows: , and This is passed to the third-level interaction structure; and so on, repeated interactions are performed to generate the question interaction representation, option interaction representation, and article mapping representation at the depth level; for the depth level, the specific implementation is shown in the following formula: （4.1） （4.2） （4.3） Wherein, formula (4.1) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the interaction results of the question at the depth-1 level and the interaction results of the option at the depth-1 level; Formula (4.2) represents the use of The mapping at the depth-1 level based on the article represents the interaction operation between the interaction results of the options at the depth-1 level and the interaction results of the questions at the depth-1 level; Formula (4.3) represents the use of a fully connected network. Perform a mapping operation on the (depth-1)th layer mapping representation of the article; Furthermore, the formulas used in formulas (2.1) to (4.3) , , All are based on the Dot Triple-Attention mechanism. The Dot Triple-Attention algorithm consists of three steps: the first step is to calculate the relevance scores of the article representation, question representation, and option representation; the second step is to perform feature aggregation operation on the article representation and option representation; and the third step is to perform a dot product operation on the relevance score obtained in the first step and the feature aggregation representation obtained in the second step to obtain the interaction result.
4. The selective reading comprehension method based on the Dot Triple-Attention mechanism according to claim 1, characterized in that, The construction process of the Dot Triple-Attention mechanism in step S204 is as follows: S20401. Calculate the relevance score: First, by processing the three input tensors query, key, and context, denoted as... , , The dot product operation is performed to achieve interactive computation among the three; then, the correlation score among the three is obtained through normalization, and denoted as... For specific implementation details, please refer to the following formula: ， i=1,2,…,I kl , j=1,2,…,J cl (5.1) ,i=1,2,…, I kl , j=1,2,…,J cl (5.2) In this context, formula (5.1) represents the interactive computation between query, key, and context through dot product operation; query represents the question information in reading comprehension; key represents the option information in reading comprehension; and context represents the joint context representation in reading comprehension. , and These represent three input tensors, where d represents the input tensor. In the index of the element, D represents the input tensor. The number of elements in the input tensor, where i, i', j, and j' represent the number of elements in the input tensor. and The element index, I kl and J cl Representing the input tensors respectively and The number of elements in; It is a shape of D D A third-order tensor of D, where the element at position (d,d,d) has a value of 1 (d=1,2,…,D), and the elements at other positions have a value of 0; , and These represent modular multiplication (1), modular multiplication (2), and modular multiplication (3), respectively. Formula (5.2) represents the correlation score obtained by normalization, with the same symbol meaning as formula (5.1). S20402, Feature Aggregation: This step mainly uses element-wise multiplication to perform feature aggregation on the two input tensors, value and context, denoted as... and To perform the aggregation operation, see the following formula for details: ,i=1,2,…,I vl , j=1,2,…,J cl (5.3) Here, 'value' represents the option information in the reading comprehension, which is the same as the input tensor 'key'; formula (5.3) indicates that the process is completed by element-wise multiplication. and The aggregation operation, where i and j represent the input tensors. and The element index, I vl and J cl Representing the input tensors respectively and The number of elements in; S20403. Dot Product Operation: Perform a dot product operation between the relevance score tensor obtained in S20401 and the feature aggregation representation obtained in S20402 to obtain the context-enhanced attention embedding representation. See the following formula for the specific implementation: （5.4） in, The relevance score obtained for S20401 Obtained for S20402 and Aggregate representation; i and j represent and Element indices; I and J represent input tensors and The length of the two dimensions; Dot Triple-Attention is a stackable structure that dynamically adjusts the number of layers based on the specific task, denoted as . depth indicates the specific number of layers set for a particular task; , , This corresponds to three input tensors: query, key, and context; for the input tensor value, i.e. , and The same, in The formal parameters are omitted.
5. The selective reading comprehension method based on the Dot Triple-Attention mechanism according to claim 1, characterized in that, The specific steps for training the selective reading comprehension model in step S3 are as follows: S301. Constructing the loss function: Cross-entropy is used as the loss function; S302. Construct the optimization function: After testing various optimization functions, the BertAdam optimization function was finally selected as the optimization function for this model. Except for its learning rate being set to 2e-5, all other hyperparameters of BertAdam were set to the default values in PyTorch. If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; once the model has been trained, step S2 predicts which of the options is the correct answer.
6. A selective reading comprehension device based on the Dot Triple-Attention mechanism, used to implement the selective reading comprehension method based on the Dot Triple-Attention mechanism as described in any one of claims 1 to 5, characterized in that, The device includes: The selective reading comprehension dataset acquisition unit is used to download publicly available selective reading comprehension datasets from the internet or to construct your own dataset. The selective reading comprehension model building unit is used to build a pre-trained embedding representation module, a feature filtering module, a DotTriple-Attention interaction module, a feature aggregation module, and a label prediction module, thereby constructing a selective reading comprehension model; The selective reading comprehension model training unit is used to construct the loss function and optimization function to complete the answer selection; The selective reading comprehension model building unit includes: The input module is responsible for preprocessing the raw dataset to construct the input data; The pre-trained embedding representation module is responsible for using a pre-trained language model to perform embedding encoding operations on the input data, thereby obtaining the embedding representations of articles, questions, and options in the input data; The feature filtering module is responsible for performing feature filtering operations on the embedded representations of articles, questions, and options using adaptive average pooling to obtain the corresponding pooled feature representations. The Dot Triple-Attention interaction module is responsible for receiving the article pooling feature representation, question pooling feature representation, and option pooling feature representation output by the feature filtering module unit, and then repeatedly interacting to generate the question interaction result, option interaction result, and article mapping result at the depth layer. The feature aggregation module is responsible for using adaptive average pooling to perform feature filtering on the question interaction results, option interaction results, and the article's mapping representation at the depth layer, and then concatenating the three filtered feature representations to obtain the final aggregated feature representation. The label prediction module is responsible for determining which candidate option is the correct option based on the final aggregated feature representation. The selective reading comprehension model training unit further includes: The loss function is responsible for calculating the error between the prediction result and the actual data using the cross-entropy loss function. The optimization function is responsible for training and adjusting the parameters during model training to reduce prediction error.
7. A storage medium storing a plurality of instructions, characterized in that, The instructions are loaded by the processor and execute the selective reading comprehension method based on the Dot Triple-Attention mechanism as described in any one of claims 1 to 5.
8. An electronic device, characterized in that, The electronic device includes: The storage medium as claimed in claim 7; and A processor for executing instructions in the storage medium.