Textbook question answering method and system based on multi-level attention

Abstract

A textbook text question and answer method and system based on multi-level attention, the method comprising: inputting the question text and the corresponding chapter context paragraph into the first attention model after tokenization and coding, performing self-attention calculation and pooling within the sentence to obtain the sentence representation vector; calculating the cosine similarity of the question sentence representation vector and the representation vector of all context paragraph sentences, retaining only the representation vector corresponding to the sentence with the maximum similarity of each paragraph as the context representation vector of each paragraph corresponding to the question; inputting the question and the context representation vector of each paragraph corresponding to the question into the second attention model, performing self-attention calculation and pooling between the paragraphs and the question to obtain the answer representation vector corresponding to the question as the input of the classifier; outputting the answer options by the classifier, and obtaining the context paragraph of the answer text corresponding to the chapter. The application can select more accurate answer options for textbook text questions.

Description

Technical Field

[0001] This invention belongs to the field of computer text processing technology, specifically relating to a textbook text question-answering method and system based on multi-level attention. Background Technology

[0002] Question answering has long been a key research area in artificial intelligence. Solving question-answering challenges effectively not only means achieving the highest possible accuracy across a wide range of questions, but also involves simulating human problem-solving processes, making the model and algorithm's problem-solving process more human-like, thus achieving a higher level of artificial intelligence. Textbook question answering tasks are an emerging field in question answering. The data includes text and images from textbooks in subjects such as biology, geography, and natural sciences. The goal is to achieve higher accuracy on multiple-choice questions constructed primarily from natural science curriculum content.

[0003] However, current research on question-answering tasks mainly focuses on areas such as logical reasoning and reading comprehension, lacking research on question-answering questions in the field of natural science education. Furthermore, compared to mainstream question-answering tasks, textbook question-answering involves longer contextual parts and uses more specialized vocabulary. Current mainstream question-answering models, whether text-based or multimodal, perform poorly on textbook question-answering tasks due to limitations in the input length of typical small language models and the differences between educational corpora and natural language corpora. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the prior art by providing a textbook text question-answering method and system based on multi-level attention. This method can select more accurate answer options for textbook text questions and provide the text on which the answer depends, resulting in more interpretable answers.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A textbook text question-answering method based on multi-level attention includes the following steps:

[0007] The question text and the corresponding chapter's context paragraphs are segmented and encoded, then input into the first-layer attention model. Self-attention calculation and pooling are performed within the sentence to obtain the sentence's representation vector.

[0008] The cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences is calculated. Only the representation vector of the sentence with the highest similarity in each paragraph is retained as the context representation vector of the question in each paragraph.

[0009] The context representation vectors of the questions and the questions corresponding to each paragraph are input into the second-layer attention model to perform self-attention calculation and pooling between paragraphs and between paragraphs and questions, and the answer representation vectors corresponding to the questions are obtained as the input of the classifier.

[0010] The classifier outputs the answer options and derives the context paragraphs of the corresponding chapters from the answer text.

[0011] As a preferred embodiment, the step of segmenting and encoding the question text and its corresponding chapter's context paragraphs involves setting a maximum encoding length; if the length exceeds the maximum encoding length, the text is truncated; if the length is less than the maximum encoding length, it is padded. <pad>Characters, when the question text is q The context paragraphs of the corresponding chapter , n For the number of paragraphs, paragraph All sentences in the text are , m Given the number of sentences, segmentation and encoding are performed using the tokenizer corresponding to the bert-base-uncased pre-trained model, with each sentence as a unit. The mask sequence is preserved and used to determine the length of the meaningful sequence in the subsequent representation. len This prevents padding characters from participating in pooling.

[0012] As a preferred approach, the step of performing self-attention calculation within the sentence takes the unpooled output of the 12th layer of the model, which is the last layer. To obtain the sentence's representation vector, we perform average pooling on the non-filled portion of the two-dimensional matrix representing the word features within the sentence. For each context sentence A new representation is obtained after performing the same calculation. The calculation expressions for the above process are as follows:

[0013]

[0014]

[0015]

[0016]

[0017] In the formula, This is a function that combines 12 BERT forward modules. Let the problem vector be... For the first i The first in the paragraph j Sentence vectors The first layer of the model after processing i The first in the paragraph j The representation matrix of each sentence with word features. The representation matrix of the problem with word features. The dimension is The real vector, This represents a real vector with dimension 768.

[0018] As a preferred approach, the step of calculating the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences, and retaining only the representation vector corresponding to the sentence with the highest similarity in each paragraph, is expressed as follows:

[0019]

[0020]

[0021] The contextual representation vectors of the questions and the corresponding questions for each paragraph are obtained as follows: .

[0022] As a preferred approach, the second-layer attention model represents the question and the contextual representation vector of the question for each paragraph. The resulting matrix Self-attention calculation is performed sequentially on several encoded blocks. First, a linear transformation is applied to X, mapping it to three matrices corresponding to the query, index, and key values, respectively, using three different matrices. The expressions are as follows:

[0023]

[0024]

[0025]

[0026] Then, attention is calculated based on scaling dot products for queries, indexes, and key values, where... This represents the initial dimension of the representation vector. Indicates the output The dimension of a vector Indicates the output The dimension of a vector Indicates the output The dimension of a vector is expressed as follows:

[0027]

[0028] Within the same EB, multiple attention heads with different transformation matrices are used. Each attention head represents a resampling of the sequence, and each attention head maps the dimension to... , To determine the number of attention heads, the outputs of multiple heads are concatenated and mapped through a fully connected layer to restore the dimensions, as shown in the following expression:

[0029]

[0030]

[0031] In the formula, The final output of the multi-head attention process is... () is used to concatenate vectors. For the output of each attention head, It is a linear mapping matrix. These are the query vector, index vector, and key-value vector mentioned above.

[0032] As a preferred approach, it also includes adding the new attention representation of each sentence to the original input to establish a residual connection; performing layer standardization on the summed output; standardizing the vectors within a single sample according to their features, making their mean 0 and their variance 1; the standardized output is Y, which still retains the initial input shape; Y then enters the feedforward network bit by bit, and the same calculation is performed on each channel.

[0033] As a preferred embodiment, the classifier is a two-layer network employing a hyperbolic sine activation function, where the first layer is a dimension-invariant mapping with a linear mapping matrix of... The deviation term is The second-level linear mapping matrix Sum of deviations Reducing the dimension to two dimensions, i.e., the number of categories to be classified, yields Logits, which are the unnormalized classification weights. These are then used to calculate the cross-entropy with the labels, as shown in the following expression:

[0034]

[0035]

[0036]

[0037] In the formula, Here, e represents the classification weights of each category after softmax normalization, and e is the natural logarithm. For the final loss, To categorize the questions, n is 2 when the question is a true / false question and 4 when the question is a multiple-choice question. Is the true answer the first? i Item, at that time It's not time , for The i Item component.

[0038] As a preferred approach, when the question is a multiple-choice question, simply replace the original sentence of the question input with a concatenation of the question and each option, while expanding the number of channels corresponding to the question. During model computation, the question and context are used to form an input matrix, and finally, the representation vector of the first channel of the output is extracted and fed into the classifier. After mapping to 1D, the vectors are concatenated, and the loss is calculated. The calculation expression is as follows:

[0039]

[0040]

[0041]

[0042] In the formula, For the question and the first i The concatenated vector representing the options () represents the word segmenter. A token used at the beginning of a sentence to mark the entire sentence. For the problem itself, For the first i One option, A token used to separate the question from the context of subsequent input. The number of options is 4 for multiple-choice questions and 2 for true / false questions. For input, The classification weights are not normalized. For vector concatenation operations, This can be compared to the Y in a true / false question.

[0043] Secondly, a textbook text question-answering system based on multi-level attention is provided, including:

[0044] The first self-attention calculation module is used to segment and encode the question text and the context paragraphs of the corresponding chapter, and then input them into the first-layer attention model. Self-attention calculation and pooling are performed within the sentence to obtain the sentence representation vector.

[0045] The similarity calculation module is used to calculate the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences. Only the representation vector of the sentence with the highest similarity is retained for each paragraph, which serves as the context representation vector for the question in each paragraph.

[0046] The second self-attention calculation module is used to input the question and the context representation vector of the question corresponding to each paragraph into the second-layer attention model, perform self-attention calculation and pooling between paragraphs and between paragraphs and questions, and obtain the answer representation vector corresponding to the question as the input of the classifier;

[0047] The classifier output module is used to output answer options from the classifier and derive the context paragraphs of the corresponding chapters for the answer text.

[0048] Thirdly, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the textbook text question-answering method based on multi-level attention.

[0049] Compared with the prior art, the present invention has at least the following beneficial effects:

[0050] This invention employs a two-layer hierarchical attention structure. The main framework of the model consists of two parts: a coarse-grained attention model for paragraph and sentence filtering, which can be considered as preprocessing the input text; and a fine-grained attention model for question answering, which takes the output of the first layer and derives the final answer. The first-layer attention model performs self-attention calculation and pooling within sentences to obtain the sentence representation vector, effectively reducing the input text from paragraphs to sentences. This allows the model to extract the most relevant parts of the question from the entire text, eliminating interference from long texts. The second-layer attention model performs self-attention calculation between paragraphs and between paragraphs and questions, allowing for specific adjustments for textbook question-and-answer tasks to more accurately select the correct option and derive the context paragraphs of the corresponding chapters, resulting in more interpretable answers. This invention's multi-layered attention-based textbook text question-and-answer method solves the problem that existing automated question-and-answer technologies are not applicable to textbooks primarily containing natural science content. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the textbook text question-answering method based on multi-level attention, according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the first-layer attention model performing coarse-grained text filtering in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating the fine-grained question-answering result output of the second-layer attention model in this embodiment of the invention;

[0055] Figure 4 This is an example diagram showing the extraction results of text content by the method of this embodiment of the invention;

[0056] Figure 5 This is a schematic diagram of the model training process according to an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, those skilled in the art can obtain other embodiments without creative effort.

[0058] like Figure 1 As shown, for multiple-choice questions constructed primarily based on natural science courses, this invention proposes a textbook text question-answering method based on multi-level attention, specifically including the following steps:

[0059] When the question to be answered is a true / false question, this embodiment sets the maximum encoding length to 64; if it is greater than 64, it will be truncated; if it is less than 64, it will be padded. Character. See also Figure 2 When the question text is q, take the paragraph from the corresponding chapter. (n is the number of paragraphs), paragraphs All sentences in the text are (m is the number of sentences). Sentences are segmented and encoded using the tokenizer corresponding to the BERT-base-uncased pre-trained model, preserving the mask sequence. This mask sequence is used to determine the length len of the meaningful sequence in the subsequent representation, preventing padding characters from participating in pooling. The encoded vector q for each question is input into the first-layer attention model BERT1, where self-attention is only performed within the sentence. The unpooled output of the 12th layer of the model is used. This is a two-dimensional matrix representing the word features within a sentence. Average pooling is then performed on the unfilled portion to obtain the sentence's representation vector. Similarly, for each context sentence A new representation is obtained after performing the same calculation. The calculation expressions for the above process are as follows:

[0060] (1)

[0061] (2)

[0062] (3)

[0063] (4)

[0064] Take the representation vector of the problem and all Calculate cosine similarity, and for each paragraph, retain only the representation vector corresponding to the sentence with the highest similarity as the first sentence. i Each paragraph corresponds to the context representation vector of this problem. The calculation expression is as follows:

[0065] (5)

[0066] (6)

[0067] At this point, the original context paragraph All were reduced to sentences. Where n is the number of paragraphs, and each representation vector has 768 dimensions. 24 is the maximum possible number of sentences for each paragraph. Therefore, in order to fix the structure of subsequent models and ensure that each paragraph can be considered, The padding is uniformly increased to 24, and a parameter is reserved for each problem input to record the effective length, which is convenient for masking the padding vector during subsequent self-attention calculation.

[0068] See Figure 3 ,Will The input is fed into the second-layer attention model, which performs inter-paragraph self-attention computation. Its purpose is to perform self-attention computation and pooling between paragraphs and between paragraphs and questions, based on the paragraph representation vectors and question representation vectors obtained in the first two steps, further learning the joint representation between the question and the context. The second-layer model uses a variant of BERT, named ParaAttention.

[0069] The original BERT model starts by encoding word sequences, performing word embeddings, and then adding them to positional encodings. This process involves multiple layers of multi-head self-attention computation and a position-based multilayer perceptron to obtain the representations needed for downstream tasks. The absolute positional encoding, crucial in the Transformer, is a remedial measure to compensate for the inherent limitation of self-attention mechanisms in effectively utilizing positional information, unlike recurrent neural networks. Positional information is encoded as trigonometric functions (odd / even values), maintaining the same dimension as the word embedding vectors, and then added to them as a representation of positional information. While it doesn't completely solve the problem, it's a valuable supplement to some extent.

[0070] This invention directly uses the output of the first-layer model as the input of the second-layer model, without requiring the addition of positional information or word embedding. In the original BERT model, before inputting the word embedding vector, it needs to be multiplied by the square root of the word embedding vector's dimension and then added to the positional encoding to form the input vector. This is because the BERT word embedding matrix is ​​initialized using Xavier initialization, resulting in a vector variance that is the reciprocal of the square root of the vector dimension. Therefore, multiplying by the square root of the dimension helps control the variance to 1, and adding it to the positional encoding is beneficial for convergence. However, in TQA tasks, most questions can be answered by a paragraph or even a single sentence, so the order of paragraphs is not necessary and may even cause interference. On the other hand, the output of BERT1 is generated by a pre-trained and initialized model and has undergone pooling, making it difficult to determine its appropriate scaling value, and thus, the method of adding it to the positional encoding is uncertain. Considering all these factors, ParaAttntion abandons positional encoding.

[0071] The matrix formed by the second-layer attention model of this invention on the input sequence representation vector To perform self-attention calculation, let a coding block be EB. First, perform a linear transformation on X, mapping it to three matrices corresponding to the query, index, and key values, respectively. The calculation expression is as follows:

[0072] (7)

[0073] (8)

[0074] (9)

[0075] Then, attention based on scaling dot product is performed on the query, index, and key-value pairs, where... The initial dimension of the representation vector is 768 in this model, and its calculation expression is as follows:

[0076] (10)

[0077] Within the same EB, this embodiment of the invention uses multiple attention heads with different transformation matrices. Each head represents a resampling of the sequence, thus expanding the information that the model can capture. At the same time, a single head will change the dimension from Mapped to This also reduces the computational load. The outputs of multiple heads are then concatenated and mapped through a fully connected layer to restore the dimensions without losing information. The computation expression is as follows:

[0078] (11)

[0079] (12)

[0080] like Figure 3 As shown, the output is the new attention representation for each sentence. This output is then added to the original input, establishing a residual connection. This allows the model to remain effectively trained after deepening, preventing gradient vanishing. Layer standardization is applied to this summed output, which standardizes the vectors within a single sample according to their features, making the mean 0 and the variance 1. This controls the range and reduces gradient vanishing and gradient explosion. The standardized output, Y, retains the initial input shape: 25 channels and 768-dimensional features. Y then enters a bitwise feedforward network, a two-layer network that performs the same calculation for each channel. The feature dimension is first increased to 1536 dimensions, then reduced back to 768 dimensions, and then residually connected to Y and layer standardized again, increasing the network depth. The overall calculation process is expressed as follows:

[0081] (13)

[0082] (14)

[0083] At this point, the operation of one coding block is complete, and the resulting vector is then used as the new input to the next coding block. In this embodiment of the invention, the model consists of eight stacked coding blocks.

[0084] The first channel of the output from the last layer, representing the final result of the representation vector corresponding to the initial problem, is extracted and used as the input to the classifier. The classifier is a two-layer network using a hyperbolic sine activation function. The first layer is a dimension-invariant mapping, and the linear mapping matrix is... The deviation term is The second-level linear mapping matrix Sum of deviations By reducing the dimension to two dimensions, i.e., the number of categories to be classified, we obtain Logits, and then calculate the cross-entropy with the labels. The calculation expression is as follows:

[0085] (15)

[0086] (16)

[0087] (17)

[0088] Similarly, when the question is a multiple-choice question, we only need to replace the original sentence of the question input with a concatenation of the question and each option, and expand the number of channels corresponding to the question (7 for plain text multiple-choice questions, 4 for multiple-choice questions with illustrations). During model operation, the question and context are used to form input matrices, and finally, the representation vector of the first channel of the output is extracted and fed into the classifier. After mapping to 1D, the vectors are concatenated, and the loss is calculated. The specific calculation expression is as follows:

[0089] (18)

[0090] (19)

[0091] (20)

[0092] The overall model can then be trained on the dataset via backpropagation.

[0093] To investigate the effectiveness of the model in information extraction in this embodiment of the invention, a specific question was selected from the dataset for testing. The entire text corresponding to that question was processed using the model, and the sentence in each paragraph with the highest relevance to the question was highlighted with a box, as shown below. Figure 4 As shown.

[0094] By reading this question and this article, it can be seen that the sentence in each paragraph of the text that is most relevant to the question is highlighted in the box. This shows that the present invention can extract the most suitable content for answering questions to a certain extent.

[0095] The textbook text question-answering method based on multi-level attention of this invention, and its accuracy performance in the text part of textbook question answering, as well as its effect compared with other models, are shown in Table 1.

[0096] Table 1

[0097]

[0098] As can be seen, compared with the performance of conventional memory neural network models on text problems, the method proposed in this invention can achieve higher accuracy and provide the text on which the answer to the question depends.

[0099] Compared to the Roberta model, the method of this invention achieves better results on judgment-based problems without pre-training, and is also more interpretable.

[0100] Compared with the BiDAF model, the method of this invention achieves better results due to its unique attention mechanism design.

[0101] In summary, the multi-level attention-based model of this invention provides a certain benefit in answering textbook text questions.

[0102] The model of this invention uses the AdamW optimizer during training, with the initial learning rate set to... The batch size was uniformly reduced to 0; the number of training epochs was 5; to ensure all models and data fit the GPU memory size, the batch size was set to 1; to prevent overfitting, the dropout after each linear layer was set to 0.1; and the random number seed for all models was set to 42. The final training process is as follows: Figure 5 .

[0103] Embodiments of the present invention also propose a textbook text question-answering system based on multi-level attention, comprising:

[0104] The first self-attention calculation module is used to segment and encode the question text and the context paragraphs of the corresponding chapter, and then input them into the first-layer attention model. Self-attention calculation and pooling are performed within the sentence to obtain the sentence representation vector.

[0105] The similarity calculation module is used to calculate the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences. Only the representation vector of the sentence with the highest similarity is retained for each paragraph, which serves as the context representation vector for the question in each paragraph.

[0106] The second self-attention calculation module is used to input the question and the context representation vector of the question corresponding to each paragraph into the second-layer attention model, perform self-attention calculation and pooling between paragraphs and between paragraphs and questions, and obtain the answer representation vector corresponding to the question as the input of the classifier;

[0107] The classifier output module is used to output answer options from the classifier and derive the context paragraphs of the corresponding chapters for the answer text.

[0108] Embodiments of the present invention also propose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the textbook text question-answering method based on multi-level attention.

[0109] For example, the instructions stored in the memory can be divided into one or more modules / units. These modules / units are stored in a computer-readable storage medium and executed by the processor to complete the multi-level attention-based textbook text question-answering method described in this invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, describing the execution process of the computer program on the server.

[0110] The electronic device may be a smartphone, laptop, PDA, or cloud server, among other computing devices. It may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the electronic device may also include more or fewer components, or combinations of certain components, or different components; for example, it may also include input / output devices, network access devices, buses, etc.

[0111] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0112] The memory can be an internal storage unit of the server, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory may include both internal and external storage units. The memory is used to store computer-readable instructions and other programs and data required by the server. It can also be used to temporarily store data that has been output or will be output.

[0113] It should be noted that the information interaction and execution process between the above-mentioned module units are based on the same concept as the method embodiment. For details on their specific functions and technical effects, please refer to the method embodiment section. They will not be repeated here.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.< / pad>

Claims

1. A textbook text question-answering method based on multi-level attention, characterized in that, Includes the following steps: The question text and the corresponding chapter's context paragraphs are segmented and encoded, then input into the first-layer attention model. Self-attention calculation and pooling are performed within the sentence to obtain the sentence's representation vector. The cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences is calculated. Only the representation vector of the sentence with the highest similarity in each paragraph is retained as the context representation vector of the question in each paragraph. The context representation vectors of the questions and the questions corresponding to each paragraph are input into the second-layer attention model to perform self-attention calculation and pooling between paragraphs and between paragraphs and questions, and the answer representation vectors corresponding to the questions are obtained as the input of the classifier. The classifier outputs the answer options and derives the context paragraphs of the corresponding chapters from the answer text. The step of calculating the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences, and retaining only the representation vector corresponding to the sentence with the highest similarity in each paragraph, is expressed as follows: In the formula, This is the representation vector of the question sentence; For the first i The first in the paragraph j The representation vector of each sentence; Indicates a paragraph; The contextual representation vectors of the questions and the corresponding questions for each paragraph are obtained as follows: ; The second-layer attention model represents the question and the contextual representation vector of the question for each paragraph. The resulting matrix Self-attention calculation is performed sequentially on several encoded blocks. First, a linear transformation is applied to X, mapping it to three matrices corresponding to the query, index, and key values, respectively, using three different matrices. The expressions are as follows: Then, attention is calculated based on scaling dot products for queries, indexes, and key values, where... This represents the initial dimension of the representation vector. Indicates the output The dimension of a vector Indicates the output The dimension of a vector Indicates the output The dimension of a vector is expressed as follows: Within the same EB, multiple attention heads with different transformation matrices are used. Each attention head represents a resampling of the sequence, and each attention head maps the dimension to... , To determine the number of attention heads, the outputs of multiple heads are concatenated and mapped through a fully connected layer to restore the dimensions, as shown in the following expression: In the formula, The final output of the multi-head attention process is... () is used to concatenate vectors. For the output of each attention head, It is a linear mapping matrix. These are the query vector, index vector, and key-value vector mentioned above.

2. The textbook text question-answering method based on multi-level attention as described in claim 1, characterized in that, The step of segmenting and encoding the question text and its corresponding chapter's context paragraphs involves setting a maximum encoding length; if the length exceeds the maximum encoding length, the text is truncated; if it is less than the maximum encoding length, it is padded. <pad>Characters, when the question text is q The context paragraphs of the corresponding chapter , n For the number of paragraphs, paragraph All sentences in the text are , m For sentence vectors, each sentence is segmented and encoded using the tokenizer corresponding to the bert-base-uncased pre-trained model, preserving the mask sequence. This mask sequence is used to determine the length of the meaningful sequence in the subsequent representation. len This prevents padding characters from participating in pooling.< / pad> 3. The textbook text question-answering method based on multi-level attention as described in claim 2, characterized in that, The step of performing self-attention calculation within the sentence takes the unpooled output of the 12th layer of the model, which is the last layer. To obtain the sentence's representation vector, we perform average pooling on the non-filled portion of the two-dimensional matrix representing the word features within the sentence. For each context sentence A new representation is obtained after performing the same calculation. The calculation expressions for the above process are as follows: In the formula, This is a function that combines 12 BERT forward modules. Let the problem vector be... For the first i The first in the paragraph j Sentence vectors The first layer of the model after processing i The first in the paragraph j The representation matrix of each sentence with word features. The representation matrix of the problem with word features. The dimension is The real vector, This represents a real vector with dimension 768.

4. The textbook text question-answering method based on multi-level attention as described in claim 3, characterized in that, It also includes adding the new attention representation of each sentence to the original input to establish a residual connection; performing layer standardization on the summed output; standardizing the vectors in a single sample according to their features so that their mean becomes 0 and their variance becomes 1; the standardized output is Y, which still retains the initial input shape; Y then enters the feedforward network bit by bit, and the same calculation is performed on each channel.

5. The textbook text question-answering method based on multi-level attention according to claim 4, characterized in that, The classifier is a two-layer network using a hyperbolic sine activation function. The first layer is a dimension-invariant mapping, and the linear mapping matrix is: The deviation term is The second-level linear mapping matrix Sum of deviations Reducing the dimension to two dimensions, i.e., the number of categories to be classified, yields Logits, which are the unnormalized classification weights. These are then used to calculate the cross-entropy with the labels, as shown in the following expression: In the formula, These are the classification weights for each category after softmax normalization. It is the natural logarithm. For the final loss, To categorize the questions, n is 2 when the question is a true / false question and 4 when the question is a multiple-choice question. Is the true answer the first? i Item, at that time It's not time , for The i Item component.

6. The textbook text question-answering method based on multi-level attention according to claim 5, characterized in that, When the question is a multiple-choice question, simply replace the original sentence of the question input with a concatenation of the question and each option, while expanding the number of channels corresponding to the question. During model computation, the question and context are used to form an input matrix, and finally, the representation vector of the first channel of the output is extracted and fed into the classifier. After mapping to 1D, the vectors are concatenated, and the loss is calculated. The calculation expression is as follows: In the formula, Let be the representation vector of the question concatenated with the i-th option. () represents the word segmenter. A token used at the beginning of a sentence to mark the entire sentence. For the problem itself, For the first i One option, A token used to separate the question from the context of subsequent input. The number of options is 4 for multiple-choice questions and 2 for true / false questions. For input, The classification weights are not normalized. For vector concatenation operations, This can be compared to the Y in a true / false question.

7. A textbook text question-answering system based on multi-level attention, characterized in that, include: The first self-attention calculation module is used to segment and encode the question text and the context paragraphs of the corresponding chapter, and then input them into the first-layer attention model. Self-attention calculation and pooling are performed within the sentence to obtain the sentence representation vector. The similarity calculation module is used to calculate the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences. Only the representation vector of the sentence with the highest similarity is retained for each paragraph, which serves as the context representation vector for the question in each paragraph. The second self-attention calculation module is used to input the question and the context representation vector of the question corresponding to each paragraph into the second-layer attention model, perform self-attention calculation and pooling between paragraphs and between paragraphs and questions, and obtain the answer representation vector corresponding to the question as the input of the classifier; The classifier output module is used to output the answer options from the classifier and derive the context paragraphs of the corresponding chapters of the answer text; The step of calculating the cosine similarity between the representation vector of the question sentence and the representation vectors of all context paragraph sentences, and retaining only the representation vector corresponding to the sentence with the highest similarity in each paragraph, is expressed as follows: In the formula, This is the representation vector of the question sentence; For the first i The first in the paragraph j The representation vector of each sentence; Indicates a paragraph; The contextual representation vectors of the questions and the corresponding questions for each paragraph are obtained as follows: ; The second-layer attention model represents the question and the contextual representation vector of the question for each paragraph. The matrix formed Self-attention calculation is performed sequentially on several encoded blocks. First, a linear transformation is applied to X, mapping it to three matrices corresponding to the query, index, and key values, respectively, using three different matrices. The expressions are as follows: Then, attention is calculated based on scaling dot products for queries, indexes, and key values, where... This represents the initial dimension of the representation vector. Indicates the output The dimension of a vector Indicates the output The dimension of a vector Indicates the output The dimension of a vector is expressed as follows: Within the same EB, multiple attention heads with different transformation matrices are used. Each attention head represents a resampling of the sequence, and each attention head maps the dimension to... , To determine the number of attention heads, the outputs of multiple heads are concatenated and mapped through a fully connected layer to restore the dimensions, as shown in the following expression: In the formula, The final output of the multi-head attention process is... () is used to concatenate vectors. For the output of each attention head, It is a linear mapping matrix. These are the query vector, index vector, and key-value vector mentioned above.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the textbook text question-answering method based on multi-level attention as described in any one of claims 1 to 6.

Images