A Biomedical Extractive Question Answering Method Based on Dynamic Routing and Answer Voting

Abstract

This invention discloses a biomedical extractive question answering method based on dynamic routing and answer voting. Specifically, it involves collecting and preprocessing a dataset into the format required for extractive question answering tasks; designing a new routing algorithm to dynamically route the hidden layers of a pre-trained model and dynamically assign appropriate weights; employing a voting mechanism in the prediction phase to select the more likely correct answer, which considers the similarity between answers; and using a pre-fine-tuning method to pre-train the model on a general corpus to improve its performance on biomedical question answering tasks. This invention adds dynamic routing and answer voting to ALBERT and pre-trains it, effectively improving model performance. It achieves good performance while maintaining a small parameter count.

Description

Technical Field

[0001] This invention relates to a biomedical extraction-based question-answering method based on dynamic routing and answer voting, belonging to the fields of Internet and artificial intelligence technology. Background Technology

[0002] Artificial intelligence is a highly challenging science, and extractive question answering is a popular and valuable research topic within it. From an academic perspective, extractive question answering aims to enable machines to extract answers to corresponding questions after a deep understanding and analysis of the semantics conveyed by a passage. From a practical application perspective, extractive question answering also has wide-ranging applications in our lives. For example, intelligent customer service uses question-answering technology to answer users' questions. At the same time, it is also important in other fields, such as biomedical question answering.

[0003] Biomedical knowledge acquisition is a crucial task in information retrieval. Professionals and the public need assistance in acquiring and understanding biomedical concepts. In recent years, with the development of network technology and the accumulation of big data, healthcare services have increasingly emerged, including online medical information retrieval and biomedical question answering. Biomedical question answering is a subtask of Natural Language Processing (NLP) in the biomedical field. It can extract biomedical text, aiding in knowledge retrieval. Biomedical question answering is an important component of question answering, and it is a challenging task. Many existing biomedical question answering methods are based on pre-trained language models. Pre-trained language models have achieved high performance on various NLP tasks. In particular, the emergence of the BERT series has greatly promoted the development of pre-trained models in the NLP field. Subsequent models such as BioBERT, pre-trained on biomedical medical databases, have effectively improved the performance of pre-trained models in biomedicine. This model has a similar structure to BERT, mainly trained on a large amount of biomedical corpus. Pre-trained models have almost become an unavoidable topic in various NLP tasks at present.

[0004] However, many existing works do not fully utilize the hidden layer knowledge of pre-trained models. For example, they only use the output of the last hidden layer of the pre-trained model, or they only apply fixed weights to the outputs of multiple hidden layers. This approach is clearly inflexible and does not effectively utilize the hidden layer knowledge learned by pre-trained models on large-scale corpora. Furthermore, a question often presents a set of answers. Most existing methods do not consider the implied meaning of the similarity between answers and other answers. For example, an answer that is similar to many other answers is likely the correct answer. This implicit information should not be ignored.

[0005] To address this, this invention proposes a biomedical extractive question answering method based on dynamic routing and answer voting. This dynamic routing mechanism adaptively and fully utilizes hidden layer knowledge. A dynamic routing algorithm is designed that dynamically assigns appropriate weights to multiple hidden layers and automatically adjusts these weights, rather than using fixed weights. This algorithm can rationally route multiple hidden layers. The answer voting strategy of this scheme can better select the optimal answer. An answer voting module is designed to calculate voting scores, taking into account the similarity between answers. The answer score is no longer just a predicted score, but is composed of both the predicted score and the voting score. Furthermore, to improve the model's performance on biomedical question answering, the model is pre-tuned on the SQuAD dataset, and a dynamic learning rate mechanism is introduced during the pre-tuning process. Summary of the Invention

[0006] To address the problems and shortcomings of existing technologies, this invention provides a biomedical extractive question answering method based on dynamic routing and answer voting. This method can dynamically route multiple hidden layers of a pre-trained model, thereby making more reasonable use of the hidden layer knowledge learned by the pre-trained model on a large-scale corpus. In addition, for a set of answers predicted by the model, the similarity between the answers is also taken into account. By utilizing this implicit information, more appropriate answers can be generated.

[0007] To achieve the above objectives, the technical solution of this invention is as follows: a biomedical extractive question-answering method based on dynamic routing and answer voting, which encompasses a dynamic routing mechanism and an answer voting strategy. The method mainly includes three steps, as detailed below:

[0008] Step 1: Collect and preprocess the dataset.

[0009] Since our research focuses on biomedical extractive question answering tasks, we first collected biomedical datasets such as BioASQfactoid4b, BioASQfactoid5b, BioASQfactoid6b, and BioASQfactoid9b, which can be found in the BioASQ challenge. We then collected the publicly available SQuAD dataset, which is used for pre-training models to improve performance. The BioASQ datasets were used for both model training and testing. We transformed the BioASQ datasets into a format similar to SQuAD. After this data processing, the BioASQ biomedical datasets are ready for use in extractive question answering tasks.

[0010] Step 2, Model Training. First, the model's input is encoded into embedding vectors by the embedding layer. These embedding vectors are then encoded by the encoding layer to obtain the output of the hidden layers. Multiple hidden layers requiring routing are defined, and their outputs are used as input to the dynamic routing module to obtain the dynamically routed output. This output is then passed through a linear layer with an output dimension of 2 to obtain the linear layer's output. The linear layer's output is passed through a pre-output layer to obtain a pre-output set of answers and their corresponding predicted scores. These are then input to the answer voting module to obtain the corresponding voting scores. The predicted scores and voting scores are combined to obtain the answer's score, which is then input to the output layer to obtain the final output answer. Details are as follows:

[0011] The model takes a sequence of paragraphs plus a question as input. First, it passes through an embedding layer to be transformed into an embedding vector E. Then, the embedding vector is input into the encoding layer.

[0012] H all =Encoder(E)

[0013] Where H all =(H1,H2,…,H 12 ) represents the output of the encoder's hidden layer. H i This represents the output of the i-th hidden layer. The encoding layer is the encoder structure of the Transformer.

[0014] Dynamic routing: For the output of the hidden layer, if we want to route the last three layers, we take the output of the last three layers, i.e., H. 10 H 11 H 12 By taking these three as inputs to the dynamic routing module, we can obtain:

[0015] H = Dynamic_Routing(H 10 H 11 H 12 )

[0016] H represents the output of the dynamic routing module.

[0017] After dynamic routing, a linear layer is added, with an output dimension of 2.

[0018] L = Linear(H)

[0019] Where L represents the output of the linear layer.

[0020] Answer voting: Input L into the pre-output layer.

[0021] A′,P=pre_output(L)

[0022] Where A′ represents the pre-output answer, and P represents the corresponding predicted score.

[0023] Enter A′ into the answer voting module.

[0024] V = answer_voting(A′)

[0025] Where V represents the corresponding voting score.

[0026] S = w1 * P + w2 * V

[0027] Where S represents the final answer score. The final answer is then obtained through the output layer.

[0028] A = output(S)

[0029] Where A represents the final answer.

[0030] Pre-training the model on the public dataset SQuAD can improve its performance. Then, fine-tuning the model on the BioASQ biomedical dataset makes it ready for testing on the BioASQ test set.

[0031] Step 3, Model Testing. Input the test set into the model, start model prediction, and obtain the test results after the model runs.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] The biomedical field requires specific domain knowledge, and existing artificial intelligence technologies, such as question answering, have performance limitations that restrict their application. This invention employs dynamic routing and answer voting techniques, combined with pre-training on publicly available corpora, significantly improving the model's performance in biomedical question answering. This will benefit the application of NLP question answering technology in the biomedical field.

[0034] (1) This invention employs a dynamic routing mechanism that adaptively and fully utilizes the hidden layer knowledge of the pre-trained model. A dynamic routing algorithm is designed that can dynamically assign appropriate weights to multiple hidden layers and automatically adjust the weights, rather than using fixed weights. This algorithm can reasonably route multiple hidden layers, thereby making full use of the knowledge learned by the pre-trained model on a large-scale corpus.

[0035] (2) The answer voting strategy of this invention can better select the optimal answer. An answer voting module is designed to calculate the voting score, which takes into account the similarity between answers. The answer score is no longer just a predicted score, but is composed of both the predicted score and the voting score. This implicit information can help the model generate a more suitable answer. Attached Figure Description

[0036] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0037] Figure 2 This is a general framework diagram of the method according to an embodiment of the present invention;

[0038] Figure 3 This is an introductory diagram of the dynamic routing method of the present invention;

[0039] Figure 4 This is a comparison chart with the sample predicted by the baseline model. Detailed Implementation

[0040] To enhance understanding of the present invention, the invention will be further illustrated below with reference to specific embodiments.

[0041] Example 1: A biomedical extractive question answering method based on dynamic routing and answer voting. This method first requires collecting a biomedical dataset, then processing it into an extractive question answering task format, and collecting publicly available datasets for pre-training. The model is pre-trained first, and then fine-tuned and tested on a biomedical task. The specific implementation steps of this invention are as follows:

[0042] Step 1: Collect and preprocess the datasets. First, we collected the BioASQfactoid4b, BioASQfactoid5b, BioASQfactoid6b, and BioASQfactoid9b biomedical datasets, which can be found in the BioASQ challenge. Then, we collected the publicly available SQuAD dataset. The SQuAD dataset was used for pre-training the model, while the BioASQ datasets were processed into an extractive question-answering task format for model training and testing.

[0043] Step 2, Model Training. This step can be divided into the following sub-steps:

[0044] Sub-step 2-1: The model input sequence, paragraph + question, is first transformed into an embedding vector E through an embedding layer, and then the embedding vector is input into the encoding layer.

[0045] H all =Encoder(E)

[0046] Where H all =(H1,H2,…,H 12 ) represents the output of the encoder's hidden layer. H i This represents the output of the i-th hidden layer.

[0047] Encoding Layer: The encoding layer is the encoder structure of the Transformer, and its key component is the attention mechanism. For a given query matrix Q, key matrix K, and value matrix V, the attention mechanism is calculated as follows:

[0048]

[0049] MultiHead(Q,K,V)=[head1,…,head h W o

[0050] Among them, W i q W i k W i v W o For learnable parameters, is the scaling factor, and h is the number of attention heads. A multi-head attention layer followed by a feedforward layer forms an encoder structure.

[0051] Sub-step 2-2: For the output of the hidden layer, assuming we want to route the last three layers, we take the output of the last three layers, i.e., H. 10 H 11 H 12 By taking these three as inputs to the dynamic routing module, we can obtain:

[0052] H = Dynamic_Routing(H 10 H 11 H 12 )

[0053] H represents the output of the dynamic routing module.

[0054] Introduction to the dynamic routing algorithm: First, we define three initial weights b1, b2, and b3, and then define the number of iterations r. Iteration: Softmax is applied to b1, b2, and b3 to calculate three routing values ​​k1, k2, and k3, so that the sum of the routing values ​​k1, k2, and k3 is 1. Then, the output vectors H12, H11, and H10 of the last three hidden layers are flattened into one-dimensional vectors, resulting in H12_f, H11_f, and H10_f. The weighted sum x is calculated as k1*H12_f + k2*H11_f + k3*H10_f. The result is y obtained through the non-linear function Squash. Then, b1, b2, and b3 are updated. y is multiplied by the transpose of H12_f and added to the original b1 to obtain the new b1. Similarly, new b2 and b3 are obtained. After the iteration is complete, H12, H11, and H10 are multiplied by their respective updated routing values ​​and summed to obtain the output after dynamic routing, i.e., k1*H12+k2*H11+k3*H10.

[0055]

[0056] Where Z represents the input vector.

[0057] Sub-steps 2-3 follow the dynamic routing with a linear layer whose output dimension is 2.

[0058] L = Linear(H)

[0059] Where L represents the output of the linear layer.

[0060] Sub-steps 2-4, answer voting: Input L into the pre-output layer.

[0061] A′,P=pre_output(L)

[0062] Where A′ represents the pre-output answer, and P represents the corresponding predicted score.

[0063] Enter A′ into the answer voting module.

[0064] V = answer_voting(A′)

[0065] Where V represents the corresponding voting score.

[0066] S = w1 * P + w2 * V

[0067] Where S represents the final answer score. The final answer is then obtained through the output layer.

[0068] A = output(S)

[0069] Where A represents the final answer.

[0070] For any x i and x j The formula for calculating the answer vote score is as follows:

[0071]

[0072] Where |x i ∩x j | represents x i and x j The total number of words in between. |x i | represents x i The number of words. N represents the number of answers in each group.

[0073] Step 3, Model Testing. We first pre-tuned the model on the SQuAD dataset, selecting a checkpoint with good performance on SQuAD as the initial checkpoint for fine-tuning on BioASQ. Then, after fine-tuning on BioASQ, we tested it on the test set. To demonstrate the advantages of our model, we also attached... Figure 4 We compared the predicted samples of our model with those of the baseline model.

[0074] In summary, this invention employs a dynamic routing mechanism to adaptively and fully utilize the hidden layer knowledge of the pre-trained model. We designed a dynamic routing algorithm that can dynamically assign appropriate weights to multiple hidden layers and automatically adjust these weights, rather than using fixed weights. This algorithm can reasonably route multiple hidden layers. This invention proposes an answer voting strategy to better select the optimal answer. An answer voting module is designed to calculate voting scores, taking into account the similarity between answers. The answer score is no longer just a predicted score, but is composed of both the predicted score and the voting score. This implicit information helps the model generate more appropriate answers.

[0075] It should be noted that the above embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

Claims

1. A biomedical extractive question-answering method based on dynamic routing and answer voting, characterized in that, The method includes the following steps: Step 1: Collect and preprocess the dataset. Step 2, Model Training Step 3, Model Testing; Step 1 involves collecting and preprocessing the dataset. Specifically, BioASQfactoid4b, BioASQfactoid5b, BioASQfactoid6b, and BioASQfactoid9b biomedical datasets were collected first. Then, the SQuAD public dataset was collected. The SQuAD dataset was used to pre-train the model. The BioASQ dataset was processed into an extractive question-answering task format for model training and testing. Step 2, Model Training: First, the model is pre-trained on the SQuAD general corpus, using a mechanism of dynamically reducing the learning rate during pre-training. Then, it is fine-tuned on the BioASQ dataset, as detailed below: Sub-step 2-1: The model input sequence, paragraph, and question are first transformed into an embedding vector E through an embedding layer, and then the embedding vector is input into the encoding layer. H all =Encoder(E) Where H all =(H1,H2,…,H 12 H represents the output of the encoder's hidden layer. i This represents the output of the i-th hidden layer; Encoding Layer: The encoding layer is the encoder structure of the Transformer. A crucial part of it is the attention mechanism. For a given query matrix Q, key matrix K, and value matrix V, the attention mechanism is calculated as follows: MultiHead(Q,K,V)=[head1,…,head h ]W o Among them, W i q W i k W i v W o For learnable parameters, is the scaling factor, h is the number of attention heads, and a multi-head attention layer followed by a feedforward layer forms an encoder structure; Sub-step 2-2: For the output of the hidden layer, assuming we want to route the last three layers, take the output of the last three layers, i.e., H. 10 H 11 H 12 By taking these three as inputs to the dynamic routing module, we can obtain: H=Dynamic_Routing(H 10 ,H 11 ,H 12 ) Where H represents the output of the dynamic routing module; Introduction to the dynamic routing algorithm: First, define three initial weights b1, b2, and b3, and then define the number of iterations r. Iteration: Calculate three routing values ​​k1, k2, and k3 using softmax on b1, b2, and b3. The sum of these routing values ​​k1, k2, and k3 is 1. Then, flatten the output vectors H12, H11, and H10 from the last three hidden layers into one-dimensional vectors, obtaining H12_f, H11_f, and H10_f. Calculate the weighted sum x. That is, k1*H12_f + k2*H11_f + k3*H10_f. y is obtained through the nonlinear function Squash, then b1, b2, and b3 are updated. y is multiplied by the transpose of H12_f and added to the original b1 to obtain the new b1. Similarly, new b2 and b3 are obtained. After iteration, H12, H11, and H10 are multiplied by their respective updated routing values ​​and summed to obtain the output after dynamic routing, i.e., k1*H12 + k2*H11 + k3*H10. Sub-steps 2-3 follow the dynamic routing with a linear layer whose output dimension is 2. L = Linear(H) Where L represents the output of the linear layer. Sub-steps 2-4, answer voting: Input L into the pre-output layer. A′,P=pre_output(L) Where A′ represents the pre-output answer, and P represents the corresponding predicted score. Enter A′ into the answer voting module. V = answer_voting(A′) Where V represents the corresponding voting score. S = w1 * P + w2 * V Where S represents the final answer score, and the final answer is obtained through the output layer. A = output(S) Where A represents the final answer. For any x i and x j The formula for calculating the answer vote score is as follows: Where |x i ∩x j | represents x i and x j The total number of words between them, |x i | represents x i The number of words, where N represents the number of answers in each group; Step 3, Model Testing: Input the test set into the model, start the model prediction, and obtain the test results after the model runs.

Images