Medical long text question answering method and system based on state space model

Abstract

The present invention belongs to the field of medical question-answering technology, and provides a medical long text question-answering method and system based on a state-space model, including constructing a medical question-answering model based on a state-space model; inputting training data into the medical question-answering model, encoding multi-document knowledge, obtaining a representation of multi-document knowledge, selecting the representation of multi-document knowledge in combination with medical-related issues, obtaining an output answer, and using distillation learning to optimize the output answer; defining a loss function, optimizing the parameters of the model, obtaining a trained medical question-answering model; and using the trained medical question-answering model to perform medical question-answering. The present invention uses a state-space model to encode long text knowledge into word-level vectors, which can cope with ultra-long text inputs, and adopts a distillation-based learning strategy to optimize the generation of answers, improve the model's ability to understand long texts, and prevent the model from drastically decreasing performance when the input length is too long.

Description

Technical Field

[0001] The present disclosure relates to the technical field of medical question answering, and in particular to a medical long text question answering method and system based on a state space model. Background Art

[0002] Existing medical question-answering work can be divided into two categories: direct question-answering and retrieval-based question-answering. In direct question-answering work, the model often directly remembers medical knowledge. When a question is input into the model, the model relies on the medical knowledge it remembers to give a response. However, when medical knowledge is updated, this type of method often requires retraining the model, and the update cost is high. In a retrieval-based question-answering system, the model responds based on the medical knowledge retrieved. In this type of work, the model does not need to remember the latest medical knowledge, but only needs to learn certain medical common sense and generate answers based on the retrieved knowledge. Currently, retrieval-based medical question-answering systems tend to dominate the mainstream.

[0003] However, this type of system often assumes that the required knowledge must be in the retrieval results, and requires that the length of the external knowledge required is less than the length that the model can input. The existence of this type of assumption limits the application of the model. Since this type of assumption requires that the required knowledge must be in the retrieval results, and the performance of the retrieval system cannot meet this requirement, generally the first 5 or 10 documents retrieved are used as external knowledge, and using multiple documents as external knowledge is likely to exceed the input length range of the model. On the other hand, even if only one document is used as external knowledge, when the document is too long, such as long medical documents or papers, it still cannot be input into the model. Therefore, the application of this type of model is also very limited. Summary of the invention

[0004] In response to the above-mentioned defects, the present invention proposes a medical long text question and answer method and system based on a state-space model, which uses the state-space model to encode long text knowledge into word-level vectors, can cope with ultra-long text input, and adopts a distillation-based learning strategy to optimize the generation of answers, improve the model's ability to understand long texts, and prevent the model's performance from degrading sharply when the input length is too long.

[0005] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0006] The first aspect of the present disclosure provides a medical long text question answering method based on a state space model, comprising the following steps:

[0007] Construct a medical question-answering model based on the state-space model;

[0008] Input the training data into the medical question-answering model, encode the multi-document knowledge, obtain the representation of the multi-document knowledge, select the representation of the multi-document knowledge in combination with medical-related questions, obtain the output answer, and use distillation learning to optimize the output answer;

[0009] Define the loss function, optimize the model parameters, and obtain a trained medical question-answering model;

[0010] Use the trained medical question-answering model to conduct medical question-answering.

[0011] As a further implementation, the training data is constructed in two ways, one is based on existing question-answering data, and the other is based on existing documents.

[0012] As a further implementation, existing question-answering data includes relevant documents, questions, and answers;

[0013] The existing documents are various collected medical documents.

[0014] As a further implementation manner, the multi-document knowledge is a mixed document of a given document and a retrieved document.

[0015] As a further implementation manner, the multi-document knowledge is a mixed document of the target document and the retrieved document.

[0016] As a further implementation, distillation learning is used to optimize the output answer, specifically:

[0017] The knowledge of the output of a given retrieval document and question is distilled into the output obtained by inputting the shuffled retrieval document as the model retrieval document input, and the knowledge of the output obtained by inputting only the target document as the model retrieval document input is distilled into the output of a given retrieval document and question.

[0018] As a further implementation, the loss functions include cross entropy loss and distillation loss.

[0019] The second aspect of the present disclosure provides a medical long text question answering system based on a state space model, comprising:

[0020] The model building module is configured to: build a medical question-answering model based on a state-space model;

[0021] The model training module is configured to: input the training data into the medical question-answering model, encode the multi-document knowledge, obtain the representation of the multi-document knowledge, select the representation of the multi-document knowledge in combination with medical-related questions, obtain the output answer, and use distillation learning to optimize the output answer;

[0022] The parameter optimization module is configured to: define a loss function, optimize the parameters of the model, and obtain a trained medical question-answering model;

[0023] The model output module is configured to use the trained medical question-answering model to perform medical question-answering.

[0024] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the medical long text question and answer method based on the state space model described in the first aspect of the present disclosure are implemented.

[0025] The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the medical long text question and answer method based on the state space model described in the first aspect of the present disclosure are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The medical long text question-answering method based on the state-space model proposed in the present invention uses the state-space model to encode the long text knowledge into word-level vectors, which can allow ultra-long external knowledge as input, and effectively solve the problem of excessive external knowledge in the medical field. A training strategy based on distillation learning is designed to optimize the generation of answers. In the case of ultra-long texts, it can further enhance the model's ability to mine and understand knowledge, and ensure the logic and accuracy of the answers. Prevent the model from drastically degrading performance when the input length is too long. The entire process of the present invention does not require the introduction of additional annotations, so that the method can be widely used.

[0028] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0030] Figure 1 is a flow chart of the medical long text question answering method based on the state space model disclosed in the present invention;

[0031] Figure 2 It is a schematic diagram of the medical question-and-answer model disclosed herein when in application. DETAILED DESCRIPTION

[0032] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0034] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0035] Embodiment 1

[0036] like Figure 1 As shown, the first embodiment of the present disclosure provides a medical long text question answering method based on a state space model, which mainly includes the following steps:

[0037] S1 builds a medical question-answering model based on the state-space model;

[0038] S2 inputs the training data into the medical question-answering model, encodes the multi-document knowledge, obtains the representation of the multi-document knowledge, selects the representation of the multi-document knowledge in combination with medical-related questions, obtains the output answer, and uses distillation learning to optimize the output answer;

[0039] S3 defines the loss function, optimizes the model parameters, and obtains a trained medical question-answering model;

[0040] S4 uses the trained medical question-answering model to conduct medical question-answering.

[0041] A medical question-answering model based on the state-space model is constructed, and the training data is input into the medical question-answering model to encode multi-document knowledge and obtain a representation of multi-document knowledge.

[0042] The model adopts an encoding-decoding framework, which consists of two layers: encoding layer and decoding layer. The encoding layer is a pre-trained multi-layer state space model layer that can efficiently handle ultra-long inputs and encode long text knowledge into word-level vectors. The decoding layer is implemented through a high-performance GPT-2 decoder, which uses a cross-attention mechanism to select knowledge representations and generate the answer to the final question.

[0043] The encoding layer is a pre-trained multi-layer state space model layer, given the concatenated long multi-document knowledge ,in, is included The first word input documents, calculate the knowledge for each word Representation:

[0044]

[0045] in, It is of the documents words, Indicates the concatenation of words, is the representation of all document words, with a total length of , the dimension is . It is a multi-layer state space model, which is implemented by Mamba and can directly use its pre-trained model parameters.

[0046] The decoding layer is implemented by a high-performance GPT-2 decoder. Given a medical-related problem , the decoder decodes the problem, that is, encodes all the words in the problem, and predicts the representation of the next word through the representation of the last word, until the next word is the end symbol <eos>:

[0047]

[0048]

[0049]

[0050]

[0051] in, The output answer is Output the answer words. is a linear layer, The hidden representation of the dimension is projected onto the vocabulary dimension , and then through the normalized exponential function Converted into a probability distribution , and through the sampling function Select words to get . For the Words , which is expressed by , generated words Splicing Feed get. The cross attention mechanism is used internally to represent the external knowledge obtained by the encoder. Make a selection and finally calculate .

[0052] Training data is constructed in two ways: one is based on existing question-and-answer data, and the other is based on existing documents. Existing question-and-answer data includes relevant documents, questions, and answers. Existing documents are various types of collected medical literature.

[0053] When the first method is used to construct training data, the multi-document knowledge is a mixed document of the given document and the retrieved document. When the second method is used to construct training data, the multi-document knowledge is a mixed document of the target document and the retrieved document.

[0054] For the construction method based on existing question and answer data, first collect the existing question and answer data and build a knowledge base based on the documents in the question and answer samples; then, train an encoding-based document retrieval model to retrieve documents related to the questions given the questions in the document samples; finally, construct question and answer data based on ultra-long text by mixing the target documents in the question and answer samples with the retrieved documents.

[0055] For the existing document construction method, first, various medical literature data are collected to build a knowledge base; then, due to the lack of question and answer data, question and answer data are constructed based on ChatGPT using instruction construction; then, an encoding-based document retrieval model is trained to retrieve documents related to the questions given the questions in the document samples; finally, the target documents in the question and answer samples are mixed with the retrieved documents to construct question and answer data based on ultra-long texts.

[0056] For the above two methods, the construction of the knowledge base and the training of the retrieval model can be combined to further simplify the data construction process.

[0057] The specific steps are as follows:

[0058] Step 1: Collect existing question-and-answer data and medical literature data to build a knowledge base. By collecting existing short-document medical question-and-answer data and downloading public medical literature data, we get the following samples:

[0059] Short document medical question answering data. Each sample Included Documentation ,question ,Answer .

[0060] Public medical literature data. Each sample Included Documentation .

[0061] Step 2: Construct question-answer data based on ChatGPT using instruction construction. First, Slice the file and randomly select one of the slices and send it to ChatGPT with the prompt. The prompt is as follows:

[0062] "Given a document #{document slice}#, please generate a document-based question-answer pair, requiring the question to be closely related to the document and the answer to be derived from the document.

[0063] Due to the randomness of ChatGPT generation, we can get multiple corresponding questions [ ] and the corresponding answer .

[0064] Finally, the two types of data are combined to obtain training data in the form of .

[0065] Step 3: Train the document retrieval model. First, use the pre-trained document retrieval model BGE (BAAI General Embedding) for fine-tuning, and the fine-tuning loss is set as follows:

[0066]

[0067] in, It represents the representation of the question, target document, and non-target document obtained by random sampling after encoding by the BGE model. Represents the inner product operation of vectors.

[0068] Step 4: Based on the document retrieval model, construct medical long text training data. For each sample, given the question ,knowledge base , retrieve the relevant documents of topk through the document retrieval model If the relevant documents Does not contain target document , then remove documents, and the target document Insert into Finally, we get the training data .

[0069] Combined with medical-related questions, the representation of multi-document knowledge is selected to obtain the output answer, and distillation learning is used to optimize the output answer.

[0070] Previous work has shown that when the output is too long, the model tends to ignore the content in the middle of the document. Therefore, two distillation learning losses are designed to force the model to focus on all valid content of long documents. First, the model is ensured to always focus on the target document by requiring the output of a given retrieval document and question to be as consistent as possible with the input after randomly shuffling the retrieval document, as follows:

[0071]

[0072]

[0073] in, This is the output of the medical question-answering model built in step 1 before sampling. To shuffle the function The output obtained by shuffling the retrieved documents as the input of the model retrieval documents, for KL Divergence loss function is used to convert The knowledge is distilled into middle.

[0074] Secondly, the model's interference with large amounts of noise in long texts is reduced by requiring the model output for a given retrieval of multiple documents to be consistent with the model output for a given target document, as follows:

[0075]

[0076]

[0077] in, The output obtained by inputting only the target document as the input of the model retrieval document. The knowledge should be distilled into In Need and Be consistent.

[0078] Through these two distillation tasks, the model can further improve its ability to accurately mine knowledge from very long texts and its ability to generate answers based on knowledge.

[0079] Define the loss function, optimize the model parameters, and obtain a trained medical question-answering model.

[0080] The basic training method is to train through cross entropy loss:

[0081]

[0082]

[0083] in, The medical question-answering model based on the state-space model constructed for step 1.

[0084] Finally, the loss function is as follows:

[0085]

[0086] Use the trained medical question-answering model to conduct medical question-answering.

[0087] After the training is completed, the weight factors and model parameters are all fixed. At this point, the model can be applied to actual medical question-and-answer scenarios.

[0088] like Figure 2 As shown in the figure, given a medical question and retrieved documents, the conversation context and the retrieved documents or user-given documents are fed into the model. The model will encode the retrieved documents through the encoding layer, filter the document knowledge through the cross-attention layer, and finally generate the answer to the question through the decoding layer, and output the answer predicted by the model.

[0089] Embodiment 2

[0090] Embodiment 2 of the present disclosure provides a medical long text question answering system based on a state space model, including:

[0091] The model building module is configured to: build a medical question-answering model based on a state-space model;

[0092] The model training module is configured to: input the training data into the medical question-answering model, encode the multi-document knowledge, obtain the representation of the multi-document knowledge, select the representation of the multi-document knowledge in combination with medical-related questions, obtain the output answer, and use distillation learning to optimize the output answer;

[0093] The parameter optimization module is configured to: define a loss function, optimize the parameters of the model, and obtain a trained medical question-answering model;

[0094] The model output module is configured to use the trained medical question-answering model to perform medical question-answering.

[0095] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0096] Embodiment 3

[0097] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the medical long text question and answer method based on the state space model as described in Embodiment 1 of the present disclosure are implemented.

[0098] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0099] Embodiment 4

[0100] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a medical long text question-and-answer method based on a state-space model as described in Embodiment 1 of the present disclosure are implemented.

[0101] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0102] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.< / eos>

Claims

1. A medical long text question answering method based on a state space model, characterized in that: The following steps are involved: Construct a medical question-answering model based on the state-space model; The medical question-answering model adopts an encoding-decoding framework, including an encoding layer and a decoding layer; The encoding layer is a pre-trained multi-layer state space model layer. ,in, is included The first word input documents, calculate the knowledge for each word Representation: in, It is of the documents words, Indicates the concatenation of words, is the representation of all document words, with a total length of , the dimension is It is a multi-layer state space model; The decoding layer is implemented by a high-performance GPT-2 decoder; given medical-related problems , the decoder decodes the question, encodes all the words in the question, and predicts the representation of the next word through the representation of the last word until the next word is the end symbol <eos> ；< / eos> Input the training data into the medical question-answering model, encode the multi-document knowledge, obtain the representation of the multi-document knowledge, select the representation of the multi-document knowledge in combination with medical-related questions, obtain the output answer, and use distillation learning to optimize the output answer; The training data is constructed in two ways: one is based on existing question-answering data, and the other is based on existing documents; Merge the knowledge base construction and retrieval model training to further simplify the data construction process, including: Collect existing question-and-answer data and medical literature data to build a knowledge base; the samples obtained are: short document medical question-and-answer data, each sample Included Documentation ,question ,Answer ; Public medical literature data, each sample Included Documentation ; Based on ChatGPT, we construct question-answering data by using instructions to obtain multiple corresponding questions [ ] and the corresponding answer ; Combine the two types of data to get training data in the form of ; Train the document retrieval model and fine-tune it using the pre-trained document retrieval model BGE; Construct medical long text training data based on document retrieval model; Define the loss function, optimize the model parameters, and obtain a trained medical question-answering model; Use the trained medical question-answering model to conduct medical question-answering.

2. The medical long text question answering method based on the state space model as claimed in claim 1, characterized in that: Existing question-answering data includes relevant documents, questions, and answers; The existing documents are various collected medical documents.

3. The medical long text question answering method based on the state space model as claimed in claim 1, characterized in that: Multi-document knowledge is a mixture of given documents and retrieved documents.

4. The medical long text question answering method based on the state space model as claimed in claim 1, characterized in that: Multi-document knowledge is a mixed document of the target document and the retrieved document.

5. The medical long text question answering method based on the state space model as claimed in claim 1, characterized in that: Use distillation learning to optimize the output answer, specifically: The knowledge of the output of a given retrieval document and question is distilled into the output obtained by inputting the shuffled retrieval document as the model retrieval document input, and the knowledge of the output obtained by inputting only the target document as the model retrieval document input is distilled into the output of a given retrieval document and question.

6. The medical long text question answering method based on the state space model as claimed in claim 1, characterized in that: The loss functions include cross entropy loss and distillation loss.

7. A medical long text question answering system based on a state space model, characterized in that: include: The model building module is configured to: build a medical question-answering model based on a state-space model; The medical question-answering model adopts an encoding-decoding framework, including an encoding layer and a decoding layer; The encoding layer is a pre-trained multi-layer state space model layer. ,in, is included The first word input documents, calculate the knowledge for each word Representation: in, It is of the documents words, Indicates the concatenation of words, is the representation of all document words, with a total length of , the dimension is It is a multi-layer state space model; The decoding layer is implemented by a high-performance GPT-2 decoder; given medical-related problems , the decoder decodes the question, encodes all the words in the question, and predicts the representation of the next word through the representation of the last word until the next word is the end symbol <eos> ；< / eos> The model training module is configured to: input the training data into the medical question-answering model, encode the multi-document knowledge, obtain the representation of the multi-document knowledge, select the representation of the multi-document knowledge in combination with medical-related questions, obtain the output answer, and use distillation learning to optimize the output answer; The training data is constructed in two ways: one is based on existing question-answering data, and the other is based on existing documents; Merge the knowledge base construction and retrieval model training to further simplify the data construction process, including: Collect existing question-and-answer data and medical literature data to build a knowledge base; the samples obtained are: short document medical question-and-answer data, each sample Included Documentation ,question ,Answer ; Public medical literature data, each sample Included Documentation ; Based on ChatGPT, we construct question-answering data by using instructions to obtain multiple corresponding questions [ ] and the corresponding answer ; Combine the two types of data to get training data in the form of ; Train the document retrieval model and fine-tune it using the pre-trained document retrieval model BGE; Construct medical long text training data based on document retrieval model; The parameter optimization module is configured to: define a loss function, optimize the parameters of the model, and obtain a trained medical question-answering model; The model output module is configured to use the trained medical question-answering model to perform medical question-answering.

8. A medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the medical long text question-answering method based on the state-space model as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the medical long text question and answer method based on the state space model as described in any one of claims 1 to 6 are implemented.

Images