A question-answering method for large medical models based on memory and reinforcement learning optimization

By introducing memory plug-ins and reinforcement learning mechanisms in the medical question-and-answer system, coding patient medical record information and optimizing models, the problem of insufficient personalized answers and real-time optimization capabilities of existing systems is solved, and the answer quality and model performance are improved.

CN118428412BActive Publication Date: 2024-11-01ANHUI PROVINCIAL HOSPITAL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410875622.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-01
Estimated Expiration
2044-07-02

AI Technical Summary

Technical Problem

The existing medical Q&A system lacks personalized characteristics when dealing with patient problems, and the model cannot be optimized in real time based on patient feedback in application, resulting in poor performance.

Method used

Using the medical big model question and answer method based on memory and reinforcement learning, the patient's medical record information is encoded through memory plug-ins, and interact with the patient's problem characteristics to generate personalized answers; at the same time, reinforcement learning is used to optimize the model based on patient feedback signals, and update parameters to improve the model performance.

Benefits of technology

The generation of personalized answers and real-time optimization of model performance is achieved, and the quality of answers during the interaction between the model and the patient is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118428412B_ABST
    Figure CN118428412B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of medical question and answer, and discloses a medical big model question and answer method based on memory and reinforcement learning optimization, comprising: using a memory plug-in to encode a structured patient medical record text to obtain a memory vector that records key information of the patient medical record; inputting a question text into the medical big model to obtain a patient question encoding feature; using an interaction module to realize the interaction between the memory vector and the patient question encoding feature to obtain an enhanced patient question encoding feature; using the medical big model for processing to obtain a personalized answer; comparing the personalized answer with a manually labeled answer, and using a cross entropy loss function to obtain the loss; based on the loss, updating the trainable parameters of the medical big model, the memory plug-in and the interaction module in the medical question and answer model through back propagation and gradient descent algorithms to realize the training of the medical question and answer model; the present invention introduces patient medical record information to solve the problem of difficulty in generating personalized answers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical question-answering technology, and in particular to a large medical model question-answering method based on memory and reinforcement learning optimization. Background Art

[0002] Automatic question answering in the medical field aims to generate targeted professional answers to patients' medical problems and meet their personalized needs. At present, with the development of big models and the expansion from general fields to professional fields, some studies have used medical big models to help interact with patients and answer their questions. In this process, current methods encounter several challenges. First, the current methods use standard big model encoding and decoding processes to process patient questions and generate answers. There is a lack of effective use of patient information, which makes the generated content lack patient personalized characteristics and difficult to meet the needs of generating personalized answers. Secondly, the current medical question answering big model adopts a model of separation of training and use, and lacks an effective mechanism for optimization in application. Once the model is trained, it is impossible to optimize the model effect in real time according to patient feedback in the application, resulting in unsatisfactory model performance. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a medical large model question answering method based on memory and reinforcement learning optimization.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A medical large model question answering method based on memory and reinforcement learning optimization, the medical question answering model used includes a medical large model, a memory plug-in, and an interactive module; specifically includes the following steps:

[0006] Step 1: Use the memory plug-in to structure the patient medical record text Encode to obtain M memory vectors that record the key information of the patient's medical record ;

[0007] Step 2: Use the patient question text for training Input to the medical large model composed of L layers of Transformer decoder, and get the first Layer Transformer decoder outputs the patient's problem encoding features ,in ;

[0008] Step 3: Use the interactive module to implement the memory vector Coding characteristics of patient problems Interaction with enhanced patient question coding features ;

[0009] Step 4: Use the remaining medical model Layer Transformer decoder processing , using the standard Transformer decoding process to obtain personalized answers to the patient's question text ;

[0010] Step 5: Personalize the answer With manual annotation answers By comparison, using the cross entropy loss function, we get the loss ;

[0011] Step 6: Loss-based , through back propagation and gradient descent algorithms, the trainable parameters of the medical big model, memory plug-in and interaction module in the medical question-answering model are updated to realize the training of the medical question-answering model;

[0012] Step 7: Patient medical records and the actual patient question text Input into the trained medical question-answering model and output personalized answers .

[0013] Furthermore, the memory plug-in includes a standard multi-layer Transformer encoder and a standard convolutional neural network; step one specifically includes the following steps:

[0014] S11, using a standard multi-layer Transformer encoder, transforms the structured patient medical record text Coding for medical record representation :

[0015] ;

[0016] Represents the Transformer encoder;

[0017] S12, using standard convolutional neural network to represent medical records Encode and pass the result through the maximum pooling layer to obtain M memory vectors that record the key information of the patient's medical record :

[0018] ;

[0019] represents a convolutional neural network, Represents a maximum pooling layer.

[0020] Furthermore, in step 3, the memory vector is realized by using the interactive module Coding characteristics of patient problems Interaction with enhanced patient question coding features , specifically including the following steps:

[0021] S31, use the fully connected layer to encode the patient's problem features Mapping to query representation , using two different fully connected layers to store M memory vectors that record key information of patient medical records Mapped to key representations And value representation ;

[0022] S32, using query representation and the mth bond characterization Calculate the corresponding weight , represents the mth value To weight, , so that the judgment value represents the contribution of the corresponding patient medical record information to answering the question;

[0023] ;

[0024] S33, repeat step S32 for each value of m to obtain the weights corresponding to all values ​​of m ;

[0025] S34, weight Applied to the corresponding value representation Perform weighted average on the memory value vector ;

[0026] ;

[0027] S35, the memory value vector Coding characteristics of patient problems Add together to get the enhanced patient problem coding features .

[0028] Furthermore, the method also includes a process of optimizing a medical question-answering model based on reinforcement learning; the process of optimizing a medical question-answering model based on reinforcement learning calculates rewards according to feedback signals from patients, and optimizes the medical question-answering model through a reinforcement learning method, specifically including the following steps:

[0029] A1, generating personalized answers Afterwards, get the patient's feedback signal ; The feedback signal includes a positive feedback signal and a negative feedback signal; when the patient is satisfied with the generated personalized answer Satisfied, and patients follow After treatment, the condition improves, giving a positive feedback signal; when the patient responds to the generated personalized answer Dissatisfied, so take other treatment options, or patients follow After treatment, the condition does not improve, giving a negative feedback signal;

[0030] A2, build instance tuple ;

[0031] A3, the patient's medical record text and the actual patient question text Input into the trained medical question-answering model to generate new personalized answers ;

[0032] A4, Use personalized answers With new personalized answers Similarities between , and feedback signal As a basis, calculate the reward R; It is a text semantic comparison function;

[0033] When the feedback signal When it is a positive feedback signal, Should be close , so The closer the value of is to 1, the greater the reward R is:

[0034] ;

[0035] When the feedback signal When it is a negative feedback signal, introduce human-annotated answers written by doctors , Should deviate and close ; therefore when The closer the value is to 1, The further away from 1 the value is, the greater the reward R is:

[0036] ;

[0037] and is a predefined positive real number, and ;

[0038] A5. After obtaining the reward R, the standard reinforcement learning rejection sampling method is used to optimize the medical question-answering model, that is, multiple instance tuples and corresponding rewards are collected, the instance tuples are sorted according to the rewards, and the instance tuple with the highest reward is selected as the new supervision data. Then, the supervised fine-tuning method is used to update the parameters of the medical question-answering model, that is, to update the parameters of the medical large model, memory plug-in and interaction module.

[0039] Compared with the prior art, the beneficial technical effects of the present invention are:

[0040] 1. The present invention introduces patient medical record information to solve the problem that existing methods are difficult to generate personalized answers.

[0041] 2. Through reinforcement learning, the model can be optimized based on patient feedback in actual applications, so that the model is always aligned with patient preferences, enhancing the quality of the model's output answers during the interaction between the model and the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the medical question-answering model of the present invention. DETAILED DESCRIPTION

[0043] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0044] The present invention is mainly divided into two parts: a medical question-answering model based on a memory mechanism, and a medical question-answering model optimization based on reinforcement learning. The two parts are introduced below.

[0045] The medical question-answering model based on the memory mechanism includes a medical big model, a memory plug-in, and an interactive module. The memory plug-in is used to encode the structured patient medical record text, the medical big model is used to encode the patient question text, and the interactive module integrates the information encoded by the memory plug-in into the medical big model to guide the medical big model to generate answers. The structural diagram of the medical question-answering model is shown in Figure 1 .

[0046] The following introduces the training process and prediction process of the medical question-answering model.

[0047] The training process is as follows:

[0048] Step 1: Encode structured patient medical record text using the memory plug-in , get M memory vectors that record the key information of the patient's medical record .

[0049] Specifically, the memory plug-in consists of a standard multi-layer Transformer encoder and a convolutional neural network. Step 1 specifically includes the following steps:

[0050] S11, using a standard multi-layer Transformer encoder, transforms the structured patient medical record text Coding for medical record representation :

[0051] ;

[0052] S12, using a standard convolutional neural network (CNN) Encode and pass the result through the maximum pooling layer MaxPooling to obtain M memory vectors that record the key information of the patient's medical record :

[0053] .

[0054] Step 2: Text the patient's question Input the medical big model, which is composed of L layers of Transformer decoder, and get the first Layer Transformer decoder outputs the patient's problem encoding features .

[0055] In a preferred embodiment, the medical macro model may employ ChiMed-GPT.

[0056] Step 3: Use the interactive module to implement the memory vector Coding characteristics of patient problems Interaction with enhanced patient question coding features .

[0057] Step three specifically includes the following steps:

[0058] S31: Use a fully connected layer to encode patient questions Mapping to query representation . Two different fully connected layers are used to store the memory vectors that record the key information of the patient's medical records. Mapped to key representations And value representation .

[0059] S32: Using Query Representation and the mth bond characterization Calculate the corresponding weight , represents the mth value To weight, , so the judgment value represents the contribution of the corresponding patient medical record information to answering the question:

[0060] .

[0061] S33: Repeat step S32 for each value of m to obtain the weights corresponding to all values ​​of m .

[0062] S34: Weight Applied to the corresponding value representation Perform weighted average on the memory value vector ;

[0063] .

[0064] S35: Memory value vector Coding characteristics of patient problems Add together to get the enhanced patient problem coding features .

[0065] Step 4: Use the remaining medical model Layer Transformer decoder processing , using the standard Transformer decoding process to obtain personalized answers to the patient's question text .

[0066] Step 5: Personalize the answer With manual annotation answers By comparison, using the cross entropy loss function, we get the loss .

[0067] Step 6: Loss-based , through back propagation and gradient descent algorithms, the trainable parameters of the medical big model, memory plug-in and interaction module in the medical question-answering model are updated to realize the training of the medical question-answering model;

[0068] Step 7: Patient medical records and the actual patient question text Input into the trained medical question-answering model and output personalized answers .

[0069] Step seven specifically includes the following steps:

[0070] S71: Encoding structured patient medical record text using memory plugins , get M memory vectors that record the key information of the patient's medical record .

[0071] S72: Text the patient's question Input the medical model and get Layer Transformer decoder outputs the patient's problem encoding features .

[0072] S73: Using interactive modules to implement memory vectors Coding characteristics of patient problems Interaction with enhanced patient question coding features .

[0073] S74: Use the remaining medical model Layer Transformer decoder processing , using the standard Transformer decoding process to obtain personalized answers to the patient's question text .

[0074] The optimization of the medical question-answering model based on reinforcement learning is based on the prediction process in step seven, the reward is calculated according to the patient feedback, and the obtained reward is used to optimize the medical question-answering model.

[0075] The optimization of the medical question answering model based on reinforcement learning includes the following steps:

[0076] A1: When generating personalized answers Afterwards, get the patient's feedback signal The feedback signals are divided into two categories. The first category is positive feedback signals, which means that the patient responds to the generated personalized answers. Satisfied, and patients follow After treatment, the condition improved. The second type is negative feedback signal, when the patient Dissatisfied, so take other treatment options, or patients follow After treatment, the condition did not improve significantly.

[0077] A2: Since the above feedback is not a real-time response, It takes some time for patient feedback to be obtained after generation. Therefore, it is necessary to construct an instance tuple , which contains the text of the patient's question , Patient medical records , personalized answers , and feedback signal .

[0078] A3: In the actual optimization process, due to the lag in feedback and the real-time optimization of the medical big model, when obtaining personalized answers for patients, The corresponding feedback signal When the parameters of the medical model may have been updated, it is not feasible to simply rely on personalized answers. and Updating the parameters of a large medical model may deteriorate the model performance. To alleviate this problem, we use the The same input is used as the input of the medical big model and the memory plug-in, guiding the medical big model to generate new personalized answers , participate in calculating rewards, thereby optimizing the model.

[0079] A4: Use personalized answers With new personalized answers Similarities between , and feedback signal As a basis, calculate the reward. is a text semantic comparison function. A text semantic comparison function based on BERT and cosine similarity can be used. When the result is close to 1, it means that the similarity between the two is high, and when it is close to 0, the similarity is low. To fully explain the reward calculation process, the following is The possible values ​​are divided into the following states for discussion.

[0080] When the feedback signal When it is a positive feedback signal, due to the personalized answer It meets the needs of patients, indicating that the medical big model can meet current needs. Keep as close as possible , which is manifested as When the value is close to 1, a higher reward is given to the medical model. The reward R is calculated as follows:

[0081] .

[0082] When the feedback signal When it is a negative feedback signal, due to the generated Failure to meet patient needs indicates that the big medical model is not up to the current needs. There should be controlled deviation from the instance tuple In this scenario, manual annotation of answers is introduced. , the answer can be written by a doctor. Considering that in actual applications, when the automatically generated answer fails to meet the patient's needs, the patient will seek help from a real doctor, and can obtain manually annotated answers in this process. , so get The process will not bring additional labor costs. When calculating rewards, we hope and As close as possible, As far away as possible. The closer the value of is to 1, The further away from 1 the value is, the greater the reward is. The reward R is calculated as:

[0083] ;

[0084] and is a predefined positive real number, and .

[0085] A5: After receiving the reward Finally, the standard reinforcement learning rejection sampling method is used to optimize the medical question answering model, that is, multiple instance tuples and their rewards are collected, the instance tuples are sorted according to the rewards, and the instance tuples with the highest rewards are selected as new supervision data, and then the supervised fine-tuning method is used to update the medical question answering model parameters.

[0086] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0087] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A medical large model question answering method based on memory and reinforcement learning optimization, characterized in that: The medical question-answering model used includes a large medical model, a memory plug-in, and an interactive module; specifically, the following steps are included: Step 1: Use the memory plug-in to structure the patient medical record text Encode to obtain M memory vectors that record the key information of the patient's medical record ; Step 2: Use the patient question text for training Input to the medical large model composed of L layers of Transformer decoder, and get the first Layer Transformer decoder outputs the patient's problem encoding features ,in ; Step 3: Use the interactive module to implement the memory vector Coding characteristics of patient problems Interaction with enhanced patient question coding features ; Step 4: Use the remaining medical model Layer Transformer decoder processing , using the standard Transformer decoding process to obtain personalized answers to the patient's question text ; Step 5: Personalize the answer With manual annotation answers By comparison, using the cross entropy loss function, we get the loss ; Step 6: Loss-based , through back propagation and gradient descent algorithms, the trainable parameters of the medical big model, memory plug-in and interaction module in the medical question-answering model are updated to realize the training of the medical question-answering model; Step 7: Patient medical records and the actual patient question text Input into the trained medical question-answering model and output personalized answers ; The memory plug-in includes a standard multi-layer Transformer encoder and a standard convolutional neural network; step one specifically includes the following steps: S11, using a standard multi-layer Transformer encoder, transforms the structured patient medical record text Coding for medical record representation : ; Represents the Transformer encoder; S12, using standard convolutional neural network to represent medical records Encode and pass the result through the maximum pooling layer to obtain M memory vectors that record the key information of the patient's medical record : ; represents a convolutional neural network, represents the maximum pooling layer; In step 3, the memory vector is realized by using the interactive module Coding characteristics of patient problems Interaction with enhanced patient question coding features , specifically including the following steps: S31, use the fully connected layer to encode the patient's problem features Mapping to query representation , using two different fully connected layers to store M memory vectors that record key information of patient medical records Mapped to key representations And value representation ; S32, using query representation and the mth bond characterization Calculate the corresponding weight , represents the mth value To weight, , so that the judgment value represents the contribution of the corresponding patient medical record information to answering the question; ; S33, repeat step S32 for each value of m to obtain the weights corresponding to all values ​​of m ; S34, weight Applied to the corresponding value representation Perform weighted averaging on the memory value vector ; ; S35, the memory value vector Coding characteristics of patient problems Add together to get the enhanced patient problem coding features .

2. The medical large model question answering method based on memory and reinforcement learning optimization according to claim 1 is characterized in that: It also includes the process of optimizing the medical question-answering model based on reinforcement learning; The process of optimizing the medical question-answering model based on reinforcement learning calculates rewards based on the patient's feedback signals and optimizes the medical question-answering model through reinforcement learning methods. Specifically, the following steps are included: A1, generating personalized answers Afterwards, get the patient's feedback signal ; The feedback signal includes a positive feedback signal and a negative feedback signal; when the patient is satisfied with the generated personalized answer Satisfied, and patients follow After treatment, the condition improves, giving a positive feedback signal; when the patient responds to the generated personalized answer Dissatisfied, so take other treatment options, or patients follow After treatment, the condition does not improve, giving a negative feedback signal; A2, build instance tuple ; A3, the patient's medical record text and the actual patient question text Input into the trained medical question-answering model to generate new personalized answers ; A4, Use personalized answers With new personalized answers Similarities between , and feedback signal As a basis, calculate the reward R; It is a text semantic comparison function; When the feedback signal When it is a positive feedback signal, Should be close , so The closer the value of is to 1, the greater the reward R is: ; When the feedback signal When it is a negative feedback signal, introduce human-annotated answers written by doctors , Should deviate and close ; therefore when The closer the value is to 1, The further away from 1 the value is, the greater the reward R is: ; and is a predefined positive real number, and ; A5. After obtaining the reward R, the standard reinforcement learning rejection sampling method is used to optimize the medical question-answering model, that is, multiple instance tuples and corresponding rewards are collected, the instance tuples are sorted according to the rewards, and the instance tuple with the highest reward is selected as the new supervision data. Then, the supervised fine-tuning method is used to update the parameters of the medical question-answering model, that is, to update the parameters of the medical large model, memory plug-in and interaction module.