Disease auxiliary diagnosis method and device based on reward sorting and online reinforcement learning

Abstract

The invention discloses a disease auxiliary diagnosis method and device based on reward sorting and online reinforcement learning, and relates to the field of medical data processing.The disease auxiliary diagnosis method comprises the steps that a fine adjustment data set and a disease auxiliary diagnosis model are constructed, and the disease auxiliary diagnosis model adopts a pre-trained large language model subjected to preliminary fine adjustment; fine-tuning the disease auxiliary diagnosis model again by adopting a fine-tuning data set and a reward sorting fine-tuning mode, so that the disease auxiliary diagnosis model can be continuously learned and improved through self-generated data under an unsupervised condition to obtain a fine-tuned disease auxiliary diagnosis model; and acquiring multi-modal medical data of a patient to be predicted, and inputting the multi-modal medical data into the disease auxiliary diagnosis model subjected to fine adjustment to obtain a corresponding diagnosis suggestion. According to the method, the problems in the aspects of model alignment, human feedback utilization, generation result control, generalization ability, online learning ability and the like in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of medical data processing, and in particular, to a disease assisted diagnosis method and device based on reward ranking and online reinforcement learning. Background Art

[0002] With the rapid development of deep learning and natural language processing technologies, large pre-trained language models (LLMs) have made breakthrough progress in natural language understanding and generation tasks. LLMs such as the GPT series, BERT, Llama, etc. perform excellently in processing complex language tasks and can understand context and generate high-quality text.

[0003] In the medical field, disease assisted diagnosis models help improve diagnosis efficiency and reduce the burden on doctors. However, medical diagnosis is highly professional and complex, with extremely high requirements for the accuracy, reliability, and interpretability of the models. Traditional medical diagnosis models often rely on rules or shallow machine learning methods and are difficult to handle complex medical knowledge and reasoning processes. Traditional medical diagnosis models are often limited by their single-modal data processing capabilities and cannot effectively integrate text, images, and structured data. At the same time, large language models with multi-modal capabilities often have not been fine-tuned for the medical field, resulting in the accuracy, reliability, and generalization ability of the diagnosis results not meeting the actual clinical needs. In addition, existing model fine-tuning methods such as supervised learning or offline reinforcement learning rely on a large amount of high-quality human-annotated data, which is costly and inefficient. Reinforcement Learning from Human Feedback (RLHF) is a method of optimizing the model through human feedback to make the model output more in line with human expectations.

[0004] The disease assisted diagnosis models of the prior art have the following defects:

[0005] 1. Insufficient model alignment and reliability.

[0006] When the existing disease assisted diagnosis models generate diagnostic suggestions, they may produce inaccurate, unreasonable, or non-compliant medical results. This is mainly due to the lack of in-depth understanding of medical expertise in the models and the lack of effective mechanisms to align the model output with human expectations. The models may make misjudgments in complex cases or provide diagnostic suggestions lacking credibility.

[0007] 2. Low efficiency in utilizing human feedback.

[0008] Traditional RLHF methods require a large amount of human feedback data, which is costly to collect and has low utilization efficiency. Data in the medical field is sensitive and private, making it even more difficult to obtain large-scale human feedback data, which limits the application of RLHF methods in the medical field.

[0009] 3. Lack of effective control over the generated results.

[0010] When existing models generate diagnostic suggestions, they lack effective control over the output results, which may produce error messages, omit important information, or generate inappropriate content. For example, the model may ignore key symptoms or provide suggestions that do not match the patient's condition, affecting the credibility and safety of the diagnosis.

[0011] 4. Limited generalization ability of the model.

[0012] Due to the complexity and diversity of medical data, the performance of the model may fluctuate significantly in different clinical scenarios and patient data. The limited generalization ability of existing models makes them unable to adapt to various disease types and patient groups, and it is difficult to meet the actual application requirements.

[0013] 5. Lack of online learning and continuous improvement capabilities.

[0014] Existing models are usually trained offline and cannot effectively utilize new data and human feedback for online updates and continuous improvement. Knowledge and practices in the medical field are constantly evolving, and the lack of online learning ability in the model makes it difficult to adapt to the rapidly changing medical environment. Summary of the Invention

[0015] The purpose of this application is to propose a disease-assisted diagnosis method and device based on reward ranking and online reinforcement learning for the above-mentioned technical problems.

[0016] In the first aspect, the present invention provides a disease-assisted diagnosis method based on reward ranking and online reinforcement learning, including the following steps:

[0017] Construct a fine-tuning dataset and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned. The disease-assisted diagnosis model is fine-tuned again using the reward ranking fine-tuning method to obtain a fine-tuned disease-assisted diagnosis model, which specifically includes: constructing a trained reward model. During the fine-tuning process of the current round, the multi-modal medical data of each patient in the fine-tuning dataset is respectively input into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round to obtain the output probability of the diagnosis advice of the current round and the output probability of the diagnosis advice of the previous round. The multi-modal medical data of each patient and the diagnosis advice of the current round are input into the trained reward model to obtain a reward score. A total loss function is constructed based on the reward score, the output probability of the diagnosis advice of the current round, and the output probability of the diagnosis advice of the previous round. The disease-assisted diagnosis model of the current round is fine-tuned based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round are respectively used as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round to repeat the above fine-tuning process of the current round, and finally a fine-tuned disease-assisted diagnosis model is obtained;

[0018] Obtain the multi-modal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain the corresponding diagnosis advice.

[0019] Preferably, constructing a trained reward model specifically includes:

[0020] Construct a reward model;

[0021] Input the multi-modal medical data of each patient in the fine-tuning dataset into the disease-assisted diagnosis model to obtain a number of candidate diagnosis advices;

[0022] Score each candidate diagnosis advice by medical experts to obtain a reward label;

[0023] Construct a reward ranking dataset based on the multi-modal medical data of each patient in the fine-tuning dataset and the corresponding each candidate diagnosis advice and reward label;

[0024] Input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnosis advice corresponding to its maximum reward label into the reward model to obtain a positive reward score; input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnosis advice corresponding to its minimum reward label into the reward model to obtain a negative reward score;

[0025] Construct a reward loss function based on the positive reward score and the negative reward score, and train the reward model based on the reward loss function to obtain a trained reward model.

[0026] Preferably, the reward model includes a Transformer model with a two-tower structure, and the Transformer model with a two-tower structure includes a first Transformer encoder, a second Transformer encoder, and a matching score calculation module;

[0027] The multimodal medical data x of the patient and the candidate diagnosis suggestion y are respectively encoded by the first Transformer encoder and the second Transformer encoder, as shown in the following formula: i as follows:

[0028] H x = Encoder1(x);

[0029]

[0030] where H x is the embedded representation of the multimodal medical data x of the patient, is the embedded representation of the candidate diagnosis suggestion y i with a dimension of d, Encoder1 represents the function corresponding to the first Transformer encoder, and Encoder2 represents the function corresponding to the second Transformer encoder;

[0031] The matching score between the embedded representation of the multimodal medical data x of the patient and the embedded representation of the candidate diagnosis suggestion y is calculated by the matching score calculation module to obtain the reward score, as shown in the following formula: i as follows:

[0032]

[0033] where T represents the transposed matrix, Sigmoid represents the Sigmoid function, and r φ (x, y i ) represents the reward score corresponding to the candidate diagnosis suggestion y i .

[0034] Preferably, the reward loss function adopts a contrastive loss function, which is expressed as:

[0035]

[0036] where x represents the multimodal medical data of the patient, y + represents the candidate diagnosis suggestion corresponding to the maximum reward label, y - represents the candidate diagnosis suggestion corresponding to the minimum reward label, r φ represents the function corresponding to the reward model, and φ is the parameter of the reward model, Denotes the expected value of the triple (x, y + , y - ), and σ denotes the Sigmoid function.

[0037] Preferably, the pre-trained large language model includes the Llama model, and the loss function used in the preliminary fine-tuning process of the pre-trained large language model is the cross-entropy loss function.

[0038] Preferably, the total loss function is expressed as:

[0039]

[0040] where p θ (y|x) and p θ (·|x) both represent the output probability of the diagnostic advice for the current round, represents the output probability of the diagnostic advice for the previous round, r φ (x, y) represents the reward score corresponding to the diagnostic advice for the current round, represents the expectation function, D KL represents the KL divergence, and α represents the weight hyperparameter.

[0041] In a second aspect, the present invention provides a disease-assisted diagnosis device based on reward ranking and online reinforcement learning, including:

[0042] A model construction module, configured to construct a fine-tuning data set and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned, and the disease-assisted diagnosis model is fine-tuned again using the reward ranking fine-tuning method to obtain a fine-tuned disease-assisted diagnosis model. Specifically, it includes: constructing a trained reward model. During the fine-tuning process of the current round, the multi-modal medical data of each patient in the fine-tuning data set is respectively input into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round to obtain the output probability of the diagnostic advice for the current round and the output probability of the diagnostic advice for the previous round. The multi-modal medical data of each patient and the diagnostic advice for the current round are input into the trained reward model to obtain a reward score. A total loss function is constructed based on the reward score, the output probability of the diagnostic advice for the current round, and the output probability of the diagnostic advice for the previous round. The disease-assisted diagnosis model of the current round is fine-tuned based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round are respectively used as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round to repeat the above fine-tuning process of the current round, and finally a fine-tuned disease-assisted diagnosis model is obtained;

[0043] The prediction module is configured to obtain multimodal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain corresponding diagnostic suggestions.

[0044] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0046] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in any implementation manner in the first aspect when the computer program is executed by a processor.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The disease-assisted diagnosis method based on reward ranking and online reinforcement learning proposed in this invention introduces a reward ranking fine-tuning method to fine-tune the Llama model through online reinforcement learning, making the diagnostic suggestions generated by it more accurate and reasonable, in line with medical expertise and diagnostic standards. Through reward ranking fine-tuning, the model can better align with human diagnostic thinking and expectations, improve the credibility of diagnostic results, better follow medical standards, avoid generating erroneous or inappropriate diagnostic results, enhance effective control over the generated results, and generate diagnostic suggestions that meet standards as expected, reducing uncertainty and risk.

[0049] (2) The disease-assisted diagnosis method based on reward ranking and online reinforcement learning proposed in this paper effectively utilizes a small amount of high-quality human feedback data through reward ranking, reducing costs and improving fine-tuning efficiency. The reward ranking fine-tuning method can fully utilize limited feedback data to achieve significant improvements in model performance and is suitable for application in the medical field.

[0050] (3) The disease-assisted diagnosis method based on reward ranking and online reinforcement learning proposed in this invention is designed to adapt to the online fine-tuning strategy in the medical field, so that the model can maintain a high level of performance under different clinical scenarios and patient data, enhance the model's generalization ability, and be able to adapt to various disease types and patient groups to meet practical application needs. It can also further utilize new data and human feedback to continuously improve model performance, adapt to the rapidly changing medical environment, and continuously update knowledge to maintain its advancement and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0052] Figure 1 It is a schematic flowchart of a disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to an embodiment of the present application;

[0053] Figure 2 It is a schematic diagram of a disease-assisted diagnosis device based on reward ranking and online reinforcement learning according to an embodiment of the present application;

[0054] Figure 3 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0055] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0056] Figure 1 It shows a disease-assisted diagnosis method based on reward ranking and online reinforcement learning provided by an embodiment of the present application, including the following steps:

[0057] S1. Construct a fine-tuning dataset and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned. The disease-assisted diagnosis model is fine-tuned using the reward ranking fine-tuning method to obtain a fine-tuned disease-assisted diagnosis model, which specifically includes: constructing a trained reward model. During the fine-tuning process of the current round, the multi-modal medical data of each patient in the fine-tuning dataset is respectively input into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round to obtain the output probability of the diagnosis advice of the current round and the output probability of the diagnosis advice of the previous round. The multi-modal medical data of each patient and the diagnosis advice of the current round are input into the trained reward model to obtain a reward score. A total loss function is constructed based on the reward score, the output probability of the diagnosis advice of the current round, and the output probability of the diagnosis advice of the previous round. The disease-assisted diagnosis model of the current round is fine-tuned based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round are respectively used as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round to repeat the above fine-tuning process of the current round, and finally a fine-tuned disease-assisted diagnosis model is obtained.

[0058] In a specific embodiment, the pre-trained large language model includes the Llama model, and the loss function used in the preliminary fine-tuning process of the pre-trained large language model is the cross-entropy loss function.

[0059] Specifically, the pre-trained large language model in the embodiment of the present application is preliminarily fine-tuned using the Llama 3.2 model as an example to adapt it to tasks in the medical field. The Llama 3.2 model is selected as the base model, which is pre-trained on large-scale general text data, has strong language understanding and generation capabilities, and supports the input of multi-modal data.

[0060] First, a fine-tuning dataset is constructed. The data in the fine-tuning dataset comes from Electronic Medical Records (EMRs), clinical dialogue data, medical textbooks and guidelines, as well as published medical papers and case reports. EMRs include patients' medical histories, symptom descriptions, test results, and diagnostic conclusions, etc. Clinical dialogue data are the conversation records between doctors and patients, including symptom inquiries, diagnostic discussions, etc. Medical textbooks and guidelines, as authoritative sources of medical knowledge, can provide standard diagnosis and treatment specifications. Published medical papers and case reports are the latest medical research results and clinical cases, which are also very informative for reference. The above data are processed to obtain the fine-tuning dataset. The data processing process includes text cleaning, annotation and structuring, and data augmentation. Irrelevant information and noise are removed through text cleaning, spelling mistakes are corrected, and medical terms are unified. Key information such as symptoms, signs, test results, diagnostic conclusions, etc. are annotated through annotation and structuring to construct a structured dataset. Data augmentation methods can use techniques such as synonym replacement, random insertion, and random deletion to increase data diversity and improve the robustness of the model. Preprocessing operations such as standardization, size adjustment, and denoising are performed on the image data to meet the input requirements of the model. Finally, the fine-tuning dataset is divided into a training set, a validation set, and a test set to ensure the objectivity of model evaluation.

[0061] Based on the fine-tuning dataset, the Llama 3.2 model is initially fine-tuned using supervised learning to learn the mapping relationship between the input data and the output data in the fine-tuning dataset. The loss function used in the initial fine-tuning process is the cross-entropy loss function, which is expressed as:

[0062]

[0063] where x is the input data, that is, the multi-modal medical data of the patient, and y t is the word generated by the large language model pre-trained at the t-th time step, and y <t represents the context, that is, all the word sequences generated by the large language model pre-trained before the t-th time step. θ represents the parameters of the pre-trained large language model, and p θ (y t ∣y <t ,x) represents the probability that the pre-trained large language model generates y t given the context and the input data.

[0064] In the initial fine-tuning process, the Adam optimizer is used to update the parameters to obtain the disease-assisted diagnosis model.

[0065] The input data of the disease-assisted diagnosis model is the multi-modal medical data of the patient. The medical data includes one or more of text data, structured data, and image data. The text data includes text information such as the chief complaint, current medical history, and past medical history. The structured data includes numerical data and / or categorical data such as physiological indicators, laboratory test data, and vital sign data. The image data includes X-ray films and MRI images, and is processed using the multi-modal interface of Llama 3.2. The disease-assisted diagnosis model is fine-tuned using the Reward Ranked Fine Tuning method to optimize the diagnostic suggestions generated by it.

[0066] In a specific embodiment, constructing a trained reward model specifically includes:

[0067] Construct a reward model;

[0068] Input the multi-modal medical data of each patient in the fine-tuning dataset into the disease-assisted diagnosis model to obtain a number of candidate diagnostic suggestions;

[0069] Score each candidate diagnostic suggestion by medical experts to obtain reward labels;

[0070] Construct a reward ranking dataset according to the multi-modal medical data of each patient in the fine-tuning dataset and the corresponding each candidate diagnostic suggestion and reward label;

[0071] Input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnostic suggestion corresponding to its maximum reward label into the reward model to obtain a positive reward score; input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnostic suggestion corresponding to its minimum reward label into the reward model to obtain a negative reward score;

[0072] Construct a reward loss function based on the positive reward score and the negative reward score, and train the reward model based on the reward loss function to obtain a trained reward model.

[0073] In a specific embodiment, the reward model includes a Transformer model with a two-tower structure. The Transformer model with a two-tower structure includes a first Transformer encoder, a second Transformer encoder, and a matching score calculation module;

[0074] Encode the multi-modal medical data x of the patient and the candidate diagnostic suggestion y respectively through the first Transformer encoder and the second Transformer encoder i as follows:

[0075] H x = Encoder1(x);

[0076]

[0077] Among them, H x is the embedded representation of the multimodal medical data x of the patient, is the embedded representation of the candidate diagnosis suggestion y i with dimension d. Encoder1 represents the function corresponding to the first Transformer encoder, and Encoder2 represents the function corresponding to the second Transformer encoder;

[0078] The matching score calculation module calculates the matching score between the embedded representation of the patient's multimodal medical data x and the embedded representation of the candidate diagnosis suggestion y i to obtain the reward score, as shown in the following formula:

[0079]

[0080] Among them, T represents the transposed matrix, Sigmoid represents the Sigmoid function, and r φ (x, y) represents the reward score corresponding to the candidate diagnosis suggestion y i corresponding to it.

[0081] In a specific embodiment, the reward loss function adopts a contrastive loss function, which is expressed as:

[0082]

[0083] Among them, x represents the multimodal medical data of the patient, and y + represents the candidate diagnosis suggestion corresponding to the maximum reward label, and y - represents the candidate diagnosis suggestion corresponding to the minimum reward label, and r φ represents the function corresponding to the reward model, φ is the parameter of the reward model, represents the expected value of the triple (x, y + , y - ), and σ represents the Sigmoid function.

[0084] Specifically, first train the reward model so that it can accurately estimate the quality of the diagnosis suggestion y generated by the disease-assisted diagnosis model. Then apply the trained reward model to the calculation process of RAFT of the disease-assisted diagnosis model. The trained reward model can provide an accurate reward signal for the output of the disease-assisted diagnosis model to further optimize the generation quality.

[0085] Build a reward model. The input data of the reward model includes medical data and candidate diagnostic suggestions generated by a disease assisted diagnosis model, and the output is the reward score corresponding to the candidate diagnostic suggestions. To process the above two inputs, the reward model needs to have the following functions:

[0086] 1. Be able to understand the medical text context.

[0087] 2. Be able to capture the correlation between the patient's multimodal medical data x and the candidate diagnostic suggestion y i between.

[0088] 3. Be able to generate a reward score representing the quality of the candidate diagnostic suggestions.

[0089] Therefore, the structure of the reward model can adopt a Siamese Transformer model to encode the patient's multimodal medical data x and the candidate diagnostic suggestion y i respectively. This structure has the following advantages:

[0090] 1. Strong semantic understanding ability: Transformer can handle the complex context of medical texts well.

[0091] 2. Correlation modeling: By calculating the similarity of the outputs of the two encoders, the matching degree between the patient's multimodal medical data x and the candidate diagnostic suggestion y i can be well evaluated.

[0092] The model structure is as follows:

[0093] Encoder 1: The first Transformer encoder, used to process the patient's multimodal medical data x.

[0094] Encoder 2: The second Transformer encoder, used to process the candidate diagnostic suggestion y i .

[0095] Matching score calculation module: Calculate the matching score between the output features of the two encoders as the reward score of the candidate diagnostic suggestion.

[0096] The specific calculation process of the reward model is as follows:

[0097] Step 1: Encode the patient's multimodal medical data x and the candidate diagnostic suggestion y i respectively.

[0098] Step 2: Use dot product similarity to calculate the matching score between the embedded representation of the patient's multimodal medical data x and the embedded representation of the candidate diagnostic suggestion y i to obtain the reward score.

[0099] The process of training the reward model is as follows:

[0100] Input the multimodal medical data of each patient in the fine-tuning dataset into the disease-assisted diagnosis model to generate K candidate diagnosis suggestions {y1, y2, …, y K}, and evaluate each candidate diagnosis suggestion y i by medical experts to obtain reward labels.

[0101] Establish the evaluation criteria for medical experts:

[0102] 1) Accuracy: Whether the diagnosis result is correct and matches the patient's condition.

[0103] 2) Rationality: Whether the reasoning process conforms to medical logic and considers relevant symptoms and test results.

[0104] 3) Normativity: Whether medical guidelines and standards are followed and correct medical terms are used.

[0105] 4) Interpretability: Whether the diagnosis suggestion is clear and contains necessary explanations and justifications.

[0106] Based on the above evaluation criteria, medical experts assign a reward label r i to each candidate diagnosis suggestion y i , which is usually a real number reflecting the quality of the output of the disease-assisted diagnosis model.

[0107] Sort the candidate diagnosis suggestions according to the size of the reward labels to form a reward ranking dataset {(x, y i , r i )}.

[0108] Adopt the reward ranking dataset and train the reward model according to the reward loss function to obtain the trained reward model.

[0109] In a specific embodiment, the total loss function is expressed as:

[0110]

[0111] Among them, p θ (y|x) and p θ (·|x) both represent the output probability of the diagnosis suggestion in the current round, represents the output probability of the diagnosis suggestion in the previous round, r φ (x, y) represents the reward score corresponding to the diagnosis suggestion in the current round, represents the expectation function, D KL represents the KL divergence, and α represents the weight hyperparameter.

[0112] In a specific embodiment, the output probability of the diagnostic suggestion is the candidate diagnostic suggestion corresponding to the maximum output probability among several candidate diagnostic suggestions output by the disease-assisted diagnosis model.

[0113] Specifically, the loss function used in the calculation process of RAFT aims to maximize the probability of the output corresponding to the high reward score and minimize the probability of the output corresponding to the low reward score. Therefore, by establishing the above total loss function, the gradient can be further calculated as shown in the following formula:

[0114]

[0115] Using gradient descent or the Adam optimizer, minimize Update the parameters θ of the disease-assisted diagnosis model ′ 。

[0116] The disease-assisted diagnosis model of the embodiment of the present application can achieve online reinforcement learning fine-tuning. The disease-assisted diagnosis model continuously generates diagnostic suggestions in actual applications and collects feedback from medical experts (such as doctors' evaluations, corrections, etc.). Regularly or in real-time, use the newly collected feedback data to update the reward model and the disease-assisted diagnosis model to achieve online learning and continuous improvement.

[0117] In the fine-tuning process of the disease-assisted diagnosis model, the following indicators are used to evaluate the fine-tuning effect of the disease-assisted diagnosis model.

[0118] 1) Accuracy: Calculate the consistency between the diagnostic suggestions generated by the model and the true diagnosis.

[0119] 2) Precision, Recall, F1 value: For multi-classification tasks, calculate the precision, recall, and F1 value of each category to evaluate the performance of the model on different diseases.

[0120] 3) Language generation metrics such as BLEU, ROUGE: Evaluate the quality of the text generated by the model and its similarity to the reference answer.

[0121] 4) Interpretability evaluation: Evaluate the interpretability and rationality of the diagnostic suggestions generated by the model through questionnaires or expert evaluations.

[0122] S2. Obtain the multi-modal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain the corresponding diagnostic suggestions.

[0123] Specifically, the fine-tuned disease-assisted diagnosis model will be integrated into medical information systems, such as electronic medical record systems and diagnostic support systems, and a friendly user interface will be provided to facilitate doctors to view the model's diagnostic recommendations and reasoning process. It is also required to ensure the security and privacy of patient data and comply with relevant regulations and ethical requirements.

[0124] In actual applications, new data and human feedback can be continuously collected to update and optimize the fine-tuned disease-assisted diagnosis model online.

[0125] The present invention discloses a disease-assisted diagnosis method and device based on reward ranking and online reinforcement learning, aiming to address the many shortcomings of existing technologies in intelligent medical-assisted diagnosis. The present invention proposes an innovative disease-assisted diagnosis method based on reward ranking and online reinforcement learning, combining a pre-trained large language model with the multimodal characteristics of the medical field, and achieving self-learning and self-optimization of the model in any downstream medical task through self-evolution. Specifically, the present invention first constructs a high-quality fine-tuning dataset and performs preliminary fine-tuning on the pre-trained large language model to form a basic disease-assisted diagnosis model. Subsequently, the model is further optimized through reward ranking fine-tuning, enabling it to continuously learn and improve through self-generated data under unsupervised conditions, ultimately obtaining a fine-tuned disease-assisted diagnosis model. In practical applications, the disease-assisted diagnosis model of the present invention can automatically receive a patient's multimodal medical data (such as text descriptions, laboratory test results, and medical images) and generate highly reliable diagnostic recommendations. Through the reward ranking strategy, the model can effectively solve the control problem of generating results and significantly improve its adaptability to complex medical tasks. At the same time, the present invention also strengthens the online learning mechanism to enable the model to have the ability to continuously update, and can quickly iterate with the changes in medical knowledge and data to maintain its advanced nature. The present invention technically solves the following problems of existing medical diagnosis models: (1) insufficient model alignment, and the generated results are inconsistent with medical logic; (2) strong dependence on high-quality human feedback data, which makes it difficult to apply on a large scale; (3) lack of accuracy and reliability of generated content, which makes it difficult to meet medical standards; (4) limited generalization ability, which makes it difficult to adapt to different scenarios and diverse patient data; (5) lack of online learning and continuous optimization capabilities, which makes it difficult to keep up with the rapidly changing medical environment. The present invention not only has a breakthrough innovation in technology, but also has a wide range of application value in practice. It provides a reliable solution for the efficient construction and actual deployment of intelligent medical systems, and is of great significance to data utilization, diagnostic accuracy and efficiency improvement in the medical field.

[0126] Further references Figure 2 As an implementation of the methods shown in the above figures, this application provides an embodiment of a disease auxiliary diagnosis device based on reward ranking and online reinforcement learning.Figure 1 It corresponds to the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0127] An embodiment of the present application provides a disease-assisted diagnosis device based on reward ranking and online reinforcement learning, including:

[0128] A model construction module 1, configured to construct a fine-tuning data set and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned, and fine-tunes the disease-assisted diagnosis model using the reward ranking fine-tuning method to obtain a fine-tuned disease-assisted diagnosis model. Specifically, it includes: constructing a trained reward model. During the fine-tuning process of the current round, input the multi-modal medical data of each patient in the fine-tuning data set into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round respectively to obtain the output probability of the diagnosis suggestion of the current round and the output probability of the diagnosis suggestion of the previous round. Input the multi-modal medical data of each patient and the diagnosis suggestion of the current round into the trained reward model to obtain a reward score. Construct a total loss function based on the reward score, the output probability of the diagnosis suggestion of the current round, and the output probability of the diagnosis suggestion of the previous round, and fine-tune the disease-assisted diagnosis model of the current round based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; use the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round respectively, and repeat the above fine-tuning process of the current round to finally obtain a fine-tuned disease-assisted diagnosis model;

[0129] A prediction module 2, configured to obtain the multi-modal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain the corresponding diagnosis suggestion.

[0130] Figure 2 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. As Figure 3 shown, the electronic device of this embodiment includes: a processor 301 and a memory 302; wherein the memory 302 is used to store computer execution instructions; the processor 301 is used to execute the computer execution instructions stored in the memory to implement the various steps executed by the electronic device in the above embodiments. Specifically, reference can be made to the relevant descriptions in the foregoing method embodiments.

[0131] Optionally, the memory 302 can be either independent or integrated with the processor 301.

[0132] When the memory 302 is independently set, the electronic device further includes a bus 303 for connecting the memory 302 and the processor 301.

[0133] An embodiment of the present invention also provides a computer storage medium, in which computer-executable instructions are stored. When the processor 301 executes the computer-executable instructions, the above method is implemented.

[0134] An embodiment of the present invention also provides a computer program product, including a computer program. When the computer program is executed by the processor 301, the above method is implemented.

[0135] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or modules can be in electrical, mechanical or other forms.

[0136] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to implement the solution of this embodiment.

[0137] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in a unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0138] The above integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or the processor 301 to execute some steps of the methods in various embodiments of the present application.

[0139] It should be understood that the above-mentioned processor 301 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor may be a microprocessor or the processor 301 may also be any conventional processor 301, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by the hardware processor 301, or can be executed and completed by a combination of hardware and software modules in the processor 301.

[0140] The memory 302 may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a portable hard drive, a read-only memory, a magnetic disk, or an optical disc, etc.

[0141] The bus 303 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus 303 may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the bus 303 in the drawings of this application is not limited to only one bus 303 or one type of bus 303.

[0142] The above-mentioned storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disc. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0143] An exemplary storage medium is coupled to the processor 301, enabling the processor 301 to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 301. The processor 301 and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor 301 and the storage medium can also exist as discrete components in an electronic device or a master control device.

[0144] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0145] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A disease assisted diagnosis method based on reward ranking and online reinforcement learning, characterized in that, It includes the following steps: Construct a fine-tuning dataset and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned. The disease-assisted diagnosis model is fine-tuned again using the reward ranking fine-tuning method to obtain a fine-tuned disease-assisted diagnosis model. Specifically, it includes: constructing a trained reward model. During the fine-tuning process of the current round, the multi-modal medical data of each patient in the fine-tuning dataset is respectively input into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round to obtain the output probability of the diagnosis suggestion of the current round and the output probability of the diagnosis suggestion of the previous round. The multi-modal medical data of each patient and the diagnosis suggestion of the current round are input into the trained reward model to obtain a reward score. A total loss function is constructed based on the reward score, the output probability of the diagnosis suggestion of the current round, and the output probability of the diagnosis suggestion of the previous round. The disease-assisted diagnosis model of the current round is fine-tuned based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round are respectively used as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round to repeat the above fine-tuning process of the current round, and finally a fine-tuned disease-assisted diagnosis model is obtained; Obtain the multi-modal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain the corresponding diagnosis suggestion.

2. The disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to claim 1, characterized in that Construct a trained reward model, specifically including: Construct a reward model; Input the multi-modal medical data of each patient in the fine-tuning dataset into the disease-assisted diagnosis model to obtain a number of candidate diagnosis suggestions; Score each candidate diagnosis suggestion by medical experts to obtain a reward label; Construct a reward ranking dataset based on the multi-modal medical data of each patient in the fine-tuning dataset and the corresponding candidate diagnosis suggestions and reward labels; Input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnosis suggestion corresponding to its maximum reward label into the reward model to obtain a positive reward score; input the multi-modal medical data of each patient in the reward ranking dataset and the candidate diagnosis suggestion corresponding to its minimum reward label into the reward model to obtain a negative reward score; Construct a reward loss function based on the positive reward score and the negative reward score, and train the reward model based on the reward loss function to obtain a trained reward model.

3. The disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to claim 2, characterized in that, The reward model includes a Transformer model with a two-tower structure. The Transformer model with a two-tower structure includes a first Transformer encoder, a second Transformer encoder, and a matching score calculation module; Encode the patient's multimodal medical data x and candidate diagnosis suggestions y through the first Transformer encoder and the second Transformer encoder respectively, as shown in the following formula: i Perform encoding, as shown in the following formula: H x = Encoder1(x); Among them, H x is the embedded representation of the multimodal medical data x of the patient, is the embedded representation of the candidate diagnosis suggestion y i with dimension d. Encoder1 represents the function corresponding to the first Transformer encoder, and Encoder2 represents the function corresponding to the second Transformer encoder; Calculate the matching score between the embedded representation of the multimodal medical data x of the patient and the embedded representation of the candidate diagnosis suggestion y through the matching score calculation module, and obtain the reward score, as shown in the following formula: i ​ where T represents the transposed matrix, Sigmoid represents the Sigmoid function, and r φ (x, y i ) represents the reward score corresponding to the candidate diagnosis suggestion y i .

4. The disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to claim 2, characterized in that The reward loss function Adopts a contrastive loss function, expressed as: Among them, x represents the multi-modal medical data of the patient, and y + represents the candidate diagnostic suggestion corresponding to the maximum reward label, and y - represents the candidate diagnostic suggestion corresponding to the minimum reward label, and r φ represents the function corresponding to the reward model, and φ is the parameter of the reward model. represents the expected value of the triple (x, y + , y - ), and σ represents the Sigmoid function.

5. The disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to claim 1, characterized in that The pre-trained large language model includes a Llama model, and the loss function used in the preliminary fine-tuning process of the pre-trained large language model is a cross-entropy loss function.

6. The disease-assisted diagnosis method based on reward ranking and online reinforcement learning according to claim 1, characterized in that, The total loss function is expressed as: where p θ (y|x) and p θ (·|x) both represent the output probabilities of the diagnostic suggestions for the current round, represents the output probability of the diagnostic suggestions for the previous round, r φ (x, y) represents the reward score corresponding to the diagnostic suggestions for the current round, represents the expectation function, D KL represents the KL divergence, and α represents the weight hyperparameter.

7. A disease-assisted diagnosis device based on reward ranking and online reinforcement learning, characterized in that, It includes: A model construction module, configured to construct a fine-tuning dataset and a disease-assisted diagnosis model. The disease-assisted diagnosis model uses a pre-trained large language model that has been preliminarily fine-tuned, and uses the reward ranking fine-tuning method to fine-tune the disease-assisted diagnosis model again to obtain a fine-tuned disease-assisted diagnosis model. Specifically, it includes: constructing a trained reward model. During the fine-tuning process of the current round, the multimodal medical data of each patient in the fine-tuning dataset is respectively input into the disease-assisted diagnosis model of the current round and the disease-assisted diagnosis model of the previous round to obtain the output probability of the diagnosis advice of the current round and the output probability of the diagnosis advice of the previous round. The multimodal medical data of each patient and the diagnosis advice of the current round are input into the trained reward model to obtain a reward score. A total loss function is constructed based on the reward score, the output probability of the diagnosis advice of the current round, and the output probability of the diagnosis advice of the previous round. The disease-assisted diagnosis model of the current round is fine-tuned based on the total loss function to obtain the fine-tuned disease-assisted diagnosis model of the current round; the disease-assisted diagnosis model of the current round and the fine-tuned disease-assisted diagnosis model of the current round are respectively used as the disease-assisted diagnosis model of the previous round and the disease-assisted diagnosis model of the current round to repeat the above fine-tuning process of the current round, and finally obtain a fine-tuned disease-assisted diagnosis model; A prediction module, configured to obtain the multimodal medical data of the patient to be predicted and input it into the fine-tuned disease-assisted diagnosis model to obtain the corresponding diagnosis advice.

8. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1-6.

Images