Large medical model based on DPO and application thereof

Abstract

The invention relates to a medical big model based on DPO and application thereof.The medical big model collects and processes data through a data preprocessing module, the mapping relation between input and output of the medical big model is obtained based on the data, and a preference calibration module optimizes the data preprocessing module based on DPO and adjusts model parameters of the medical big model; establishing a medical preference alignment quality evaluation strategy and a preference alignment optimization strategy through a feedback optimization module; the large medical model can be applied to a medical diagnosis question-answering system. According to the method, the problem of preference alignment of an existing medical large model is solved, the precision and efficiency of the medical large model are improved, and the steps of reward model construction and reinforcement learning in traditional RLHF are avoided by directly optimizing the language model, so that human preferences are better met.

Description

Technical Field

[0001] The present invention relates to the field of medical care informatics, namely the technical field of information and communication technologies specifically used for handling or processing medical or health data, and in particular to a DPO-based medical macro model and its application. Background Art

[0002] In the medical field, with the rapid development of artificial intelligence technology, medical big models, as an important part of it, are gradually showing their huge potential. The application scenarios of medical big models show their diversity and importance in the medical field. They can improve diagnosis, treatment and disease prevention, improve the quality and efficiency of medical care, and cover many aspects such as clinical, management, education, and public health, which is expected to promote innovation and progress in medical care.

[0003] However, the medical big model contains a lot of professional terms, and accurately understanding and applying this knowledge is undoubtedly a huge challenge for the general audience. Therefore, preference alignment of the medical big model becomes particularly important.

[0004] Preference alignment refers to adjusting the output of a large language model to make it more consistent with human preferences and expectations. It adjusts the probability distribution of token generation in the model so that responses that meet human preferences have a higher probability, while responses that do not meet human preferences have a lower probability. This is usually achieved by using a reward model that represents human preferences. Therefore, the core goal of preference alignment is to make the results of the machine learning model output consistent with human preferences. For large medical models, this means that the results generated by the model during the diagnostic question-answering process should meet the standards of clinical practice.

[0005] At present, the RLHF (Reinforcement Learning with Human Feedback) algorithm can achieve the best results in the alignment of large models. Through human feedback and the PPO algorithm, RLHF can effectively align the model with human preferences. However, RLHF has the disadvantages of being difficult to train and requiring a large amount of training memory. Summary of the invention

[0006] The present invention solves the problems existing in the prior art and provides a large medical model based on DPO and its application.

[0007] The technical solution adopted by the present invention is a medical large model based on DPO, and the medical large model includes:

[0008] A data pre-processing module, used to collect and process data, and obtain the mapping relationship between the input and output of the medical model based on the data;

[0009] A preference calibration module, used to optimize the data pre-processing module based on DPO and adjust the model parameters of the medical large model;

[0010] A feedback optimization module is used to establish a medical preference alignment quality assessment strategy and a preference alignment optimization strategy.

[0011] Preferably, in the data pre-processing module, the collected data include symptom descriptions and diagnosis results, invalid data and / or duplicate data are removed based on the symptom descriptions and diagnosis results, tokens are extracted after unifying the format, and a data set is constructed.

[0012] Preferably, the probability of each sequence in the data set meeting the medical text standard is calculated, and the model parameters of the data pre-processing module are updated using a gradient descent algorithm;

[0013] Extract question sequences and corresponding answer sequences from the data set, fill in the gaps, and combine the question sequence and the corresponding answer sequence into a sequence pair;

[0014] Use the function to maximize the probability of sequence pairs and update the parameters of the medical model using the gradient descent algorithm.

[0015] Preferably, the preference calibration module defines a preference-based objective function and converts the preference into a loss function for model training; after inputting the original data into the pre-trained LLM, the model is adjusted based on the instruction data set, and finally the comparison data set is input into the DPO-based model to obtain the result.

[0016] Preferably, the loss function satisfies,

[0017]

[0018] Where x is a given input, y1 and y2 are preferences, and D is a dataset containing input and preference pairs;

[0019] By minimizing L(π), the model learns to produce more preferred outputs from human preferences.

[0020] Preferably, the feedback optimization module includes a quality assessment unit and a preference optimization unit. The quality assessment unit extracts features from user feedback information and obtains trends of user feedback issues based on a quality assessment strategy. The preference optimization unit obtains feedback information based on trends and outputs a preference alignment optimization strategy.

[0021] Preferably, the quality assessment unit defines indicators for assessing the response quality of the large medical model and sets the weights of various indicators, constructs a neural network, and maps the output values ​​of the fuzzy neural network to corresponding preference alignment quality evaluation levels.

[0022] Preferably, the preference optimization unit collects user feedback information, evaluates the user feedback information with a quality evaluation unit, modifies the rating of feedback information that does not meet human preference requirements, and saves the data for training the quality evaluation unit;

[0023] Sparse autoencoding and width learning are used to map, encode and fuse feedback information that does not meet human preference requirements, and Markov decision making is used to adjust the parameters of the preference calibration module.

[0024] An application of the DPO-based medical large model is applied to a medical diagnosis question-answering system.

[0025] The application comprises the following steps:

[0026] S1 collects information about patients’ preferences for medical information and decision-making and defines preference criteria;

[0027] S2 data collection, and collecting feedback to correct preference information;

[0028] S3 uses the DPO method to train the model;

[0029] S4 uses standardized indicators to evaluate whether the model output meets the preset preference standards, collects feedback and corrects preference information.

[0030] The present invention relates to a medical big model based on DPO and its application. The medical big model collects and processes data with a data pre-processing module, and obtains the mapping relationship between the input and output of the medical big model based on the data. The preference calibration module optimizes the data pre-processing module based on DPO and adjusts the model parameters of the medical big model. The feedback optimization module establishes a medical preference alignment quality evaluation strategy and a preference alignment optimization strategy. The medical big model can be applied to a medical diagnosis question-answering system.

[0031] The beneficial effect of the present invention is to solve the preference alignment problem of existing large medical models, improve the accuracy and efficiency of large medical models, and avoid the reward model construction and reinforcement learning steps in traditional RLHF by directly optimizing the language model to better meet human preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural diagram of the medical large model of the present invention;

[0033] Figure 2 is a working schematic diagram of the preference calibration module in the present invention;

[0034] Figure 3 The figure is an application flow chart of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0036] like Figure 1 As shown, the present invention relates to a medical large model based on DPO, and the medical large model includes:

[0037] A data pre-processing module, used to collect and process data, and obtain the mapping relationship between the input and output of the medical model based on the data;

[0038] A preference calibration module, used to optimize the data pre-processing module based on DPO and adjust the model parameters of the medical large model;

[0039] A feedback optimization module is used to establish a medical preference alignment quality assessment strategy and a preference alignment optimization strategy.

[0040] In the present invention, the computation of direct preference optimization (DPO) is lightweight and stable, and does not require explicit reward modeling or reinforcement learning process. It is optimized through a simple binary cross-entropy (BCE) objective, which greatly simplifies the preference learning process.

[0041] In the present invention, the data pre-processing module of the medical big model collects clinical data from consultations, including patient symptom descriptions and diagnosis results, etc. The data processing includes but is not limited to clearing invalid and duplicate data and unifying the format. The processed data is then used to train the medical big model to master the mapping relationship between symptoms and diagnosis results, so that it has the ability to initially solve consultation tasks; the preference calibration module optimizes the data pre-processing module of the medical big model by using DPO, continuously adjusts the model parameters of the medical big model during the optimization, and improves the consistency between the output results of the medical big model and human preferences; the feedback optimization module establishes a medical preference alignment quality assessment strategy, extracts features from user feedback information, understands the main trends of user feedback issues, and uses feedback information to build a preference alignment optimization strategy.

[0042] In the data pre-processing module, the collected data include symptom descriptions and diagnosis results. Invalid data and / or duplicate data are removed based on the symptom descriptions and diagnosis results, tokens are extracted after unifying the format, and a data set is constructed.

[0043] Calculate the probability that each sequence in the data set meets the medical text standard, and use the gradient descent algorithm to update the model parameters of the data pre-processing module;

[0044] Extract question sequences and corresponding answer sequences from the data set, fill in the gaps, and combine the question sequence and the corresponding answer sequence into a sequence pair;

[0045] Use the function to maximize the probability of sequence pairs and update the parameters of the medical model using the gradient descent algorithm.

[0046] In the present invention, the data pre-processing module of the medical large model is established based on the Qwen-32B model using self-supervised learning, and its construction method is as follows:

[0047] (1) Collect and organize the consultation-related data to construct dataset a;

[0048] (2) Calculate the probability that each sequence in data set a meets the medical text standard;

[0049] (3) Use the gradient descent algorithm to update the model parameters of the medical large model data pre-processing module;

[0050] (4) Extract question sequences and corresponding answer sequences from dataset a;

[0051] (5) When the answer sequence is incomplete, complete it according to the question sequence;

[0052] (6) Combine the question sequence and the corresponding answer sequence into a sequence pair;

[0053] (7) Use the function to maximize the probability of sequence pairs and use the gradient descent algorithm to update the parameters of the medical model.

[0054] like Figure 2 As shown, the preference calibration module defines a preference-based objective function and converts the preference into a loss function for model training; after the raw data (Raw Text Corpus) is input into the pre-trained LLM (Pre-trained LLM), the model (Supervised Fine Tuned Model) is adjusted based on the instruction dataset (Instruction Dataset), and finally the comparison dataset (Comparison Dataset) is input into the DPO-based model (DPO optimized Model) to obtain the result.

[0055] The loss function satisfies,

[0056]

[0057] Where x is a given input, y1 and y2 are preferences, and D is a dataset containing input and preference pairs;

[0058] By minimizing L(π), the model learns to produce more preferred outputs from human preferences.

[0059] In the present invention, the preference calibration module is implemented using DPO. First, a preference-based objective function is defined to directly convert human preferences into a loss function L(π) for model training. Specifically, if human preferences are biased towards one of the two potential outputs y1 relative to y2, the following binary cross entropy loss function (π) can be used to directly optimize the language model π:

[0060]

[0061] Among them, p(y1>y2|x) represents the probability that humans prefer y1 over y2 given an input x. By minimizing this loss function, the model learns to produce more preferred outputs directly from human preferences without indirectly going through a reward model.

[0062] The feedback optimization module includes a quality evaluation unit and a preference optimization unit. The quality evaluation unit extracts features from user feedback information and obtains trends of user feedback issues based on a quality evaluation strategy. The preference optimization unit obtains feedback information based on trends and outputs a preference alignment optimization strategy.

[0063] The quality assessment unit defines indicators for assessing the response quality of the large medical model and sets the weights of various indicators, constructs a neural network, and maps the output values ​​of the fuzzy neural network to corresponding preference alignment quality evaluation levels.

[0064] The preference optimization unit collects user feedback information, and the quality evaluation unit evaluates the user feedback information, and modifies the rating of feedback information that does not meet human preference requirements, and saves the data for training the quality evaluation unit;

[0065] Sparse autoencoding and width learning are used to map, encode and fuse feedback information that does not meet human preference requirements, and Markov decision making is used to adjust the parameters of the preference calibration module.

[0066] In the present invention, the feedback optimization module includes two parts: quality evaluation and preference optimization.

[0067] The quality assessment unit construction method includes:

[0068] (1) Define the indicators for evaluating the response quality of the medical big model and obtain the weights of each indicator;

[0069] (2) Construct a fuzzy neural network including input layer, fuzzy layer, inference layer and output layer;

[0070] (3) Map the output values ​​of the fuzzy neural network to the corresponding preference alignment quality evaluation levels.

[0071] The construction methods of the preference optimization unit include:

[0072] (1) Collect user feedback information and use the quality assessment unit to evaluate the feedback information;

[0073] (2) Human evaluation is performed again on the feedback information that meets human preference requirements. If there is feedback information that does not meet human preference requirements, the rating is modified and the data is saved for training the quality evaluation unit;

[0074] (3) Use sparse autoencoding and width learning to map, encode and fuse feedback information that does not meet human preference requirements;

[0075] (4) Use Markov decision making to adjust the parameters of the preference calibration module.

[0076] The following are embodiments of the present invention:

[0077] S1 Collect and organize common relevant data during diagnosis, such as "What are your main symptoms?", "When did your symptoms start?", "Are your symptoms constant or intermittent?", "Do you have any relevant past medical history or family history?", "Do you feel pain?" "If yes, please describe the nature, location and intensity of the pain", "Do you have any history of surgery or hospitalization?", "Do you have any cardiovascular symptoms such as palpitations, chest pain or syncope?", etc.;

[0078] S2 model initialization: start the model with random parameters and define the probability function p(X i )=p(x i,1 ,x i,2 ,…,x i,m ), represents the i-th sequence X i The probability of conforming to human preferences, p ≥ 0;

[0079] S3 defines the loss function Represents the negative log-likelihood estimate of the next Chinese character predicted by the model, where p(x i,j+1 |x i,1,…,j ,θ1) represents the number of i The first j Chinese character sequences x are known in i,1,…,j In the case of predicting x i,j+1 The probability of the Chinese character at the position, θ1 represents the trainable parameters of the model in the medical large model data pre-processing module;

[0080] S4 uses the gradient descent algorithm to minimize the loss function L[X] and update the trainable parameter θ1, so that the accuracy of the medical large model in predicting the next Chinese character based on the knowledge of common diagnostic problems is improved;

[0081] S5 iterates until the questions it answers are consistent with human preferences.

[0082] The present invention also relates to an application of the DPO-based medical large model, which is applied to a medical diagnosis question-answering system.

[0083] Specifically, if Figure 3 As shown, the application includes the following steps:

[0084] S1 collects information about patients’ preferences for medical information and decision-making, including language style, whether answers to questions are easy to understand, etc., and defines preference criteria;

[0085] S2 Data collection and feedback collection from medical professionals and patients to understand their preferences for model output and calibrate preference information;

[0086] S3 uses the DPO method to train the model; using the DPO method, by optimizing the model output, the loss function is adjusted to make it more consistent with the collected preference data; in practical applications, it can also be combined with the reinforcement learning method to further optimize the model output through the reward mechanism;

[0087] S4 uses standardized metrics to evaluate whether the model's output meets preset preference criteria, collects feedback from medical professionals and patients, and corrects preference information.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A medical large model based on DPO, characterized by: The medical big model comprises: a data pre-processing module, which is used to collect and process data, and obtain the mapping relationship between the input and output of the medical big model based on the data; A preference calibration module, used to optimize the data pre-processing module based on DPO and adjust the model parameters of the medical large model; A feedback optimization module is used to establish a medical preference alignment quality assessment strategy and a preference alignment optimization strategy.

2. A medical large model based on DPO according to claim 1, characterized in that: In the data pre-processing module, the collected data include symptom descriptions and diagnosis results. Invalid data and / or duplicate data are removed based on the symptom descriptions and diagnosis results, tokens are extracted after unifying the format, and a data set is constructed.

3. A medical large model based on DPO according to claim 2, characterized in that: Calculate the probability that each sequence in the data set meets the medical text standard, and use the gradient descent algorithm to update the model parameters of the data pre-processing module; Extract question sequences and corresponding answer sequences from the data set, fill in the gaps, and combine the question sequence and the corresponding answer sequence into a sequence pair; Use the function to maximize the probability of sequence pairs and update the parameters of the medical model using the gradient descent algorithm.

4. The DPO-based medical model according to claim 1, characterized in that: The preference calibration module defines a preference-based objective function and converts the preference into a loss function for model training; after inputting the original data into the pre-trained LLM, the model is adjusted based on the instruction data set, and finally the comparison data set is input into the DPO-based model to obtain the result.

5. A medical large model based on DPO according to claim 4, characterized in that: The loss function satisfies, Where x is a given input, y1 and y2 are preferences, and D is a dataset containing input and preference pairs; By minimizing L(π), the model learns to produce more preferred outputs from human preferences.

6. The DPO-based medical model according to claim 1, characterized in that: The feedback optimization module includes a quality evaluation unit and a preference optimization unit. The quality evaluation unit extracts features from user feedback information and obtains trends of user feedback issues based on a quality evaluation strategy. The preference optimization unit obtains feedback information based on trends and outputs a preference alignment optimization strategy.

7. A medical macro model based on DPO according to claim 6, characterized in that: The quality assessment unit defines indicators for assessing the response quality of the large medical model and sets the weights of various indicators, constructs a neural network, and maps the output values ​​of the fuzzy neural network to corresponding preference alignment quality evaluation levels.

8. The DPO-based medical model according to claim 7, characterized in that: The preference optimization unit collects user feedback information, and the quality evaluation unit evaluates the user feedback information, and modifies the rating of feedback information that does not meet human preference requirements, and saves the data for training the quality evaluation unit; Sparse autoencoding and width learning are used to map, encode and fuse feedback information that does not meet human preference requirements, and Markov decision making is used to adjust the parameters of the preference calibration module.

9. An application of a DPO-based medical large model according to any one of claims 1 to 8, characterized in that: Applied to medical diagnosis question answering system.

10. The application of the DPO-based medical large model according to claim 9, characterized in that: The application comprises the following steps: S1 collects information about patients’ preferences for medical information and decision-making and defines preference criteria; S2 data collection, and collecting feedback to correct preference information; S3 uses the DPO method to train the model; S4 uses standardized indicators to evaluate whether the model output meets the preset preference standards, collects feedback and corrects preference information.

Images