Medical question and answer large model fine tuning method and device, equipment and storage medium

Abstract

The invention discloses a medical question and answer large model fine tuning method and device, equipment and a storage medium, and relates to the technical field of computers, and the method comprises the steps: obtaining an initial medical question and answer large model constructed based on a basic large language model, and carrying out the supervision fine tuning of the initial medical question and answer large model based on a preset cue word instruction data set; screening sample data pairs in a preset cue word instruction data set based on the medical question and answer large model after supervision fine tuning, a preset screening rule and a pre-trained target model; and a target preference training data set is determined based on a sample screening result, the target preference training data set is utilized to perform fine tuning on the medical question and answer large model after supervision fine tuning, and a target medical question and answer large model after fine tuning is obtained, so that a response is made to medical related questions initiated by the user side based on the target medical question and answer large model. According to the method, the construction efficiency and quality of the preference data can be preferentially improved, the training process of model alignment is simplified, and the stability is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for fine-tuning a large medical question-and-answer model. Background Art

[0002] In recent years, supervised fine-tuning (SFT), a standard instruction fine-tuning method for large pre-trained language models (LLMs), has been effective in guiding model behavior to follow instructions. However, it still has inherent limitations: First, data quality dependence and bias amplification: Because SFT's effectiveness is extremely dependent on the quality of training data, building large, high-quality, and diverse instruction datasets is expensive. If the dataset contains noise, bias, factual errors, or suboptimal answers, the model may learn and amplify these undesirable characteristics during fine-tuning, resulting in inconsistent quality of generated answers and even harmful content. Second, preference alignment is insufficient: Because SFT is essentially imitation learning, it is difficult to explicitly incorporate more nuanced human preferences (such as the safety, usefulness, honesty, and style of answers) into the model. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus, device, and storage medium for fine-tuning a large medical question-answering model. These methods prioritize improving the efficiency and quality of constructing preference data, simplifying the training process for model alignment, and improving stability. This method further enhances the alignment accuracy and effectiveness of the fine-tuned target large medical question-answering model in a more efficient and stable manner. The specific solution is as follows:

[0004] In a first aspect, this application provides a method for fine-tuning a large medical question-answering model, comprising:

[0005] Obtaining an initial medical question-answering model constructed based on the basic large language model, and performing supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset to determine a supervised fine-tuned medical question-answering model;

[0006] Screening the sample data pairs in the preset prompt word instruction dataset based on the supervised fine-tuned medical question-answering model, the preset screening rules, and the pre-trained target model to determine the sample screening results;

[0007] Based on the sample screening results, a target preference training data set is determined, and the supervised fine-tuned medical question-answering big model is fine-tuned again using the target preference training data set to obtain a fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

[0008] Optionally, obtaining an initial medical question-answering model constructed based on the basic large language model, and performing supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset, includes:

[0009] Obtain an initial medical question-answering model pre-trained based on a basic large language model;

[0010] Acquire a pre-organized prompt word instruction data set to determine a preset prompt word instruction data set; the preset prompt word instruction data set includes a plurality of sample data pairs;

[0011] The initial medical question-answering big model is supervised fine-tuned based on each of the sample data pairs in the preset prompt word instruction data set and the preset supervised fine-tuning termination condition to determine the medical question-answering big model after supervised fine-tuning.

[0012] Optionally, screening the sample data pairs in the preset prompt word instruction dataset based on the supervised fine-tuned medical question-answering model, preset screening rules, and a pre-trained target model includes:

[0013] Based on the supervised fine-tuned medical question-answering model, generating a response corresponding to the prompt word instruction in each of the sample data pairs in the preset prompt word instruction dataset to determine corresponding model response information; the sample data pairs include the corresponding prompt word instruction and standard response information;

[0014] The sample data pairs are screened based on a pre-trained target model, preset screening rules and the model response information corresponding to each of the sample data pairs to determine a sample screening result.

[0015] Optionally, screening the sample data pairs based on a pre-trained target model, preset screening rules, and the model response information corresponding to each of the sample data pairs includes:

[0016] Get the pre-trained target model;

[0017] For the prompt word instruction in any of the sample data pairs, determining, based on the target model, a first reward score corresponding to the model response information of the current prompt word instruction and a second reward score corresponding to the standard response information of the current prompt word instruction;

[0018] Determining whether the first reward score is less than the second reward score to determine a corresponding determination result;

[0019] If the judgment result indicates that the score is less than the second reward score, then determining that the current sample data pair is a valid sample;

[0020] If the judgment result shows that the reward score is not less than the second reward score, the current sample data pair is determined to be an invalid sample, and a corresponding rejection sampling operation is triggered.

[0021] Optionally, determining a target preference training data set based on the sample screening result includes:

[0022] Based on a preset data format and the corresponding model response information, format conversion is performed on the sample data pairs determined to be valid samples in the sample screening results to determine corresponding preferred training data pairs;

[0023] The target preference training data set is determined by collecting the constructed preference training data pairs.

[0024] Optionally, fine-tuning the supervised fine-tuned medical question answering model again using the target preference training dataset includes:

[0025] Based on the preset preference optimization algorithm and the target preference training data set, the supervised fine-tuned medical question-answering model is iteratively trained, and the parameters of the trained model in each iterative round are adjusted using a preset loss function until the model converges, and the fine-tuned target medical question-answering model is determined.

[0026] In a second aspect, the present application provides a device for fine-tuning a large medical question-answering model, comprising:

[0027] A supervised fine-tuning module is used to obtain an initial medical question-answering model constructed based on the basic large language model, and to perform supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset to determine a supervised fine-tuned medical question-answering model;

[0028] A sample screening module is used to screen the sample data pairs in the preset prompt word instruction data set based on the supervised fine-tuned medical question-answering model, preset screening rules and a pre-trained target model to determine the sample screening results;

[0029] The instruction fine-tuning completion module is used to determine the target preference training data set based on the sample screening results, and use the target preference training data set to further fine-tune the supervised fine-tuned medical question-answering big model to obtain the fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

[0030] Optionally, the sample screening module includes:

[0031] a model response determination unit, configured to generate, based on the supervised fine-tuned medical question-answering macromodel, responses corresponding to the prompt word instructions in each of the sample data pairs in the preset prompt word instruction dataset, to determine corresponding model response information; wherein the sample data pairs include the corresponding prompt word instructions and standard response information;

[0032] The sample screening unit is used to screen the sample data pairs based on the pre-trained target model, preset screening rules and the model response information corresponding to each of the sample data pairs to determine the sample screening results.

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the steps of the aforementioned medical question-answering large model fine-tuning method.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of the aforementioned medical question-and-answer large model fine-tuning method.

[0037] It can be seen that in this application, an initial medical question and answer big model constructed based on the basic big language model is obtained, and the initial medical question and answer big model is supervised and fine-tuned based on the preset prompt word instruction data set to determine the supervised fine-tuned medical question and answer big model; based on the supervised fine-tuned medical question and answer big model, the sample data pairs in the preset prompt word instruction data set are screened to determine the sample screening results; based on the sample screening results, the target preference training data set is determined, and the target preference training data set is used to fine-tune the supervised fine-tuned medical question and answer big model again to obtain the fine-tuned target medical question and answer big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question and answer big model. That is to say, in this application, the initial medical question-answering big model that needs to be fine-tuned and is built based on the basic big language model is first subjected to an initial supervised fine-tuning operation using a preset prompt word instruction data set. Afterwards, the sample data pairs in the preset prompt word instruction data set are screened by the supervised fine-tuned medical question-answering big model and the target model to construct a target preference training data set. The target preference training data set is then used to fine-tune the supervised fine-tuned medical question-answering big model again to determine the fine-tuned target medical question-answering big model. In this way, the construction efficiency and quality of the preference data can be improved first, and the training process of the model alignment can be simplified and the stability can be improved, thereby improving the alignment accuracy and effect of the determined fine-tuned target medical question-answering big model in a more efficient and stable manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 A flowchart of a method for fine-tuning a large medical question-answering model provided in this application;

[0040] Figure 2 A schematic diagram of the fine-tuning process of a specific medical question-answering model provided for this application;

[0041] Figure 3 A schematic diagram of the structure of a medical question-and-answer large model fine-tuning device provided in this application;

[0042] Figure 4 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] In recent years, supervised fine-tuning, a standard instruction fine-tuning method for large-scale pre-trained language models, has been able to effectively guide the model's behavior to follow instructions. However, it still has inherent limitations: First, data quality dependence and bias amplification: Because the effectiveness of SFT is extremely dependent on the quality of the training data, building large-scale, high-quality, and diverse instruction datasets is expensive. If the dataset contains noise, bias, factual errors, or non-optimal answers, the model may learn and amplify these undesirable characteristics during the fine-tuning process, resulting in uneven quality of generated answers and even the generation of harmful content. Second, there is insufficient preference alignment: Because SFT is essentially imitation learning, it is difficult to explicitly incorporate more detailed human preferences (such as the safety, usefulness, honesty, and style of answers) into the model.

[0045] To this end, this application provides a fine-tuning solution for a large medical question-answering model, which can prioritize improving the efficiency and quality of constructing preference data, simplify the training process of model alignment, and improve stability.

[0046] See also Figure 1 As shown, an embodiment of the present invention discloses a method for fine-tuning a large medical question-answering model, comprising:

[0047] Step S11: Obtain an initial medical question-answering model constructed based on the basic large language model, and perform supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction data set to determine the medical question-answering model after supervised fine-tuning.

[0048] Specifically, in this embodiment, initial instruction fine-tuning is first performed, with the following steps: first, obtaining an initial medical question-answering model pre-trained based on a basic large language model; then, obtaining a pre-organized prompt word instruction dataset to determine a preset prompt word instruction dataset; the preset prompt word instruction dataset includes a number of sample data pairs; and, based on each sample data pair in the preset prompt word instruction dataset and a preset supervised fine-tuning termination condition, performing supervised fine-tuning on the initial medical question-answering model to determine a supervised fine-tuned medical question-answering model. In other words, the initial medical question-answering model for the medical field, constructed based on the basic large language model, is first subjected to supervised fine-tuning using the preset prompt word instruction dataset to complete the initial instruction fine-tuning operation.

[0049] Combine Figure 2 As shown, it can be understood that, regarding the preliminary instruction fine-tuning operation, this embodiment may include the following steps during specific implementation:

[0050] (1) Model selection: Select a suitable base large language model (Base LLM) as a starting point to build the initial medical question-answering model. This base large language model is usually pre-trained with large-scale corpus and has strong general language understanding and generation capabilities;

[0051] (2) Data preparation: Prepare or obtain an instruction fine-tuning dataset , which is the preset prompt word instruction dataset. This dataset consists of a series of sample data pairs. The format of the i-th sample data pair can be expressed as ,in, Represents the instruction, question or context input to the model for the i-th sample data pair, that is, the prompt word instruction; It is for this A high-quality, expected standard answer or model answer, that is, standard response information;

[0052] (3) SFT training: using the prepared dataset For the initial medical question answering model built on the selected basic LLM, standard supervised fine-tuning is performed. The training goal is to minimize the model generation The negative log-likelihood loss enables the model to learn based on Generate something like After the training is completed, a fine-tuning model with the ability to follow instructions is obtained, which is recorded as This step lays the foundation for subsequent optimization.

[0053] In addition, it should be understood that the solution logic proposed in this embodiment is also applicable to other fields besides medicine. The initial model and data set can be changed according to the needs of the field and the same effect can be achieved.

[0054] Step S12: Screen the sample data pairs in the preset prompt word instruction data set based on the supervised fine-tuned medical question-answering model, the preset screening rules and the pre-trained target model to determine the sample screening results.

[0055] In this embodiment, after completing the supervised fine-tuning, the supervised fine-tuned medical question answering large model is obtained. , use the large model to generate model answers, and filter the data set based on the answer generation situation In the sample data pairs, that is, based on the supervised fine-tuned medical question-answering big model, a response corresponding to the prompt word instruction in each of the sample data pairs in the preset prompt word instruction data set is generated to determine the corresponding model response information; the sample data pairs include the corresponding prompt word instruction and standard response information; based on the pre-trained target model, the preset screening rules and the model response information corresponding to each of the sample data pairs, the sample data pairs are screened to determine the sample screening results. That is to say, in this stage, in addition to using the supervised fine-tuned medical question-answering big model, this embodiment also uses the pre-trained target model as a reward model to score the answers generated by the big model to screen the sample data pairs.

[0056] Furthermore, in this embodiment, regarding the screening of the sample data pairs based on a pre-trained target model, preset screening rules and the model response information corresponding to each of the sample data pairs, the relevant steps are as follows: first obtain a pre-trained target model; for the prompt word instruction in any of the sample data pairs, determine the first reward score corresponding to the model response information of the current prompt word instruction and the second reward score corresponding to the standard response information of the current prompt word instruction based on the target model; judge whether the first reward score is less than the second reward score to determine the corresponding judgment result; if the judgment result shows that it is less than the second reward score, determine that the current sample data pair is a valid sample; if the judgment result shows that it is not less than the second reward score, determine that the current sample data pair is an invalid sample, and trigger the corresponding rejection sampling operation.

[0057] Combine Figure 2As shown, it can be understood that regarding the generation of model answers, this embodiment may include the following steps during specific implementation:

[0058] (1) Input and Generation: Traversing the Instruction Fine-tuning Dataset Each (or a selected subset) of ;

[0059] (2) Model reasoning: Each Input the initial instruction fine-tuning model obtained by step 1 training Utilize the generative capabilities of the model (e.g., using decoding strategies such as sampling and beam search) to generate a corresponding answer, denoted as ;

[0060] (3) Data pair collection: Through this process, a series of data pairs containing the model’s own output are obtained. These responses reflect The actual performance of these instructions at the current stage.

[0061] Regarding the reward model scoring, this embodiment may include the following steps during implementation:

[0062] (1) Reward model preparation: Obtain or train a target model as a reward model (RM). The core function of the RM is to evaluate the quality of an answer or the degree to which it conforms to human preferences under a given prompt and output a scalar score. The RM can be a pre-trained general reward model or a model trained for a specific domain or preference standard. The quality of the RM affects the effectiveness of this solution, but the solution itself does not have an absolute requirement for the perfection of the RM. It only needs to be able to roughly distinguish between good and bad answers.

[0063] (2) Score calculation: For the dataset For each sample index i in:

[0064] a) Calculate the bonus score of the standard answer: Enter RM and get the score ;

[0065] b) Calculate the reward score of the model answer: Enter RM and get the score .

[0066] Step S13: Determine a target preference training data set based on the sample screening results, and use the target preference training data set to further fine-tune the supervised fine-tuned medical question-answering big model to obtain a fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

[0067] In this embodiment, a target preference training data set is constructed based on the determined sample screening, that is, based on the preset data format and the corresponding model response information, the sample data pairs determined to be valid samples in the sample screening results are format converted to determine the corresponding preference training data pairs; the target preference training data set is determined by aggregating the constructed preference training data pairs.

[0068] Combine Figure 2 As shown, it can be understood that regarding the construction of the target preference training data set, this embodiment may include the following steps during specific implementation:

[0069] (1) Setting of screening conditions (based on actual conditions, the sample screening results may include valid sample results and / or invalid sample results determined based on the screening conditions, or may include results without conditional screening. If the latter is the case, only the scores corresponding to different sample data pairs are included): For each sample data pair, the core is to compare the scores of the corresponding standard answers. and model answer score ;

[0070] (2) Core selection logic (rejection sampling): Establish clear screening rules: if and only if When , the i-th sample data pair is selected to construct the preference data, otherwise it is ignored. This condition means that the reward model judges that the standard answer is better than the answer currently generated by the model. This screening based on score comparison is similar to "rejection sampling" - rejecting samples where the model performs well or better (relative to the standard answer, as judged by RM) and only retaining samples where the model has clear room for improvement;

[0071] (3) Construction of preference data pairs: For each For the i-th sample data pair of the condition, a preference data triple is constructed according to the data format required by the DPO algorithm (Direct Preference Optimization, an algorithm that optimizes directly based on human preferences):

[0072] ;

[0073] Here the standard answer is considered the "chosen" better answer , while the poor answers generated by the model are regarded as “rejected” suboptimal answers ;

[0074] (4) Dataset formation: All the data obtained through the above screening and construction steps are Collected together to form the final preference dataset for DPO training, denoted as .

[0075] Afterwards, use The supervised fine-tuned medical question-answering big model is directly subjected to preference optimization training for further fine-tuning. That is, the supervised fine-tuned medical question-answering big model is iteratively trained based on the preset preference optimization algorithm and the target preference training data set, and the parameters of the trained model in each iterative round are adjusted using the preset loss function until the model converges, and the fine-tuned target medical question-answering big model is determined.

[0076] It is understandable that regarding direct preference optimization training, this embodiment may include the following steps during specific implementation:

[0077] (1) Model and data: Fine-tune the model using the initial instructions obtained in step S11 As the model to be optimized, use the preference dataset generated in step 4 As training data;

[0078] (2) DPO training implementation: Apply the Direct Preference Optimization (DPO) algorithm to the model Further fine-tuning is performed. DPO training usually involves a specific loss function that directly operates on the model and The goal is to adjust the model parameters so that under the given prompt, the model gives The probability of giving The probability of is increased, thereby implicitly learning and encoding preference information;

[0079] (3) Final model acquisition: After the DPO training process converges, the final optimized language model is obtained, which is recorded as Compared with , it is expected to perform better in following human instructions and generating high-quality and preference-compliant answers.

[0080] In summary, this embodiment provides a large-model instruction fine-tuning solution based on rejection sampling. The solution's core innovation lies in cleverly leveraging the evaluation capabilities of the reward model (RM) to compare standard answers with answers generated by the model itself. Based on the comparison results, a "rejection sampling" strategy is implemented to select the most valuable samples to construct preference data. Finally, the direct preference optimization (DPO) algorithm is used for model training. This solution can demonstrate one or more of the following significant benefits:

[0081] (1) Significantly improve alignment efficiency and targeting: The core advantage of this solution lies in its "rejection sampling" strategy. By rewarding the precise evaluation and comparison of the model, the DPO training process can focus on key samples where the model is indeed deficient and the generated answers are inferior to the standard answers. This targeted optimization avoids redundant training on samples where the model has performed well or is of the same quality as the standard answers, thereby greatly improving learning efficiency and computing resource utilization. The model can learn to correct errors faster, making the overall alignment process more targeted and ultimately achieving better alignment results;

[0082] (2) Automatically generate high-quality preference data: This solution provides an automated, principle-based (reward model scoring) approach to construct the preference dataset required for DPO. Compared to manual labeling, which is expensive and potentially subjective, or random sampling, which is inefficient and may lack clear signals, the preference data pairs generated by this method have stronger signal strength (clear comparison of good and bad) and higher quality because they directly reflect the direction in which the model needs to be improved. This provides better "fuel" for DPO training and helps achieve better model performance;

[0083] (3) Simplify the alignment process and improve training stability: Compared with the traditional RLHF framework (Reinforcement Learning from Human Feedback), which requires three complex and potentially interdependent stages: SFT, RM training, and PPO reinforcement learning (Proximal Policy Optimization), the method flow of this solution is more streamlined. It reduces the alignment process to data screening under the guidance of SFT and RM evaluation and DPO training. Compared with reinforcement learning algorithms such as PPO, DPO itself is usually simpler to implement, has a more stable training process, and is less sensitive to hyperparameters, thereby reducing the technical threshold and failure risk of model alignment;

[0084] (4) Enhance model robustness and reliability: By continuously monitoring and correcting poor model performance, this solution helps improve the stability and reliability of the model when processing various instructions, and reduces the possibility of generating low-quality, irrelevant, or even harmful content.

[0085] (5) Good compatibility and scalability: The solution design of this embodiment has modular characteristics, which can flexibly select different basic language models, instruction data sets, reward models, and DPO implementation libraries. It can be easily integrated into the existing large-scale model training and deployment process as an effective enhancement. In addition, this solution framework also provides a foundation for the future integration of more advanced reward models or preference learning algorithms.

[0086] It can be seen that in this application, first of all, the initial medical question-answering big model that needs to be fine-tuned and is built based on the basic big language model is subjected to initial supervised fine-tuning operation using the preset prompt word instruction data set. Afterwards, the sample data pairs in the preset prompt word instruction data set are screened by the supervised fine-tuned medical question-answering big model and the target model to construct a target preference training data set. Then, the supervised fine-tuned medical question-answering big model is fine-tuned again using the target preference training data set to determine the fine-tuned target medical question-answering big model. In this way, the construction efficiency and quality of the preference data can be improved first, and the training process of the model alignment can be simplified and the stability can be improved, thereby improving the alignment accuracy and effect of the determined fine-tuned target medical question-answering big model in a more efficient and stable manner.

[0087] See also Figure 3 As shown, the embodiment of the present application also discloses a medical question-answering large model fine-tuning device, including:

[0088] A supervised fine-tuning module 11 is configured to obtain an initial medical question-answering model constructed based on the basic large language model, and to perform supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset to determine a supervised fine-tuned medical question-answering model.

[0089] A sample screening module 12 is configured to screen the sample data pairs in the preset prompt word instruction dataset based on the supervised fine-tuned medical question-answering model, preset screening rules, and a pre-trained target model to determine a sample screening result;

[0090] The instruction fine-tuning completion module 13 is used to determine the target preference training data set based on the sample screening results, and use the target preference training data set to further fine-tune the supervised fine-tuned medical question-answering big model to obtain the fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

[0091] It can be seen that in this application, first of all, the initial medical question-answering big model that needs to be fine-tuned and is built based on the basic big language model is subjected to initial supervised fine-tuning operation using the preset prompt word instruction data set. Afterwards, the sample data pairs in the preset prompt word instruction data set are screened by the supervised fine-tuned medical question-answering big model and the target model to construct a target preference training data set. Then, the supervised fine-tuned medical question-answering big model is fine-tuned again using the target preference training data set to determine the fine-tuned target medical question-answering big model. In this way, the construction efficiency and quality of the preference data can be improved first, and the training process of the model alignment can be simplified and the stability can be improved, thereby improving the alignment accuracy and effect of the determined fine-tuned target medical question-answering big model in a more efficient and stable manner.

[0092] In some specific embodiments, the supervised fine-tuning module 11 can be specifically used to: obtain an initial medical question and answer big model pre-trained based on a basic big language model; obtain a pre-organized prompt word instruction data set to determine a preset prompt word instruction data set; the preset prompt word instruction data set includes a number of sample data pairs; based on each of the sample data pairs in the preset prompt word instruction data set and the preset supervised fine-tuning termination conditions, the initial medical question and answer big model is supervised fine-tuned to determine the medical question and answer big model after supervised fine-tuning.

[0093] In some specific embodiments, the sample screening unit can be specifically used to: obtain a pre-trained target model; for the prompt word instruction in any of the sample data pairs, determine, based on the target model, a first reward score corresponding to the model response information of the current prompt word instruction and a second reward score corresponding to the standard response information of the current prompt word instruction; judge whether the first reward score is less than the second reward score to determine a corresponding judgment result; if the judgment result shows that it is less than the second reward score, determine that the current sample data pair is a valid sample; if the judgment result shows that it is not less than the second reward score, determine that the current sample data pair is an invalid sample, and trigger a corresponding rejection sampling operation.

[0094] In some specific embodiments, the instruction fine-tuning completion module 13 can be specifically used to: based on a preset data format and the corresponding model response information, perform format conversion on the sample data pairs determined to be valid samples in the sample screening results to determine the corresponding preference training data pairs; and determine the target preference training data set by aggregating the constructed preference training data pairs.

[0095] In some specific embodiments, the instruction fine-tuning completion module 13 can be specifically used to: perform iterative model training on the supervised fine-tuned medical question-answering model based on a preset preference optimization algorithm and the target preference training data set, and use a preset loss function to adjust the parameters of the trained model in each iterative round until the model converges, thereby determining the fine-tuned target medical question-answering model.

[0096] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0097] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the medical question-and-answer large model fine-tuning method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of implementing the medical question-and-answer large model fine-tuning method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for fine-tuning the large medical question-answering model. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0106] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for fine-tuning a large medical question-answering model, characterized in that: include: Obtaining an initial medical question-answering model constructed based on the basic large language model, and performing supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset to determine a supervised fine-tuned medical question-answering model; Screening the sample data pairs in the preset prompt word instruction dataset based on the supervised fine-tuned medical question-answering model, the preset screening rules, and the pre-trained target model to determine the sample screening results; Based on the sample screening results, a target preference training data set is determined, and the supervised fine-tuned medical question-answering big model is fine-tuned again using the target preference training data set to obtain a fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

2. The method for fine-tuning the medical question-answering large model according to claim 1 is characterized in that: The step of obtaining an initial medical question-answering model based on the basic large language model and fine-tuning the initial medical question-answering model based on a preset prompt word instruction dataset includes: Obtain an initial medical question-answering model pre-trained based on a basic large language model; Acquire a pre-organized prompt word instruction data set to determine a preset prompt word instruction data set; the preset prompt word instruction data set includes a plurality of sample data pairs; The initial medical question-answering big model is supervised fine-tuned based on each of the sample data pairs in the preset prompt word instruction data set and the preset supervised fine-tuning termination condition to determine the medical question-answering big model after supervised fine-tuning.

3. The method for fine-tuning the medical question-answering large model according to claim 1, characterized in that: The screening of sample data pairs in the preset prompt word instruction data set based on the supervised fine-tuned medical question-answering large model, preset screening rules, and pre-trained target model includes: Based on the supervised fine-tuned medical question-answering model, generating a response corresponding to the prompt word instruction in each of the sample data pairs in the preset prompt word instruction dataset to determine corresponding model response information; the sample data pairs include the corresponding prompt word instruction and standard response information; The sample data pairs are screened based on a pre-trained target model, preset screening rules and the model response information corresponding to each of the sample data pairs to determine a sample screening result.

4. The method for fine-tuning the medical question-answering large model according to claim 3, characterized in that: The screening of the sample data pairs based on the pre-trained target model, the preset screening rules and the model response information corresponding to each of the sample data pairs includes: Get the pre-trained target model; For the prompt word instruction in any of the sample data pairs, determining, based on the target model, a first reward score corresponding to the model response information of the current prompt word instruction and a second reward score corresponding to the standard response information of the current prompt word instruction; Determining whether the first reward score is less than the second reward score to determine a corresponding determination result; If the judgment result indicates that the score is less than the second reward score, then determining that the current sample data pair is a valid sample; If the judgment result shows that the reward score is not less than the second reward score, the current sample data pair is determined to be an invalid sample, and a corresponding rejection sampling operation is triggered.

5. The method for fine-tuning the medical question-answering large model according to claim 4 is characterized in that: The determining of the target preference training data set based on the sample screening result includes: Based on a preset data format and the corresponding model response information, format conversion is performed on the sample data pairs determined to be valid samples in the sample screening results to determine corresponding preferred training data pairs; The target preference training data set is determined by collecting the constructed preference training data pairs.

6. The method for fine-tuning the medical question-answering model according to claim 1, characterized in that: The method of further fine-tuning the supervised fine-tuned medical question answering model using the target preference training dataset includes: Based on the preset preference optimization algorithm and the target preference training data set, the supervised fine-tuned medical question-answering model is iteratively trained, and the parameters of the trained model in each iterative round are adjusted using a preset loss function until the model converges, and the fine-tuned target medical question-answering model is determined.

7. A medical question-answering large model fine-tuning device, characterized in that: include: A supervised fine-tuning module is used to obtain an initial medical question-answering model constructed based on the basic large language model, and to perform supervised fine-tuning on the initial medical question-answering model based on a preset prompt word instruction dataset to determine a supervised fine-tuned medical question-answering model; A sample screening module is used to screen the sample data pairs in the preset prompt word instruction data set based on the supervised fine-tuned medical question-answering model, preset screening rules and a pre-trained target model to determine the sample screening results; The instruction fine-tuning completion module is used to determine the target preference training data set based on the sample screening results, and use the target preference training data set to further fine-tune the supervised fine-tuned medical question-answering big model to obtain the fine-tuned target medical question-answering big model, so as to respond to medical-related questions initiated by the user terminal based on the target medical question-answering big model.

8. The medical question-answering large model fine-tuning device according to claim 7, characterized in that: The sample screening module comprises: a model response determination unit, configured to generate, based on the supervised fine-tuned medical question-answering macromodel, responses corresponding to the prompt word instructions in each of the sample data pairs in the preset prompt word instruction dataset, to determine corresponding model response information; wherein the sample data pairs include the corresponding prompt word instructions and standard response information; The sample screening unit is used to screen the sample data pairs based on the pre-trained target model, preset screening rules and the model response information corresponding to each of the sample data pairs to determine the sample screening results.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for fine-tuning the medical question-answering large model as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the medical question-answering large model fine-tuning method as described in any one of claims 1 to 6.

Images