Medical field large model answer quality evaluation method and related device
By refining the evaluation basis answering in the medical field and entered into the large model of evaluation, it solves the problem that the existing technology is difficult to accurately automatically evaluate the quality of large models to answer the quality in the medical field, and achieves efficient and accurate evaluation results.
Patent Information
- Application Number
- CN202411661156.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-01-28
AI Technical Summary
The prior art is difficult to accurately and automatically evaluate the quality of answers in large models in the medical field, resulting in inefficiency.
By obtaining the answers to the large models and corresponding questions in the medical field to be approved, the evaluation basis for refining the problem, including the essential points, selection points, and error -prone points. Essence
The automatic and accurate evaluation of the quality of the large models of the medical field has been realized, which improves efficiency and reduces dependence on doctors.
Smart Images
Figure CN119378688A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and related device for evaluating the quality of answers given by a large model in the medical field. Background Art
[0002] With the development of artificial intelligence technology, big models have made great progress and have been implemented in various professional fields. In the medical field, there are already many big models focusing on this field, providing disease diagnosis for users and auxiliary diagnosis and analysis for doctors in the form of questions and answers. In the process of developing big models in the medical field, it is necessary to determine whether the effect of the model has been improved, which requires evaluating the quality of the answers of big models in the medical field.
[0003] Traditional methods for evaluating the quality of answers from large models include BLEU (calculating the N-gram match between the generated answer and the reference standard answer), ROUGE (scoring by evaluating the overlap between the answer and the standard text (words, sequences, etc.), BERTScore (using pre-trained language models such as BERT to calculate the semantic similarity between the generated text and the reference text), etc. However, medical questions answered by large models in the medical field, such as medical rehabilitation plans, involve many aspects of knowledge and require answers to many questions. Traditional methods for evaluating the quality of answers from large models in the medical field cannot accurately evaluate the quality of answers from large models in the medical field. Currently, the most accurate way to evaluate the quality of answers from large models in the medical field is to have doctors conduct manual evaluations, which is very inefficient.
[0004] Therefore, how to provide a method for evaluating the quality of answers of large models in the medical field to achieve automatic and accurate evaluation of the quality of answers of large models in the medical field has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention
[0005] In view of the above problems, this application provides a method and related device for evaluating the quality of answers of large models in the medical field, so as to achieve the purpose of automatically and accurately evaluating the quality of answers of large models in the medical field. The specific scheme is as follows:
[0006] The first aspect of the present application provides a method for evaluating the quality of answers of a large model in the medical field, comprising:
[0007] Obtaining the answers of the big model in the medical field to be evaluated, and the questions corresponding to the answers of the big model in the medical field to be evaluated;
[0008] Obtaining an evaluation basis for the question; the evaluation basis for the question is extracted based on a reference answer to the question; the evaluation basis for the question includes at least one of the required answer points, optional answer points, and easy-to-make-mistake points of the question;
[0009] The medical field big model answer to be evaluated, the question and the evaluation basis of the question are input into the evaluation big model to obtain the evaluation result.
[0010] In a possible implementation, the answer to the medical field big model to be evaluated, the question, and the evaluation basis of the question are input into the evaluation big model to obtain the evaluation result, including:
[0011] Obtain a preconfigured quality assessment prompt template, wherein the quality assessment prompt template includes preset quality assessment task description information, assessment content filling slot, and assessment basis filling slot;
[0012] Fill the medical field big model answer to be evaluated and the question into the evaluation content filling slot, and fill the evaluation basis of the question into the evaluation basis filling slot to obtain a quality evaluation prompt;
[0013] The quality assessment prompt is input into the assessment large model to obtain the assessment result output by the assessment large model.
[0014] In a possible implementation, the method for determining the evaluation basis of the problem includes:
[0015] Obtain reference answers to the questions;
[0016] Inputting the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question output by the evaluation basis generation model; the evaluation basis generation model is obtained by supervised fine-tuning of the pre-trained large model based on the reference answer to the sample question and the evaluation basis label of the sample question;
[0017] The evaluation basis of the problem is determined based on the evaluation basis generation model output.
[0018] In a possible implementation, inputting the reference answer to the question into an evaluation basis generation model to obtain the evaluation basis of the question includes:
[0019] Obtain a preconfigured evaluation basis generation prompt template, wherein the evaluation basis generation prompt template includes preset evaluation basis generation task description information and reference answer filling slots;
[0020] Fill the reference answer of the question into the reference answer filling slot, and obtain the evaluation basis to generate prompt;
[0021] The evaluation basis generation prompt is input into the evaluation basis generation model to obtain the evaluation basis of the problem output by the evaluation basis generation model.
[0022] In a possible implementation, for each reference answer to a sample question, a method of determining an evaluation basis label of the sample question corresponding to the reference answer to the sample question includes:
[0023] Determine the quality evaluation results of the reference answers to the sample questions;
[0024] A strategy is determined based on an evaluation basis corresponding to a quality evaluation result of a reference answer to the sample question, and an evaluation basis label of the sample question corresponding to the reference answer to the sample question is determined.
[0025] In a possible implementation, if there are multiple reference answers to the question, determining the evaluation basis for the question based on the evaluation basis generation model outputted by the evaluation basis includes:
[0026] The evaluation bases of the multiple problems output by the evaluation base generation model are cross-checked and integrated to obtain the evaluation bases of the problem.
[0027] In a possible implementation, cross-checking and integrating the evaluation bases of the multiple problems output by the evaluation base generation model to obtain the evaluation bases of the problem includes:
[0028] Determine whether there is a conflict between the evaluation criteria of the plurality of problems output by the evaluation criteria generation model;
[0029] If there is a conflict, manually review the evaluation bases of the multiple questions output by the evaluation base generation model;
[0030] If there is no conflict, the evaluation bases of the multiple problems output by the evaluation base generation model are input into the evaluation base fusion model to obtain the output of the evaluation base fusion model as the evaluation base for the problem.
[0031] In a possible implementation, the step of inputting the evaluation bases of the multiple problems output by the evaluation base generation model into an evaluation base fusion model to obtain the output of the evaluation base fusion model as the evaluation base for the problem, includes:
[0032] Obtain a preconfigured evaluation basis fusion prompt template, wherein the evaluation basis fusion prompt template includes preset evaluation basis fusion task description information and an evaluation basis filling slot;
[0033] Fill the evaluation basis of the plurality of questions output by the evaluation basis generation model into the evaluation basis filling slot to obtain an evaluation basis fusion prompt;
[0034] The evaluation basis fusion prompt is input into the evaluation basis fusion model, and the output of the evaluation basis fusion model is obtained as the evaluation basis of the problem.
[0035] The second aspect of the present application provides a large model answer quality evaluation device in the medical field, comprising:
[0036] A unit for acquiring content to be evaluated, used for acquiring the medical field big model answer to be evaluated, and the question corresponding to the medical field big model answer to be evaluated;
[0037] An evaluation basis acquisition unit, used to acquire the evaluation basis of the question; the evaluation basis of the question is extracted based on the reference answer of the question; the evaluation basis of the question includes at least one of the required answer points, optional answer points and easy-to-make-mistake points of the question;
[0038] The evaluation unit is used to input the medical field big model answer to be evaluated, the question and the evaluation basis of the question into the evaluation big model to obtain the evaluation result.
[0039] In a possible implementation, the evaluation unit is specifically used to:
[0040] Obtain a preconfigured quality assessment prompt template, wherein the quality assessment prompt template includes preset quality assessment task description information, assessment content filling slot, and assessment basis filling slot;
[0041] Fill the medical field big model answer to be evaluated and the question into the evaluation content filling slot, and fill the evaluation basis of the question into the evaluation basis filling slot to obtain a quality evaluation prompt;
[0042] The quality assessment prompt is input into the assessment large model to obtain the assessment result output by the assessment large model.
[0043] In a possible implementation, the device further includes: an evaluation basis determination unit;
[0044] The evaluation is based on the determination unit, including:
[0045] A reference answer obtaining unit, used to obtain a reference answer to the question;
[0046] An evaluation basis generation unit is used to input the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question output by the evaluation basis generation model; the evaluation basis generation model is obtained by fine-tuning the pre-trained large model in a supervised manner based on the reference answer to the sample question and the evaluation basis label of the sample question. A large model with evaluation basis generation capability obtained by training;
[0047] The evaluation basis determination subunit is used to determine the evaluation basis of the problem based on the evaluation basis of the problem output by the evaluation basis generation model.
[0048] In a possible implementation, the evaluation basis generating unit is specifically used to:
[0049] Obtain a preconfigured evaluation basis generation prompt template, wherein the evaluation basis generation prompt template includes preset evaluation basis generation task description information and reference answer filling slots;
[0050] Fill the reference answer of the question into the reference answer filling slot, and obtain the evaluation basis to generate prompt;
[0051] The evaluation basis generation prompt is input into the evaluation basis generation model to obtain the evaluation basis of the problem output by the evaluation basis generation model.
[0052] In a possible implementation, the device further includes: an evaluation basis label determination unit;
[0053] The evaluation basis label determination unit is specifically used to determine, for each sample question, a quality evaluation result of the reference answer to the sample question; and to determine an evaluation basis label for the sample question corresponding to the reference answer to the sample question based on an evaluation basis determination strategy corresponding to the quality evaluation result of the reference answer to the sample question.
[0054] In a possible implementation, if there are multiple reference answers to the question, the evaluation basis determines the sub-unit, including:
[0055] The cross-fusion unit is used to cross-check and fuse the evaluation bases of the multiple problems output by the evaluation base generation model to obtain the evaluation bases of the problems.
[0056] In a possible implementation, the cross-fusion unit includes:
[0057] A conflict determination unit, used to determine whether the evaluation criteria of the plurality of problems output by the evaluation criteria generation model conflict with each other;
[0058] A first processing unit is configured to manually review the evaluation criteria of the plurality of questions output by the evaluation criteria generation model if there is a conflict;
[0059] The second processing unit is used to input the evaluation basis of the multiple problems output by the evaluation basis generation model into the evaluation basis fusion model if there is no conflict, and obtain the output of the evaluation basis fusion model as the evaluation basis of the problem.
[0060] In a possible implementation, the second processing unit is specifically configured to:
[0061] Obtain a preconfigured evaluation basis fusion prompt template, wherein the evaluation basis fusion prompt template includes preset evaluation basis fusion task description information and an evaluation basis filling slot;
[0062] Fill the evaluation basis of the plurality of questions output by the evaluation basis generation model into the evaluation basis filling slot to obtain an evaluation basis fusion prompt;
[0063] The evaluation basis fusion prompt is input into the evaluation basis fusion model, and the output of the evaluation basis fusion model is obtained as the evaluation basis of the problem.
[0064] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for evaluating the quality of answers of a large model in the medical field according to the first aspect or any implementation of the first aspect.
[0065] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0066] The memory is used to store computer programs;
[0067] The processor is used to execute the computer program so that the electronic device can implement the medical field large model answer quality assessment method of the above-mentioned first aspect or any implementation method of the first aspect.
[0068] In a fifth aspect, the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the medical field big model answer quality assessment method according to the above-mentioned first aspect or any implementation of the first aspect.
[0069] By means of the above technical scheme, the medical field big model answer quality evaluation method and related devices provided by the present application, taking into account the high professionalism and expression diversity of the questions corresponding to the medical field big model answers to be evaluated, firstly extract the evaluation basis of the question based on the reference answer to the question, and the evaluation basis of the question includes at least one of the required answer points, optional answer points and easy-to-make mistakes points of the question; when evaluating the medical field big model answers to be evaluated, the evaluation results can be obtained by inputting the medical field big model answers, questions and the evaluation basis of the questions to be evaluated into the evaluation big model. Since the evaluation basis is provided for the evaluation big model, the evaluation big model does not need to have strong medical capabilities. As long as it has basic semantic understanding and logical capabilities, it can be judged whether the core part of the question is answered in the medical field big model answers to be evaluated, and whether there are any detailed medical errors. Therefore, the scheme can realize automatic and accurate evaluation of the quality of medical field big model answers. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0071] Figure 1 A flowchart of a method for evaluating the quality of a large model answer in the medical field provided in an embodiment of the present application;
[0072] Figure 2 A schematic diagram of the structure of a large model answer quality evaluation device in the medical field provided in an embodiment of the present application;
[0073] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0074] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0077] With the development of artificial intelligence technology, big models have made great progress and have been implemented in various professional fields. In the medical field, there are already many big models focusing on this field, providing disease diagnosis for users and auxiliary diagnosis and analysis for doctors in the form of questions and answers. In the process of developing big models in the medical field, it is necessary to determine whether the effect of the model has been improved, which requires evaluating the quality of the answers of big models in the medical field.
[0078] Traditional methods for evaluating the quality of answers from large models include BLEU (calculating the N-gram match between the generated answer and the reference standard answer), ROUGE (scoring by evaluating the overlap between the answer and the standard text (words, sequences, etc.), BERTScore (using pre-trained language models such as BERT to calculate the semantic similarity between the generated text and the reference text), etc. However, medical questions answered by large models in the medical field, such as medical rehabilitation plans, involve many aspects of knowledge and require a lot of answers. Traditional methods for evaluating the quality of answers from large models in the medical field cannot accurately evaluate the quality of answers from large models in the medical field.
[0079] Currently, there is also a method of directly using large models such as GPT to evaluate the quality of large model answers. The method is generally as follows:
[0080] (1) Determine the task description, such as summarizing the user input into a sentence;
[0081] (2) Determine the description of the evaluation criteria, such as correctness, completeness, relevance, fluency, etc.;
[0082] (3) Input the above two description texts and the model’s answers to the questions into a large model such as GPT;
[0083] (4) Large models such as GPT score the quality of answers based on task type and criteria.
[0084] In order to increase the accuracy of the evaluation, some methods will optimize the description of the evaluation criteria by adding examples or using the Chain of Thought.
[0085] For the evaluation of the answer quality of large models in the medical field, if the above method of directly using large models such as GPT to evaluate the answer quality of large models is adopted, it will have a certain effect, but the following problems still exist:
[0086] (1) The field is relatively professional, and the direct evaluation effect of large models such as GPT is still insufficient;
[0087] (2) Medical answers involve a lot of relevant content, and the degree of necessity of the response content varies. It is difficult to achieve good results by directly evaluating them from the perspectives of correctness and completeness.
[0088] (3) Some detailed medical errors are difficult to detect using large models such as GPT.
[0089] In order to solve the above problems, the embodiment of the present application provides a method for evaluating the quality of answers of a large model in the medical field. The method for evaluating the quality of answers of a large model in the medical field according to the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.
[0090] Reference Figure 1 , Figure 1 A flowchart of a method for evaluating the quality of a large model answer in the medical field provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a method for evaluating the quality of answers of a large model in the medical field provided in an embodiment of the present application may include the following steps, which are described in detail below.
[0091] S101: Obtaining the medical field big model answer to be evaluated, and the question corresponding to the medical field big model answer to be evaluated;
[0092] S102: Obtaining an evaluation basis for the question; the evaluation basis for the question is extracted based on a reference answer to the question; the evaluation basis for the question includes at least one of the required answer points, optional answer points, and easy-to-make-mistake points of the question;
[0093] In the present application, the reference answer to the question may be a professional answer manually given by a doctor, or an answer from a large model in the medical field that has passed manual quality evaluation by a doctor.
[0094] In this application, the mandatory points of the question refer to the points that must be answered when answering the question. Optional points refer to the less important points to answer the question in addition to the mandatory points, which can be used as a supplement and extension of the mandatory points. Easy-to-make mistakes refer to the mistakes that are easy to make when answering the question.
[0095] It should be noted that the evaluation basis of different questions is different. For example, the key points of the question "What medicine to take for fever" are to answer the type of fever cause, answer the corresponding medicine, and give one of the specific mainstream medicine names (such as ×××). The key points for optional answers are other precautions for relieving fever. The key points of the question "Symptoms of male aging" are the names of the symptoms, such as "Symptoms of male aging include loose skin, decreased physical strength, thinning hair, osteoporosis, decreased sexual function, memory loss and mood swings." The key points for optional answers are the specific descriptions of each symptom, such as "With age, the skin will lose elasticity and become loose", "The body's metabolic rate decreases, muscle mass decreases, leading to decreased physical strength", "Hair will become sparse, and even hair loss will occur", "Bone density will decrease, and osteoporosis is prone to occur", "Sexual function will decline with age, including erectile function and ejaculation function", "Brain function will gradually decline, memory will also decline", "Emotions will become unstable and prone to mood swings".
[0096] S103: Input the medical field big model answer to be evaluated, the question and the evaluation basis of the question into the evaluation big model to obtain the evaluation result.
[0097] In this application, the evaluation big model can be any existing big model, such as GPT, Spark big model, Wenxin Yiyan big model, etc. This application does not make any limitation on this.
[0098] For ease of understanding, this application provides an example of a large model answer quality evaluation in the medical field, as follows:
[0099] The answer to the large model in the medical field to be evaluated is:
[0100] "Ankle injury massage therapy is mainly used to treat ankle sprains, pain, swelling and other symptoms. Specifically, it includes the following situations:
[0101] 1. Ankle sprain: Through massage techniques, blood circulation can be promoted, muscle spasms can be relieved, pain can be alleviated, and wound healing can be accelerated.
[0102] 2. Ankle arthritis: Massage can stimulate meridian points, regulate qi and blood, relieve inflammation and reduce pain.
[0103] 3. Swelling of ankle joint: Massage can promote lymphatic circulation, eliminate edema and reduce swelling.
[0104] 4. Limited ankle joint mobility: Massage can relieve muscle tension, increase joint mobility, and improve joint function.
[0105] 5. Ankle pain: Massage can stimulate meridian points, regulate qi and blood, relieve pain and improve joint function.
[0106] In short, ankle injury massage treatment mainly treats ankle sprains, pain, swelling and other symptoms. It stimulates meridian points, regulates qi and blood, relieves inflammation and pain, promotes wound healing, and improves joint function. "
[0107] The big model in the medical field to be evaluated answers the corresponding question: "What disease can be cured by massage treatment for ankle injury?"
[0108] The evaluation basis for the above questions is:
[0109] "Key points that must be answered: 1. Massage therapy can relieve pain, promote blood circulation, and restore joint function in ankle injuries. 2. Massage therapy needs to be combined with specific circumstances, such as injury type, severity, and personal physique. Mild sprains, ligament strains and other soft tissue injuries can be treated with massage. Severe joint injuries such as fractures may be aggravated by massage.
[0110] Key points for answering: 1. During the rehabilitation process of ankle joint injury, massage therapy can be combined with other treatment methods (such as drug therapy, physical therapy, rehabilitation training, etc.) to improve the treatment effect. 2. Ankle joint injuries can be prevented by strengthening exercises to improve joint stability, paying attention to sports safety and avoiding trauma. 3. Stay relaxed when receiving massage therapy to avoid excessive tension that causes muscle stiffness and affects the treatment effect.
[0111] The evaluation results are shown in the following table:
[0112]
[0113] In this embodiment, considering that the questions corresponding to the answers of the medical field big model to be evaluated are highly professional and diverse in expression, the evaluation basis of the question is first extracted based on the reference answer to the question, and the evaluation basis of the question includes at least one of the required points, optional points and easy-to-make mistakes of the question; when evaluating the answers of the medical field big model to be evaluated, the evaluation results can be obtained by inputting the answers of the medical field big model to be evaluated, the questions and the evaluation basis of the questions into the evaluation big model. Since the evaluation basis is provided for the evaluation big model, the evaluation big model does not need to have strong medical capabilities. As long as it has basic semantic understanding and logical capabilities, it can be judged whether the core part of the question is answered in the answer of the medical field big model to be evaluated, and whether there are any detailed medical errors. Therefore, this solution can realize automatic and accurate evaluation of the quality of the medical field big model answers.
[0114] In another embodiment of the present application, the implementation method of inputting the medical field big model answer to be evaluated, the question, and the evaluation basis of the question into the evaluation big model in S103 to obtain the evaluation result is described, and the method may include:
[0115] Obtain a preconfigured quality assessment prompt template, wherein the quality assessment prompt template includes preset quality assessment task description information, an assessment content filling slot, and an assessment basis filling slot; fill the medical field big model answer to be evaluated and the question into the assessment content filling slot, and fill the assessment basis of the question into the assessment basis filling slot to obtain a quality assessment prompt; input the quality assessment prompt into the assessment big model to obtain the assessment result output by the assessment big model.
[0116] In this application, the quality assessment task description information is used to indicate the execution points of the quality assessment task. For ease of understanding, this application provides an example of quality assessment task description information, as follows:
[0117] The quality assessment task description information mainly includes:
[0118] (1) Use the chain of thought method to gradually infer the evaluation results.
[0119] (2) First, for each required point, determine whether the model answers the point correctly. Explain the degree of compliance (not answered, basically met, completely met, not met), and explain the reason for the judgment.
[0120] (3) For each key point, determine whether the model answers the question correctly.
[0121] (4) For each error-prone point, determine whether the model’s answer violates this error-prone point and explain the reason.
[0122] (5) If the model encounters content that cannot be judged during the judgment process, the output will be content for manual review.
[0123] (6) Assign different scores and weights to the correct answers to the required points and optional points, and finally calculate the weighted score.
[0124] (7) Evaluate the quality of the answer based on the weighted score (wrong, acceptable, excellent)
[0125] (8) However, if a point that is prone to error is violated, it will be directly judged as an error.
[0126] (9) If no error is identified and there is content awaiting manual review, the evaluation result is uncertain.
[0127] In the present application, in a possible implementation, the evaluation basis of the question can be manually generated by the doctor based on the reference answer to the question. Considering that the doctor manually generates the evaluation basis of the question, it is inefficient and costly, and is greatly affected by the doctor's professional level. In the present application, a method for automatically generating the evaluation basis of the question is also provided.
[0128] In the present application, a method for automatically generating an evaluation basis for the problem is also provided, and the method may include the following steps:
[0129] S201: Obtaining a reference answer to the question;
[0130] S202: Inputting the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question output by the evaluation basis generation model; the evaluation basis generation model is obtained by fine-tuning the pre-trained large model in a supervised manner based on the reference answer to the sample question and the evaluation basis label of the sample question;
[0131] In this application, the pre-trained large model can be a large model with a parameter scale of 2.6B to 13B, such as the open source Llama3. In this application, the pre-trained iFlytek Medical large model can be used.
[0132] In the present application, the reference answer to the sample question may be the answer to the sample question manually given by a doctor, or the answer to the sample question given by a large medical field model that has passed manual quality evaluation by a doctor.
[0133] For the answers to sample questions given by the big model in the medical field, after the doctors manually conduct quality evaluation, the quality evaluation results can be divided into excellent answers, qualified answers and incorrect answers. The quality evaluation results of the answers to sample questions manually given by doctors can be considered excellent answers.
[0134] The strategies for determining the evaluation basis for excellent answers, qualified answers, and incorrect answers can be as follows:
[0135] (1) Excellent answers: Select the core points from the excellent answers as the required points. Select the secondary points as the optional points.
[0136] (2) Qualified answer: Based on the doctor's evaluation reasons, correct the shortcomings of the qualified answer, and then select the core points as the required points. Select the secondary points as the optional points.
[0137] (3) Incorrect answers: Arrange and rewrite the answers based on the reasons for the errors.
[0138] In the present application, for each reference answer to a sample question, the method for determining an evaluation basis label for the sample question corresponding to the reference answer to the sample question can be as follows: determine a quality evaluation result of the reference answer to the sample question, and then determine a strategy based on the evaluation basis corresponding to the quality evaluation result of the reference answer to the sample question, and determine an evaluation basis label for the sample question corresponding to the reference answer to the sample question.
[0139] In the present application, the step of inputting the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis for the question includes: obtaining a preconfigured evaluation basis generation prompt template, wherein the evaluation basis generation prompt template includes preset evaluation basis generation task description information and a reference answer filling slot; filling the reference answer to the question into the reference answer filling slot to obtain an evaluation basis generation prompt; inputting the evaluation basis generation prompt into the evaluation basis generation model to obtain the evaluation basis for the question output by the evaluation basis generation model.
[0140] S203: Determine the evaluation basis of the problem based on the evaluation basis generation model output.
[0141] In one possible implementation, if there is only one reference answer to the question, then there is also only one evaluation basis for the question output by the evaluation basis generation model. In this case, the evaluation basis for the question output by the evaluation basis generation model can be determined as the evaluation basis for the question.
[0142] In some scenarios, there may be multiple reference answers to the question, and the evaluation basis for the question output by the evaluation basis generation model may also be multiple. In this case, the multiple evaluation bases for the question output by the evaluation basis generation model may be cross-checked and integrated to obtain the evaluation basis for the question. By cross-checking and integrating the multiple evaluation bases for the question output by the evaluation basis generation model, the reliability of the evaluation basis for the question can be increased.
[0143] The step of cross-checking and integrating the evaluation bases of the multiple problems output by the evaluation base generation model to obtain the evaluation bases of the problems includes:
[0144] Determine whether there is a conflict between the evaluation criteria of the plurality of problems output by the evaluation criteria generation model;
[0145] If there is a conflict, manually review the evaluation bases of the multiple questions output by the evaluation base generation model;
[0146] If there is no conflict, the evaluation bases of the multiple problems output by the evaluation base generation model are input into the evaluation base fusion model to obtain the output of the evaluation base fusion model as the evaluation base for the problem.
[0147] In a possible implementation, the evaluation basis for the multiple problems output by the evaluation basis generation model is input into the evaluation basis fusion model to obtain the output of the evaluation basis fusion model as the evaluation basis for the problem, including: obtaining a preconfigured evaluation basis fusion prompt template, wherein the evaluation basis fusion prompt template contains preset evaluation basis fusion task description information and an evaluation basis filling slot; filling the evaluation basis for the multiple problems output by the evaluation basis generation model into the evaluation basis filling slot to obtain an evaluation basis fusion prompt; inputting the evaluation basis fusion prompt into the evaluation basis fusion model to obtain the output of the evaluation basis fusion model as the evaluation basis for the problem.
[0148] It should be noted that the cross-checking and fusion of the evaluation criteria of the multiple questions output by the evaluation basis generation model includes the cross-checking and fusion of the required points, the cross-checking and fusion of the optional points and the cross-checking and fusion of the easy-to-make mistakes points.
[0149] In the present application, the evaluation basis fusion model is obtained by fine-tuning a pre-trained large model in a supervised manner using multiple evaluation basis samples and fused evaluation basis labels corresponding to the multiple evaluation basis samples.
[0150] In this application, the pre-trained large model can be a large model with a parameter scale of 2.6B to 13B, such as the open source Llama3. In this application, the pre-trained iFlytek Medical large model can be used.
[0151] The above introduces a method for evaluating the quality of answers of a large model in the medical field provided by an embodiment of the present application. The following will introduce a device for executing the above-mentioned method for evaluating the quality of answers of a large model in the medical field.
[0152] See also Figure 2 , Figure 2 This is a schematic diagram of a medical field large model answer quality evaluation device provided in an embodiment of the present application. Figure 2 As shown, the medical field large model answer quality evaluation device includes:
[0153] The to-be-evaluated content acquisition unit 11 is used to acquire the medical field big model answer to be evaluated, and the question corresponding to the medical field big model answer to be evaluated;
[0154] The evaluation basis acquisition unit 12 is used to acquire the evaluation basis of the question; the evaluation basis of the question is extracted based on the reference answer of the question; the evaluation basis of the question includes at least one of the required answer points, optional answer points and easy-to-make mistakes points of the question;
[0155] The evaluation unit 13 is used to input the medical field big model answer to be evaluated, the question and the evaluation basis of the question into the evaluation big model to obtain the evaluation result.
[0156] In a possible implementation, the evaluation unit is specifically used to:
[0157] Obtain a preconfigured quality assessment prompt template, wherein the quality assessment prompt template includes preset quality assessment task description information, assessment content filling slot, and assessment basis filling slot;
[0158] Fill the medical field big model answer to be evaluated and the question into the evaluation content filling slot, and fill the evaluation basis of the question into the evaluation basis filling slot to obtain a quality evaluation prompt;
[0159] The quality assessment prompt is input into the assessment large model to obtain the assessment result output by the assessment large model.
[0160] In a possible implementation, the device further includes: an evaluation basis determination unit;
[0161] The evaluation is based on the determination unit, including:
[0162] A reference answer obtaining unit, used to obtain a reference answer to the question;
[0163] An evaluation basis generation unit is used to input the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question output by the evaluation basis generation model; the evaluation basis generation model is obtained by supervised fine-tuning of a pre-trained large model based on the reference answer to the sample question and the evaluation basis label of the sample question;
[0164] The evaluation basis determination subunit is used to determine the evaluation basis of the problem based on the evaluation basis of the problem output by the evaluation basis generation model.
[0165] In a possible implementation, the evaluation basis generating unit is specifically used to:
[0166] Obtain a preconfigured evaluation basis generation prompt template, wherein the evaluation basis generation prompt template includes preset evaluation basis generation task description information and reference answer filling slots;
[0167] Fill the reference answer of the question into the reference answer filling slot, and obtain the evaluation basis to generate prompt;
[0168] The evaluation basis generation prompt is input into the evaluation basis generation model to obtain the evaluation basis of the problem output by the evaluation basis generation model.
[0169] In a possible implementation, the device further includes: an evaluation basis label determination unit;
[0170] The evaluation basis label determination unit is specifically used to determine, for each sample question, a quality evaluation result of the reference answer to the sample question; and to determine an evaluation basis label for the sample question corresponding to the reference answer to the sample question based on an evaluation basis determination strategy corresponding to the quality evaluation result of the reference answer to the sample question.
[0171] In a possible implementation, if there are multiple reference answers to the question, the evaluation basis determines the sub-unit, including:
[0172] The cross-fusion unit is used to cross-check and fuse the evaluation bases of the multiple problems output by the evaluation base generation model to obtain the evaluation bases of the problems.
[0173] In a possible implementation, the cross-fusion unit includes:
[0174] A conflict determination unit, used to determine whether the evaluation criteria of the plurality of problems output by the evaluation criteria generation model conflict with each other;
[0175] A first processing unit is configured to manually review the evaluation criteria of the plurality of questions output by the evaluation criteria generation model if there is a conflict;
[0176] The second processing unit is used to input the evaluation basis of the multiple problems output by the evaluation basis generation model into the evaluation basis fusion model if there is no conflict, and obtain the output of the evaluation basis fusion model as the evaluation basis of the problem.
[0177] In a possible implementation, the second processing unit is specifically configured to:
[0178] Obtain a preconfigured evaluation basis fusion prompt template, wherein the evaluation basis fusion prompt template includes preset evaluation basis fusion task description information and an evaluation basis filling slot;
[0179] Fill the evaluation basis of the plurality of questions output by the evaluation basis generation model into the evaluation basis filling slot to obtain an evaluation basis fusion prompt;
[0180] The evaluation basis fusion prompt is input into the evaluation basis fusion model, and the output of the evaluation basis fusion model is obtained as the evaluation basis of the problem.
[0181] The present application also provides an electronic device in an embodiment. Figure 3 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0182] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0183] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0184] Also provided in an embodiment of the present application is a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the medical field large model answer quality assessment methods provided in the embodiments of the present application.
[0185] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any one of the medical field large model answer quality assessment methods provided in the embodiment of the present application.
[0186] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0187] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0188] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A method for evaluating the quality of answers of large models in the medical field, characterized in that: include: Obtaining the answers of the big model in the medical field to be evaluated, and the questions corresponding to the answers of the big model in the medical field to be evaluated; Obtaining an evaluation basis for the question; the evaluation basis for the question is extracted based on a reference answer to the question; the evaluation basis for the question includes at least one of the required answer points, optional answer points, and easy-to-make-mistake points of the question; The medical field big model answer to be evaluated, the question and the evaluation basis of the question are input into the evaluation big model to obtain the evaluation result.
2. The method according to claim 1, characterized in that The answer to the medical field big model to be evaluated, the question and the evaluation basis of the question are input into the evaluation big model to obtain the evaluation result, including: Obtain a preconfigured quality assessment prompt template, wherein the quality assessment prompt template includes preset quality assessment task description information, assessment content filling slot, and assessment basis filling slot; Fill the medical field big model answer to be evaluated and the question into the evaluation content filling slot, and fill the evaluation basis of the question into the evaluation basis filling slot to obtain a quality evaluation prompt; The quality assessment prompt is input into the assessment large model to obtain the assessment result output by the assessment large model.
3. The method according to claim 1, characterized in that The method for determining the evaluation basis of the said question includes: Obtain reference answers to the questions; Inputting the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question output by the evaluation basis generation model; the evaluation basis generation model is obtained by supervised fine-tuning of the pre-trained large model based on the reference answer to the sample question and the evaluation basis label of the sample question; The evaluation basis of the problem is determined based on the evaluation basis generation model output.
4. The method according to claim 3, characterized in that The step of inputting the reference answer to the question into the evaluation basis generation model to obtain the evaluation basis of the question includes: Obtain a preconfigured evaluation basis generation prompt template, wherein the evaluation basis generation prompt template includes preset evaluation basis generation task description information and reference answer filling slots; Fill the reference answer of the question into the reference answer filling slot, and obtain the evaluation basis to generate prompt; The evaluation basis generation prompt is input into the evaluation basis generation model to obtain the evaluation basis of the problem output by the evaluation basis generation model.
5. The method according to claim 3, characterized in that: For each reference answer to a sample question, a method for determining an evaluation basis label of the sample question corresponding to the reference answer to the sample question includes: Determine the quality evaluation results of the reference answers to the sample questions; A strategy is determined based on an evaluation basis corresponding to a quality evaluation result of a reference answer to the sample question, and an evaluation basis label of the sample question corresponding to the reference answer to the sample question is determined.
6. The method according to claim 3, characterized in that If there are multiple reference answers to the question, determining the evaluation basis for the question based on the evaluation basis generation model outputted by the evaluation basis generation model includes: The evaluation bases of the multiple problems output by the evaluation base generation model are cross-checked and integrated to obtain the evaluation bases of the problem.
7. The method according to claim 6, characterized in that The step of cross-checking and integrating the evaluation bases of the multiple problems output by the evaluation base generation model to obtain the evaluation bases of the problem includes: Determine whether there is a conflict between the evaluation criteria of the plurality of problems output by the evaluation criteria generation model; If there is a conflict, manually review the evaluation bases of the multiple questions output by the evaluation base generation model; If there is no conflict, the evaluation bases of the multiple problems output by the evaluation base generation model are input into the evaluation base fusion model to obtain the output of the evaluation base fusion model as the evaluation base for the problem.
8. The method according to claim 7, characterized in that The step of inputting the evaluation basis of the multiple problems output by the evaluation basis generation model into the evaluation basis fusion model to obtain the output of the evaluation basis fusion model as the evaluation basis of the problem includes: Obtain a preconfigured evaluation basis fusion prompt template, wherein the evaluation basis fusion prompt template includes preset evaluation basis fusion task description information and an evaluation basis filling slot; Fill the evaluation basis of the plurality of questions output by the evaluation basis generation model into the evaluation basis filling slot to obtain an evaluation basis fusion prompt; The evaluation basis fusion prompt is input into the evaluation basis fusion model, and the output of the evaluation basis fusion model is obtained as the evaluation basis of the problem.
9. A large model answer quality evaluation device in the medical field, characterized in that: include: A unit for acquiring content to be evaluated, used for acquiring the medical field big model answer to be evaluated, and the question corresponding to the medical field big model answer to be evaluated; An evaluation basis acquisition unit, used to acquire the evaluation basis of the question; the evaluation basis of the question is extracted based on the reference answer of the question; the evaluation basis of the question includes at least one of the required answer points, optional answer points and easy-to-make-mistake points of the question; The evaluation unit is used to input the medical field big model answer to be evaluated, the question and the evaluation basis of the question into the evaluation big model to obtain the evaluation result.
10. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the medical field large model answer quality assessment method as described in any one of claims 1 to 8.
11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the medical field large model answer quality assessment method as described in any one of claims 1 to 8.
12. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the medical field large model answer quality assessment method as described in any one of claims 1 to 8.