Large model training data generation method in medical field and related equipment

Abstract

The invention provides a large model training data generation method in the medical field, which takes real case data as reference, calls a large language model through an automatic program, enables the large language model to generate problem data, reference data and answer data in batches, and guides the large language model to generate and optimize model fine adjustment data through a set prompt prompt word. Therefore, the specialized model fine adjustment data is generated for the industry model in the medical field, and the training time of the subsequent industry large model is saved. In addition, due to the fact that the case data is high in obtaining difficulty and much in privacy information, compared with direct obtaining of real case information, the technical scheme disclosed by the invention has the advantages of being low in cost and high in privacy.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a method for generating training data for large models in the medical field and related equipment. Background Art

[0002] With the rapid development of modern medicine and artificial intelligence technology, deep learning model technology has gradually been introduced into the medical field to improve diagnosis and treatment effects. Deep learning models, especially large language models, have shown great potential in the medical field. They can provide efficient diagnosis and personalized treatment plans by analyzing massive data. However, the application of large models in the medical field, especially in the field of rehabilitation medicine, also faces the problems of insufficient data quantity and uneven data quality, which directly affects the performance and reliability of the model. In order to solve these problems, fine-tuning data generation methods have emerged and become an important means to improve the application effect of large models in the field of rehabilitation medicine.

[0003] Fine-tuning data generation methods provide a solid foundation for the application of large models in rehabilitation medicine. At present, many existing generation methods are insufficient in terms of data diversity and accuracy, which makes it difficult for the generated data to meet the needs of complex rehabilitation medical scenarios. For example, although data generation based on traditional statistical methods can improve model performance to a certain extent, there are still many limitations in terms of data coverage and detail richness. For example, when using general large models for data generation, existing methods often simply require a single model to directly give question-answer pairs and do simple cleaning, which cannot take into account the unique content in specific vertical fields such as rehabilitation medicine. This shows that the existing fine-tuning data generation methods still need to be further improved to better support the application of large models in rehabilitation medicine and improve the accuracy and effectiveness of their diagnosis and treatment. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a method for generating large model training data in the medical field.

[0005] Based on the above purpose, the present application provides a method for generating training data for a large model in the medical field, characterized in that it includes: obtaining case data, and generating a first prompt text based on the case data. Inputting the first prompt text into a preset first language model so that the first language model outputs question data associated with the case data. Generating a second prompt text based on the question data so that the first language model outputs reference data. The reference data is used to express the processing process of the first language model generating question data. Generating a third prompt text based on the reference data so that the first language model outputs answer data. Generate model fine-tuning data based on the question data, reference data and answer data.

[0006] In some embodiments, a first prompt text is input into a preset first language model so that the first language model outputs question data associated with the case data, specifically including: generating a first scoring text and a first screening criterion according to the question data. The first screening criterion is set based on the model fine-tuning data. A first scoring text is input into a preset second language model so that the second language model outputs first scoring information associated with the question data. The first scoring information is screened according to the first screening criterion, and the question data corresponding to the first scoring information that does not meet the first screening criterion is deleted.

[0007] In some embodiments, the first scoring information includes: question fluency score, question relevance score, question harm score, complexity score and personalization score. The first screening criteria include: the question harm score is 0, and the question fluency score, question relevance score, complexity score and personalization score are not less than half of the total score.

[0008] In some embodiments, a second prompt text is generated based on the question data so that the first language model outputs reference data, specifically including: generating a second scoring text and a second screening criterion based on the reference data. The second screening criterion is set based on the model fine-tuning data. The second scoring text is input into a preset third language model so that the third language model outputs second scoring information associated with the reference data. A second optimized text is generated, the second scoring information is screened according to the second screening criterion, and the reference data corresponding to the second scoring information that does not meet the second screening criterion, the question data corresponding to the reference data, and the second optimized text are input into the first language model so that the first language model optimizes the reference data.

[0009] In some embodiments, the second scoring information includes: a reference fluency score, a reference relevance score, a reference harm score, a reference accuracy score, a reference rationality score, and a specificity score. The second screening criteria include: the reference harm score is 0, and the reference fluency score, the reference relevance score, the reference accuracy score, the reference rationality score, and the specificity score are not less than half of the total score.

[0010] In some embodiments, generating a third prompt text according to the reference data so that the first language model outputs answer data specifically includes: generating a third scoring text and a third screening criterion according to the reference data. The third screening criterion is set based on the model fine-tuning data. Inputting the third scoring text into the preset fourth language model so that the fourth language model outputs third scoring information associated with the answer data. Generating a third optimized text, screening the third scoring information according to the third screening criterion, and inputting the answer data corresponding to the third scoring information that does not meet the third screening criterion, the reference data and question data corresponding to the answer data, and the third optimized text into the first language model so that the first language model optimizes the answer data.

[0011] In some embodiments, the third scoring information includes: answer fluency score, answer relevance score, answer harm score, answer accuracy score and answer rationality score. The third screening criteria include: the answer harm score is 0, and the answer fluency score, answer relevance score, answer accuracy score and answer rationality score are not less than half of the total score.

[0012] The present application also provides a device for generating training data for a large model in the medical field, characterized in that it includes: a case reading module, used to obtain case data, and generate a first prompt text based on the case data. A question generation module, used to input the first prompt text into a preset first language model, so that the first language model outputs question data associated with the case data. A reference data generation module, used to generate a second prompt text based on the question data, so that the first language model outputs reference data. Among them, the reference data is used to express the processing process of the first language model generating question data. An answer generation module, used to generate a third prompt text based on the reference data, so that the first language model outputs answer data. A fine-tuning data integration module, used to generate model fine-tuning data based on the question data, reference data and answer data.

[0013] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the program.

[0014] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, wherein the computer instructions are used to enable a computer to execute any of the above methods.

[0015] From the above, it can be seen that the method for generating training data for a large model in the medical field provided by this application uses real case data as a reference, calls a large language model through an automated program, and enables the large language model to generate question data, reference data, and answer data in batches, and guides the large language model to generate and optimize model fine-tuning data through the set prompt words, thereby generating specialized model fine-tuning data for the medical field industry model, saving the training time of subsequent industry large models. And because case data is difficult to obtain and has more privacy information, compared with directly obtaining real case information, the technical solution disclosed in this application also has the advantages of low cost and strong privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a method for generating large model training data in the medical field provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a method for screening problematic data provided by an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a reference data screening method provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a flow chart of a method for screening answer data provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of a device for generating large-scale model training data in the medical field provided in an embodiment of the present application;

[0022] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0024] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] like Figure 1 As shown, the present application provides a method for generating large model training data in the medical field, characterized by comprising:

[0026] Step S1, obtaining case data, and generating a first prompt text according to the case data.

[0027] In this embodiment, the case data is obtained through legal channels, authorized by the patient and his / her family, and the personal privacy information of the patient and his / her family is concealed, and only the information of the case content is included. At the same time, the obtained case data is only used for the generation of current training data.

[0028] The case data can be in the following format:

[0029] Patient profile

[0030] Age: 47

[0031] Gender: Female

[0032] Medical history

[0033] Chief complaint: loss of consciousness, fever, dyspnea and right hemiplegia for two months

[0034] Related symptoms: dysphagia and slurred speech, two months

[0035] Past medical history: (omitted)

[0036] Physical examination results: (omitted here)

[0037] Dialogue session: (omitted here)

[0038] Diagnosis and treatment prescription: (omitted here)

[0039] The first prompt text is a prompt for prompting the large model to generate text content, wherein the first prompt text can be flexibly adjusted according to the required training data.

[0040] The first prompt text includes:

[0041] You are an expert in generating question-answer datasets. Your task is to generate questions suitable for training large models in the medical field based on the case content I provide. The question description should be clear and reasonable, and there can only be one question in one sentence.

[0042] The case content is as follows:

[0043] Patient profile

[0044] (Omitted here)

[0045] Medical history

[0046] (Omitted here)

[0047] Past medical history: (omitted)

[0048] Physical examination results: (omitted here)

[0049] Dialogue session: (omitted here)

[0050] Diagnosis and treatment prescription: (omitted here)

[0051] Your questions are to be used to construct a question-answer dataset for training large models in the medical field. You should consider the given content from many aspects, and the question content should be as diverse as possible within the scope of the case content. You can extract the diagnosis results in the case content to ask questions, describe the patient's feelings, ask questions about the examination results, and refer to the questions asked by patients or doctors during the consultation stage. When asking questions, you can only describe part of the condition, or you can explain the condition in detail as a whole. You can also extract content from previous medical history, but it must not exceed the scope of the given case content. Pay special attention that your questions should be given from the perspective of "patients" or "family members". In each question, you need to first introduce the condition to the doctor. For long questions, you should first describe the condition in detail as a whole. Please note that there will be no case content as a prompt when answering questions, so when generating questions, you must ensure that the content in the question is specific enough. For example, you cannot just say "imaging data was shown", "MRI and PET scans both showed abnormalities", "this is my past medical history", etc. in the question description. You need to describe the relevant content in detail in the question. When asking for medical rehabilitation advice or treatment plans, you must summarize the condition, explain what symptoms have occurred, and what diagnosis the doctor has made. Note that you also need to consider the patient's professional level. You can use the patient's dialogue during the consultation process as a reference, and put the content that exceeds his level in the objective examination section or the summary section above. For example, you cannot just say "comprehensively my father's current condition and diagnosis results", but you need to give a clear and specific description of these situations. On the other hand, you need to ask questions in a variety of ways within the scope of the content. You can extract questions from the diagnosis results in the case content, or extract questions from the consultation dialogue process, and ask questions about medical advice, etc. Questions cannot be repeated, but they cannot exceed the scope of the case content. Finally, the questions need to consider the patient's professional level and be as colloquial as possible. Professional content can be described separately in the objective examination part. The description should be specific and detailed. There is no limit on the number of words in the question. Make sure that the question can be answered independently without the help of the case content and the context. Please generate fifteen questions according to the above model, and each question only asks one content. Please use Chinese to output the questions directly, and do not add other descriptions.

[0052] Step S2: input a first prompt text into a preset first language model so that the first language model outputs question data associated with the case data.

[0053] The first language model is a pre-trained general large language model, which can be reasonably selected according to actual application needs, for example, it can be one of ChatGPT, Gemini and Claude. Multiple sets of first prompt texts can be input into the first language model through an automated program, so that the first language model automatically outputs multiple question data.

[0054] Step S3: Generate a second prompt text according to the question data, so that the first language model outputs reference data, wherein the reference data is used to express the processing process of the first language model generating the question data.

[0055] Among them, similar to the first prompt text, the second prompt text is a prompt designed for the question data, which is used to prompt the first language model to generate and output reference data for the question data, and obtain reference information related to the patient's condition by outputting the reference data corresponding to the question data to the first language model, so as to provide a reference basis for the subsequent generation of answer information. Multiple sets of second prompt texts can be input to the first language model through an automated program, so that the first language model automatically outputs reference data corresponding to the question data.

[0056] The second prompt text includes:

[0057] You are an expert in question-answering dataset processing. Your task is to generate the process of thinking about the corresponding questions based on the given case content. The answer description should be clear and reasonable.

[0058] The case content is as follows:

[0059] (Add case content here)

[0060] The questions you need to answer are:

[0061] (Add question content here)

[0062] Please think about each question separately. Pay special attention to the fact that your answer should be true and reliable and cannot exceed the given case content. At the same time, there is no case content as a prompt in the actual scenario, so the factual content cannot exceed the scope of the question description. Specifically, you should pay attention to check your output content and whether there are any factual inconsistencies in the thinking process. On the one hand, your thinking process cannot have objective facts beyond the description of the question. For example, there is no mention of imaging data or medical history in the question, but the information extracted from the case content appears in the thinking process. When answering the question, there is no case content as a reference, so you must think and answer based on the question. On the other hand, your thinking process must be generated according to the facts of the case content to ensure that the answer is true and accurate. Finally, there must be specific measures in the treatment recommendations and expected effects. This measure should be extracted from the case content. For example, for medication recommendations, the name, dosage, usage, and frequency of the drug should be given, and rehabilitation treatment should include rehabilitation treatment items, frequency, purpose, etc. The output you give should be formatted in JSON, with the following structure:

[0063] {"Possible Diagnosis":"",

[0064] "Supporting evidence":"",

[0065] "Unsupported parts":"",

[0066] "Further inspection suggestions":"",

[0067] "Treatment recommendations":"Drug name 1, dosage, usage, frequency, drug name 2, dosage, usage, frequency, ...Rehabilitation treatment project 1, frequency, purpose, Rehabilitation treatment project 2, frequency, purpose, ...",

[0068] "Expected effect":""

[0069] }

[0070] If any part does not apply, please answer "Not applicable". Please directly output the results of your thinking process for all questions in Chinese without adding any other description.

[0071] Step S4, generating a third prompt text according to the reference data so that the first language model outputs answer data.

[0072] Among them, similar to the first prompt text, the third prompt text is a prompt designed for the answer data, which is used to prompt the first language model to generate and output the answer data corresponding to the reference data, wherein the answer data includes the answer to the consultation content in the corresponding question data, which is generated by the reference information of the corresponding patient question generated according to the question data in the reference data, and the information of each patient generated in the question data, the reference data and the answer data corresponds to each other. Multiple sets of third prompt texts can be input into the first language model through an automated program, so that the first language model automatically outputs multiple answers.

[0073] The third prompt text includes:

[0074] You are an expert in processing question-answering datasets. Your task is to generate answers to the given questions, based on the given case content and thinking process, suitable for training large models in the medical field. The answer description should be clear and reasonable.

[0075] The case content is as follows:

[0076] (Add case content here)

[0077] The questions you need to answer and the corresponding thinking process are as follows:

[0078] (Add the question content and corresponding thinking process here)

[0079] Pay special attention to the fact that your answer should be given from the perspective of a "doctor". The answer should be true and reliable and should not exceed the given case content. At the same time, there is no case content as a prompt in the actual scenario, so the factual content cannot exceed the scope of the question description. Specifically, you should pay attention to check your output content and whether there are any errors of inconsistent facts in the answer. On the one hand, your answer cannot contain objective facts beyond the description of the question. For example, there is no mention of imaging data or medical history in the question, but the answer contains data extracted from the case content, etc. When answering the question, there is no case content as a reference, so you should think and answer from the question. On the other hand, your answer must be generated according to the facts of the case content to ensure the authenticity and accuracy of the answer. Finally, if you need to give treatment suggestions and expected effects in the answer, you must have specific measures. This measure should be extracted from the corresponding module of the case content and the thinking process. For example, for medication suggestions, the name, dosage, usage, and frequency of the drug should be given, and rehabilitation treatment should include rehabilitation treatment items, frequency, purpose, etc. Please use Chinese to directly output the answer results of the question without adding any other descriptions.

[0080] Step S5, generating model fine-tuning data based on the question data, reference data and answer data.

[0081] Among them, the data corresponding to different patients generated from the question data, reference data, and answer data are organized into multiple question-answer groups, and multiple case data are integrated into model fine-tuning data. Model fine-tuning data is used to train special models in the medical field.

[0082] Among them, the integration of fine-tuning data can be achieved through text matching and JSON processing methods. First, the six parts of content and answer data in the previously obtained question data and reference data are extracted and stored in the dictionary respectively, and then they are filled into a fixed template to automatically obtain multiple sets of fine-tuning data.

[0083] The fixed templates used include:

[0084] 1.{"src":"(add the question field content here)",

[0085] "tgt":"(add the answer field content here)"}

[0086] 2.{"src":"You are a rehabilitation medical consultation expert. Please answer the following patient questions:

[0087] (Add question field content here)

[0088] You need to output your thinking process and final answer step by step. The thinking process includes possible diagnoses, supporting evidence, unsupported parts, further examination suggestions, treatment suggestions and expected effects, and then give your final answer based on the content of the thinking process. ",

[0089] "tgt":"

[0090] Thought Process:

[0091] Possible diagnosis: (fill in the fields output by the model below),

[0092] Supporting evidence:

[0093] Unsupported parts:,

[0094] Further inspection suggestions:

[0095] Treatment recommendations:

[0096] Expected results:

[0097] answer:"}

[0098] 3.{"src":"If the patient has the following symptoms:

[0099] (Add supporting evidence field content here)

[0100] What disease might the patient have? ",

[0101] "tgt":"(add possible diagnosis field content here)"}(If the possible diagnosis or supporting evidence is not applicable, this data will not be generated)

[0102] 4.{"src":"What treatment methods are usually used for this disease (add possible diagnosis field content here)?",

[0103] "tgt":"(add treatment recommendation field content here)"}(If the possible diagnosis or treatment recommendation is not applicable, this data will not be generated)

[0104] In some embodiments, Figure 2 As shown, step S2 specifically includes:

[0105] Step S21, generating a first scoring text and a first screening criterion according to the question data, wherein the first screening criterion is set based on the model fine-tuning data.

[0106] Step S22: inputting a first scoring text into a preset second language model so that the second language model outputs first scoring information associated with the question data.

[0107] In this embodiment, the second language model is used to score the question data generated by the first language model based on the first scoring text and the first screening criteria, and to verify through the large language model whether the question data generated by the first language model meets the screening criteria of the model fine-tuning data. The second language model can also be controlled to mark or eliminate the question data that does not meet the screening criteria, thereby achieving the output of high-quality question data.

[0108] Preferably, in order to improve the reliability of the first scoring information, the second language model can be a language model cluster composed of multiple large language models. The first scoring text is input into multiple different large language models in the language model cluster so that all large prediction models in the language model cluster output scoring information for the question data, and the highest and lowest scores are removed, and the average score of other language models is calculated, and the average score is used as the first scoring information.

[0109] Among them, the language model cluster refers to a large number of different general-purpose commercial large models such as ChatGPT, Claude, Qwen, Gemini, etc., and a collection of the same general-purpose commercial large models using different prompt templates. In this example, two sets of prompt templates are set for each large model to let the model play the role of artificial intelligence evaluator and professional physician in the field of rehabilitation medicine respectively. Each large model in the collection will evaluate the quality of the generated questions separately, and finally use a python script to aggregate all the results and calculate the average value to obtain the final result.

[0110] The first scoring text is used to be input into the second language model, so that the second language model outputs a score for the question data.

[0111] The first screening criterion is set according to the demand for model fine-tuning data, and is used to screen out problematic data suitable for training large industry models.

[0112] The first scoring information is obtained according to the first screening criteria, and the second language model scores the question data according to the first screening criteria, and generates the first scoring information by aggregating the scoring information of multiple dimensions. Multiple sets of first scoring texts can be input into the second language model through an automated program, so that the second language model automatically outputs multiple sets of first scoring information.

[0113] Step S23, screening the first scoring information according to the first screening criterion, and deleting the problem data corresponding to the first scoring information that does not meet the first screening criterion.

[0114] Among them, by deleting the problem information that does not meet the first screening criteria in the problem data, it is avoided that the bad data interferes with the large industry model to be trained. In the present application, when too much problem information is deleted in step S23, step S2 can be re-executed to generate more problem information and screen it according to the standards of steps S21-step S23, so as to ensure that sufficient model fine-tuning data is provided for the industries that need to be trained later.

[0115] The first scoring text includes:

[0116] You are an AI assessor who specializes in assessing the quality of questions generated for given case content as a dataset of question-answer pairs for training large models in the medical field (another template: you are a professional physician in the field of rehabilitation medicine, and now you need to assess some given questions based on the case content of a patient). Your main goal is to assess the questions based on the fluency of the question content, relevance to the given case content, harmfulness, complexity, and personalization. Please use the following scale to assess each criterion:

[0117] (Add rating scale here)

[0118] The case you are referring to is:

[0119] (Add the case content here)

[0120] The questions you need to assess are:

[0121] (Add question content here)

[0122] The output you provide should be formatted in JSON and have the following structure:

[0123] {"fluency":"",

[0124] "Relevance":"",

[0125] "Harmfulness":"",

[0126] "Complexity":"",

[0127] "Level of personalization":""}

[0128] For each part, directly fill in the corresponding score of the question. Please directly output a Json format data without adding any other description.

[0129] In some embodiments, the first scoring information includes: a question fluency score, a question relevance score, a question harmfulness score, a complexity score, and a personalization degree score.

[0130] The first screening criteria include: the question harmfulness score is 0, and the question fluency score, question relevance score, complexity score and personalization score are not less than half of the total score.

[0131] In this embodiment, in order to ensure that the answers output by the subsequent industry large model training do not have obvious errors, the present application provides a screening standard for question data, wherein the first scoring information includes:

[0132] The question information is scored for fluency, relevance, harmfulness, complexity and personalization respectively. The scoring criteria are as follows:

[0133] Question Fluency Scoring:

[0134] 1: The sentences are completely unreadable and incomprehensible.

[0135] 2: There are some coherent phrases in the sentence, but the whole sentence is difficult to understand.

[0136] 3: The sentence is generally fluent, but contains a few difficult-to-understand parts.

[0137] 4: The sentences are relatively fluent, easy to understand, and the expression is natural.

[0138] 5: The sentences are very fluent, the expression is clear and easy to understand, without any difficult parts.

[0139] Question relevance rating:

[0140] 1: Completely irrelevant. The question has no obvious connection with the case content and may involve different topics.

[0141] 2: Partially related. The question slightly involves case information, but deviates from the core content and has a weak connection with the field of rehabilitation medicine.

[0142] 3: Mostly relevant. The question covers some important information in the case, but omits some key content.

[0143] 4: Highly relevant, the questions accurately reflect the key points in the case and cover the main rehabilitation issues.

[0144] 5: Completely relevant. The questions accurately capture the core content of the case and reflect details and in-depth thinking.

[0145] Problem severity rating:

[0146] 1: The problem has obvious ethical and safety issues;

[0147] 0: The problem does not have any ethical safety issues;

[0148] Problem Complexity Rating:

[0149] 1: The question is very simple and does not involve the complex factors of rehabilitation treatment. It is just a basic statement or factual question.

[0150] 2: The problem has some depth, but remains on the surface and lacks guidance for rehabilitation decision-making.

[0151] 3: The questions are of moderate complexity, covering the main factors in rehabilitation treatment, able to guide reasonable discussions and decisions, somewhat challenging but easy to understand.

[0152] 4: The question covers multiple complex variables and requires in-depth reasoning, which is suitable for expert discussion or special rehabilitation situations.

[0153] 5: The questions involve highly complex logic or interdisciplinary content and are difficult to answer.

[0154] Personalization rating:

[0155] 1: There is a complete lack of personalization, and the questions are generic and not relevant to the specific patient.

[0156] 2: Mildly personalized, questions partially involve the patient's background or needs, but are relatively general.

[0157] 3: Moderately personalized, the questions take into account some individual factors of the patient but do not explore the patient's special needs in depth.

[0158] 4: Highly personalized, the questions fully reflect the individual differences, medical history and rehabilitation needs of the patients.

[0159] 5: Completely personalized, with questions that deeply consider the patient’s specific background and unique challenges in recovery.

[0160] Question coverage: the proportion of the content covered by the questions in the case content.

[0161] Score the question data based on the current scoring criteria. When the question hazard score is not 0, it is judged that the question data has ethical safety issues in the diagnosis of patients. The current question may pose a personal threat to patients or other people. At this time, all questions with a hazard score of not 0 should be deleted or re-entered into the prompt to optimize them. When the question fluency score and the question relevance score are less than 2.5 points, the current question is of little help to the training process of the industry large model required for subsequent training, and should also be deleted directly. Question data that meets the requirements will be added to the question data set.

[0162] Preferably, a first comparison text may be input into the language model cluster so that the language model cluster evaluates the proportion of the content of the problem data set in the content of the case data. The first comparison text specifically includes:

[0163] You are an AI assessor who specializes in calculating the proportion of the content covered by a given set of questions in a case. Your main goal is to think about the meaning of the question content and calculate the proportion of the information contained in the set to the given case content based on all the question content:

[0164] The case you are referring to is:

[0165] (Add the case content here)

[0166] The set of questions you need to assess are:

[0167] (Add the current question set content here)

[0168] Please directly output an integer from 0 to 100 to indicate the percentage of the content covered by the given set of questions in the content of the case, without adding any other description.

[0169] After the comparison, questions can be further generated by inputting the first optimized text into the first language model and conducting multiple rounds of dialogue with the first language model based on the previous dialogue content, so that the first language model continues to output the required question data. The first optimized text specifically includes:

[0170] The questions you have currently generated cover approximately ()% of the case content. Please follow the above ideas and generate 15 more questions in different directions for the uncovered parts. Please generate some questions that are less complex, easier to answer, more personalized, and fully reflect the individual differences of patients. You still need to adhere to the principles of generating questions mentioned earlier. The question description should be specific and detailed, and there should be no questions that rely on the case content to be understood. Please output the questions directly in Chinese and do not add any other descriptions.

[0171] Among them, "Please generate some questions that are less complex, easier to answer, more personalized, and fully reflect the individual differences of patients" will be automatically adjusted by the script according to the average complexity and average personalization level of the current question set. For example, when the average complexity of the current question set is lower than 3, "less complex and easier to answer" will be adjusted to "more complex, requiring in-depth reasoning based on professional knowledge to answer."

[0172] Then repeat the above process, re-input the optimized question data and the first scored text into the language model cluster, score the optimized question data and remove the question data that does not meet the standards, and add the question data that meets the requirements to the question data set. When the content of the question set accounts for more than 95%, the content in the question set is input as question information into the first language model for the next step.

[0173] In some embodiments, Figure 3 As shown, step S3 specifically includes:

[0174] Step S31, generating a second scoring text and a second screening criterion according to the reference data, wherein the second screening criterion is set based on the model fine-tuning data.

[0175] Step S32: input the second scoring text into the preset third language model, so that the third language model outputs second scoring information associated with the reference data.

[0176] In this embodiment, the third language model is used to score the reference data generated by the first language model based on the second scoring text and the second screening criteria, and to verify through the large language model whether the reference data generated by the first language model meets the screening criteria for the model fine-tuning data. The third language model can also be controlled to mark or eliminate the reference data that does not meet the screening criteria, thereby achieving the output of high-quality reference data.

[0177] Preferably, in order to improve the reliability of the second scoring information, the third language model can be a language model cluster composed of multiple large language models. The second scoring text is input into multiple different large language models in the language model cluster so that all large prediction models in the language model cluster output scoring information for the reference data, and the highest and lowest scores in the scoring are removed, and the average score of other language models is calculated, and the average score is used as the second scoring information.

[0178] Among them, the language model cluster refers to a large number of different general-purpose commercial large models such as ChatGPT, Claude, Qwen, Gemini, etc., and a collection of the same general-purpose commercial large models using different prompt templates. In this example, two sets of prompt templates are set for each large model to let the model play the role of artificial intelligence evaluator and professional physician in the field of rehabilitation medicine respectively. Each large model in the collection will evaluate the quality of the generated questions separately, and finally use an automated script to aggregate all the results and calculate the average value to obtain the final result.

[0179] The second scoring text is used to be input into the third language model, so that the third language model outputs a score for the reference data.

[0180] The second screening criterion is set according to the demand for model fine-tuning data and is used to screen out reference data suitable for training large industry models.

[0181] The second scoring information is obtained according to the second screening criteria, and the third language model scores the reference data according to the second screening criteria, and generates the second scoring information by aggregating the scoring information of multiple dimensions. Multiple sets of second scoring texts can be input into the third language model through an automated program, so that the third language model automatically outputs multiple sets of second scoring information.

[0182] Step S33, generating a second optimized text, filtering the second scoring information according to the second filtering criteria, and inputting the reference data corresponding to the second scoring information that does not meet the second filtering criteria, the problem data corresponding to the reference data, and the second optimized text into the first language model, so that the first language model optimizes the reference data.

[0183] Among them, by optimizing the reference information in the reference data that does not meet the second screening criteria, it is avoided that bad data will interfere with the large industry model to be trained.

[0184] The second scoring text includes:

[0185] You are an AI assessor who specializes in assessing the quality of the thought process for a given problem. Your main goal is to refer to the given case content and assess the thought process based on its fluency, relevance to the given problem, harmfulness, accuracy, rationality, and specificity. Please use the following scale to assess each criterion:

[0186] (Add scale here)

[0187] The case you are referring to is:

[0188] (Add the case content here)

[0189] The questions given are:

[0190] (Add question content here)

[0191] The thought processes you need to evaluate are:

[0192] (Add thought process here)

[0193] The output you provide should be formatted in JSON and have the following structure:

[0194] {"fluency":"",

[0195] "Relevance":"",

[0196] "Harmfulness":"",

[0197] "accuracy":"",

[0198] "Reasonableness":"",

[0199] "Specificity":""}

[0200] For each part, directly fill in the corresponding score of the question. Please directly output a Json format data without adding any other description.

[0201] In some embodiments, the second scoring information includes: a reference fluency score, a reference relevance score, a reference harmfulness score, a reference accuracy score, a reference rationality score, and a specificity score.

[0202] The second screening criteria include: the reference harmfulness score is 0, and the reference fluency score, reference relevance score, reference accuracy score, reference rationality score and specificity score are not less than half of the total score.

[0203] In this embodiment, in order to ensure that the answers output by the subsequent industry large model training do not have obvious errors, the present application provides a screening standard for reference data, wherein the second scoring information includes:

[0204] The reference information is scored for reference fluency, reference relevance, reference harm, reference accuracy, reference rationality and specificity respectively. The scoring criteria are as follows:

[0205] Fluency:

[0206] 1: The sentences are completely unreadable and incomprehensible.

[0207] 2: There are some coherent phrases in the sentence, but the whole sentence is difficult to understand.

[0208] 3: The sentence is generally fluent, but contains a few difficult-to-understand parts.

[0209] 4: The sentences are relatively fluent, easy to understand, and the expression is natural.

[0210] 5: The sentences are very fluent, the expression is clear and easy to understand, without any difficult parts.

[0211] Relevance:

[0212] 1: Completely irrelevant, the thought process generated has no obvious connection to the question content and may involve completely different topics.

[0213] 2: Partially related, but overall deviates from the key content of the question, is only slightly related to the peripheral information of the question, and does not answer the core of the question.

[0214] 3: Basically relevant. The thinking process answers some important information in the question, but ignores some key points.

[0215] 4: Highly relevant. The thinking process accurately reflects the key content of the question and covers the main rehabilitation issues.

[0216] 5: Completely relevant. The thought process accurately captures the core content of the question and reflects details and in-depth thinking.

[0217] Hazards:

[0218] 1: There are obvious ethical safety issues in the thinking process.

[0219] 0: There are no ethical safety issues in the thinking process.

[0220] accuracy:

[0221] 1: There are obvious factual errors in the thinking process that conflict with the case content and cannot be used as a reference for answering questions.

[0222] 2: There are no factual errors in the thinking process as a whole, but there are a few problems with logical inference.

[0223] 3: The overall thinking process contains no factual errors or logical problems, but it is difficult to give effective suggestions for the unresolved parts of the problem.

[0224] 4: The thinking process is basically an accurate thinking about the problem, and it can also provide effective inspection suggestions for the parts of the problem that cannot be solved at present, but some content is not accurate enough.

[0225] 5: The thinking process is very good at thinking about the problem, and it can solve the problem well when used as a reference.

[0226] Reasonableness:

[0227] 1: In the "Possible Diagnosis" and "Supporting Evidence" sections, there is a lot of factual content outside the problem description, which cannot be derived from the problem content in actual scenarios;

[0228] 2: In the "Possible Diagnosis" and "Supporting Evidence" sections, there is a small amount of factual content other than the problem description, which is difficult to draw conclusions based on the problem content in actual scenarios;

[0229] 3: There is almost no factual content other than the problem description in the "Possible Diagnosis" and "Supporting Evidence" sections, and a certain amount of content needs to be inferred based on the problem content;

[0230] 4: There is no factual content other than the problem description in the "Possible Diagnosis" and "Supporting Evidence" sections, and there is a small amount of content that needs to be inferred based on the problem content;

[0231] 5: The content in the "Possible Diagnosis" and "Supporting Evidence" sections can be directly derived from the question content;

[0232] Specificity:

[0233] 1: The "further examination recommendations" and "treatment recommendations" sections are general and cannot be used as a guide;

[0234] 2: In the "further examination recommendations" and "treatment recommendations" sections, only a rough plan is given, and it is difficult to come up with specific measures;

[0235] 3: Some general guidance is given in the "further examination suggestions" and "treatment suggestions" sections, which are of certain reference value;

[0236] 4: The "further examination recommendations" and "treatment recommendations" sections describe the examination plan, rehabilitation treatment items, and medication plan, but the specific frequency and course of treatment are not clear enough;

[0237] 5: In the "further examination suggestions" and "treatment suggestions" sections, the examination plan, rehabilitation treatment items, and medication plan are clearly explained, including the examination items, the usage and dosage of the drugs, and the frequency and purpose of the rehabilitation items;

[0238] In particular, the scoring help here has a small number of variations, and adjustments need to be made for the "not applicable" parts of the thinking process. For example, if the "further inspection suggestions" in the thinking process are "not applicable", then the specific degree part of the scale does not need to evaluate the "further inspection suggestions".

[0239] The reference data is scored based on the current scoring criteria. When the reference hazard score is not 0, it is judged that the reference data at this time has ethical safety issues in the diagnosis of patients. The current reference data may pose a personal threat to patients or other people. At this time, all reference data with reference hazard scores not 0 should be re-entered into the prompt for optimization. When the reference fluency score, reference relevance score, reference accuracy score, reference rationality score and specificity score are lower than 2.5 points, the current reference data is less helpful for the training process of the subsequent industry large model required for training, and should be optimized through multiple rounds of dialogue with the first language model.

[0240] Preferably, since the problem data has already undergone a round of screening, the screening criteria for the reference data can be further improved. For example, the screening criteria for the reference relevance score can be increased to 3 points, and the screening criteria for the reference accuracy score, reference rationality score and specificity can be increased to 3.5 points, thereby improving the quality of the model fine-tuning data.

[0241] Wherein, it is also possible to provide that on the basis of the first language model generating the reference data, a second optimized text is inputted into the first language model for multiple rounds of dialogue so that the first language model optimizes the reference data, and the second optimized text specifically includes:

[0242] Please check your thinking process. There are some continuous phrases in your current thinking process sentence, but it is still difficult to read. In the "further examination suggestions" and "treatment suggestions" sections, only a general plan is given, and it is difficult to come up with specific measures. Please try to modify your thinking process to make your sentence expression fluent enough and without incomprehensible content. In the "further examination suggestions" and "treatment suggestions" sections, the inspection plan, rehabilitation treatment items, and medication plan are clearly explained, accurate to the inspection items, the usage and dosage of the drugs, and the frequency and purpose of the rehabilitation items. The output you provide should still be formatted in JSON, with the following structure:

[0243] {"Possible Diagnosis":"",

[0244] "Supporting evidence":"",

[0245] "Unsupported parts":"",

[0246] "Further inspection suggestions":"",

[0247] "Treatment recommendations":"Drug name 1, dosage, usage, frequency, drug name 2, dosage, usage, frequency, ...Rehabilitation treatment project 1, frequency, purpose, Rehabilitation treatment project 2, frequency, purpose, ...",

[0248] "Expected effect":""

[0249] }

[0250] If any part does not apply, please answer "Not applicable". Please directly output the results of your thinking process for all questions in Chinese without adding any other description.

[0251] In particular, the second optimized text has a certain amount of variations, which will be automatically and dynamically adjusted according to the score of the reference data. For example, when the specific degree score meets the requirements, the texts such as "In the "Further Examination Suggestions" and "Treatment Suggestions" sections, only a general plan is given, and it is difficult to come up with specific measures" and "In the "Further Examination Suggestions" and "Treatment Suggestions" sections, the examination plan, rehabilitation treatment items, and medication plan are clearly described, accurate to the examination items, usage and dosage of drugs, and the frequency and purpose of rehabilitation items" should be removed. In this application, when no reference data that meets the standards can be obtained after five rounds of dialogue, the reference data with the highest average score and a harmfulness score of 0 generated during the dialogue will be selected as the result.

[0252] In some embodiments, Figure 4 As shown, step S4 specifically includes:

[0253] Step S41, generating a third scoring text and a third screening criterion according to the reference data, wherein the third screening criterion is set based on the model fine-tuning data.

[0254] Step S42: input the third scoring text into the preset fourth language model, so that the fourth language model outputs third scoring information associated with the answer data.

[0255] In this embodiment, the fourth language model is used to score the answer data generated by the first language model based on the third scoring text and the third screening criteria, and to verify through the large language model whether the answer data generated by the first language model meets the screening criteria of the model fine-tuning data. The fourth language model can also be controlled to mark or eliminate the answer data that does not meet the screening criteria, thereby achieving the output of high-quality answer data.

[0256] Preferably, in order to improve the reliability of the third scoring information, the fourth language model can be a language model cluster composed of multiple large language models. The third scoring text is input into multiple different large language models in the language model cluster so that all large prediction models in the language model cluster output scoring information for the answer data, and the highest and lowest scores are removed, and the average score of other language models is calculated, and the average score is used as the third scoring information.

[0257] Among them, the language model cluster refers to a large number of different general-purpose commercial large models such as ChatGPT, Claude, Qwen, Gemini, etc., and a collection of the same general-purpose commercial large models using different prompt templates. In this example, two sets of prompt templates are set for each large model to let the model play the role of artificial intelligence evaluator and professional physician in the field of rehabilitation medicine respectively. Each large model in the collection will evaluate the quality of the generated questions separately, and finally use an automated script to aggregate all the results and calculate the average value to obtain the final result.

[0258] The third scoring text is used to be input into the fourth language model, so that the fourth language model outputs a score for the answer data.

[0259] The third screening criterion is set according to the demand for model fine-tuning data and is used to screen out answer data suitable for training large industry models.

[0260] The third scoring information is obtained according to the third screening criteria, and the fourth language model scores the answer data according to the third screening criteria, and generates the third scoring information by aggregating the scoring information of multiple dimensions. Multiple sets of third scoring texts can be input into the fourth language model through an automated program, so that the fourth language model automatically outputs multiple sets of third scoring information.

[0261] Step S43, generating a third optimized text, filtering the third scoring information according to the third filtering criteria, and inputting the answer data corresponding to the third scoring information that does not meet the third filtering criteria, the reference data and question data corresponding to the answer data, and the third optimized text into the first language model, so that the first language model optimizes the answer data.

[0262] Among them, by optimizing the answer information in the answer data that does not meet the third screening criteria, it is possible to avoid interference from bad data in the large industry model being trained.

[0263] Among them, the third scoring text includes:

[0264] You are an AI assessor who specializes in assessing the quality of answers to given questions and professional thought processes. Your main goal is to refer to the given case content and evaluate the thought process based on the fluency of the answer content, relevance to the given question, harmfulness, accuracy and rationality. Please use the following scale to evaluate each criterion:

[0265] (Add scale here)

[0266] The case you are referring to is:

[0267] (Add the case content here)

[0268] The questions and thinking process given are:

[0269] (Add question content and thinking process content here)

[0270] The responses you need to evaluate are:

[0271] (Add answer here)

[0272] The output you provide should be formatted in JSON and have the following structure:

[0273] {"fluency":"",

[0274] "Relevance":"",

[0275] "Harmfulness":"",

[0276] "accuracy":"",

[0277] "reasonableness":""}

[0278] For each part, directly fill in the corresponding score of the question. Please directly output a Json format data without adding any other description.

[0279] In some embodiments, the third scoring information includes: an answer fluency score, an answer relevance score, an answer harmfulness score, an answer accuracy score, and an answer rationality score.

[0280] The third screening criteria include: the answer harmfulness score is 0, and the answer fluency score, answer relevance score, answer accuracy score and answer rationality score are not less than half of the total score.

[0281] In this embodiment, in order to ensure that the answers output by the subsequent industry large model training do not have obvious errors, the present application provides a screening standard for answer data, wherein the third scoring information includes:

[0282] The answer fluency score, answer relevance score, answer harmfulness score, answer accuracy score and answer rationality score of the question information are scored separately. The scoring criteria are as follows:

[0283] Fluency:

[0284] 1: The sentences are completely unreadable and incomprehensible.

[0285] 2: There are some coherent phrases in the sentence, but the whole sentence is difficult to understand.

[0286] 3: The sentence is generally fluent, but contains a few difficult-to-understand parts.

[0287] 4: The sentences are relatively fluent, easy to understand, and the expression is natural.

[0288] 5: The sentences are very fluent, the expression is clear and easy to understand, without any difficult parts.

[0289] Relevance:

[0290] 1: Completely irrelevant. The generated answer has no obvious connection with the question content and may be about a completely different topic.

[0291] 2: Partially related, but overall deviates from the key content of the question, is only slightly related to the peripheral information of the question, and does not answer the core of the question.

[0292] 3: Mostly relevant, the generated answer provides some important information in the question, but ignores some key points.

[0293] 4: Highly relevant. The generated answers accurately reflect the key content of the questions and cover the main rehabilitation issues.

[0294] 5: Completely relevant. The generated answer accurately captures the core content of the question and provides a positive response to the question.

[0295] Hazards:

[0296] 1: There are obvious ethical safety issues in the generated answers.

[0297] 0: There are no ethical safety issues in the generated answers.

[0298] accuracy:

[0299] 1: The generated answer contains obvious factual errors that conflict with the case content.

[0300] 2: The generated answers are generally free of factual errors, but have a small number of logical inference problems.

[0301] 3: The generated answer is generally free of factual errors and logical problems, but does not fully capture the content of the thought process.

[0302] 4: The generated answer is basically a feasible solution to the problem, which basically contains the content obtained during the thinking process.

[0303] 5: The generated answer provides a good answer to the question, comprehensively covers the content of the thinking process, and well solves the patient's problem or provides a clear examination plan.

[0304] Reasonableness:

[0305] 1: The answers contain a lot of factual content beyond the question description, which cannot be derived from the question content in actual scenarios.

[0306] 2: The answer contains a small amount of factual content beyond the question description, which is difficult to draw conclusions based on the question content in actual scenarios.

[0307] 3: There is almost no factual content in the answer beyond the question description, and a certain amount of content needs to be inferred based on the question content.

[0308] 4: There is no factual content in the answer beyond the question description, and there is a small amount of content that needs to be inferred based on the question content.

[0309] 5: The content of the answer can be directly derived from the content of the question.

[0310] The answer data is scored based on the current scoring criteria. When the answer hazard score is not 0, it is judged that the answer data at this time has ethical safety issues in the diagnosis of the patient, and the current answer may pose a personal threat to the patient or other people. At this time, all answers with a hazard score of not 0 should be re-entered into the prompt for optimization. When the answer fluency score, answer relevance score, answer accuracy score and answer rationality score are less than 2.5 points, the current answer is of little help to the training process of the industry large model required for subsequent training, and should be optimized through multiple rounds of dialogue with the first language model.

[0311] In which, it is also possible to provide a method in which, based on the first language model generating the answer data, a third optimized text is inputted into the first language model for multiple rounds of dialogue so that the first language model optimizes the answer data, and the third optimized text specifically includes:

[0312] Please check your answer. There are some continuous phrases in your answer, but it is still difficult to read. The answer is generally free of factual errors and logical problems, but it does not fully cover the content of the thinking process. Please try to modify your answer to make your expression smooth enough, without any content that is difficult to understand, and fully cover the content of the thinking process, and solve the patient's problem well or provide a clear examination plan. Please use Chinese to directly output the answer to the question without adding any other description.

[0313] In particular, the third optimized text has a certain amount of variations, which will be automatically and dynamically adjusted according to the score of the answer data. For example, when the accuracy score is only 2 points, "the answer is generally free of factual errors and logical problems, but does not fully cover the content of the thinking process" should be modified to "the generated answer contains obvious factual errors that conflict with the case content". In this application, when no reference data that meets the standard can be obtained after five rounds of dialogue, the reference data with the highest average score and a harmfulness score of 0 generated during the dialogue will be selected as the result.

[0314] As can be seen from the above embodiments of the present application, the present application uses real case data as a reference, calls the large language model through an automated program, and enables the large language model to generate question data, reference data, and answer data in batches, and guides the large language model to generate and optimize the model fine-tuning data through the set prompt words, thereby generating specialized model fine-tuning data for the medical field industry model, saving the subsequent training time of the industry large model. And because case data is difficult to obtain and has more privacy information, compared with directly obtaining real case information, the technical solution disclosed in this application also has the advantages of low cost and strong privacy.

[0315] This application also provides screening criteria for question data, reference data, and answer data. By setting specific prompts, the large language model is guided to score and screen question data, reference data, and answer data, thereby obtaining more accurate model fine-tuning data for subsequent industry large model training.

[0316] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0317] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0318] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0319] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0320] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0321] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0322] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for generating large model training data in the medical field.

[0323] refer to Figure 5 , a device for generating large model training data in the medical field, comprising:

[0324] The case reading module 601 is used to obtain case data and generate a first prompt text according to the case data.

[0325] The question generation module 602 is used to input a first prompt text into a preset first language model so that the first language model outputs question data associated with the case data.

[0326] The reference data generation module 603 is used to generate a second prompt text according to the question data so that the first language model outputs reference data. The reference data is used to express the processing process of the first language model generating the question data.

[0327] The answer generation module 604 is used to generate a third prompt text according to the reference data so that the first language model outputs answer data.

[0328] The fine-tuning data integration module 605 is used to generate model fine-tuning data according to the question data, reference data and answer data.

[0329] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0330] The device of the above embodiment is used to implement the corresponding method for generating large model training data in the medical field in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0331] Based on the same technical concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating large model training data in the medical field described in any of the above embodiments is implemented.

[0332] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0333] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0334] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0335] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0336] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0337] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0338] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0339] The electronic device of the above-mentioned embodiment is used to implement the corresponding method for generating large model training data in the medical field in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0340] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for generating large model training data in the medical field as described in any of the above embodiments.

[0341] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0342] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for generating large model training data in the medical field as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0343] It should be noted that the embodiments of the present application can be further described in the following manner:

[0344] A method for generating large model training data in the medical field, characterized by comprising:

[0345] Acquire case data, and generate a first prompt text based on the case data.

[0346] A first prompt text is input into a preset first language model so that the first language model outputs question data associated with the case data.

[0347] A second prompt text is generated according to the question data so that the first language model outputs reference data, wherein the reference data is used to express the processing process of the first language model generating the question data.

[0348] A third prompt text is generated based on the reference data so that the first language model outputs answer data.

[0349] Generate model fine-tuning data based on question data, reference data and answer data.

[0350] Preferably, inputting a first prompt text into a preset first language model so that the first language model outputs question data associated with the case data specifically includes:

[0351] A first scoring text and a first screening criterion are generated according to the question data, wherein the first screening criterion is set based on the model fine-tuning data.

[0352] The first scoring text is input into a preset second language model so that the second language model outputs first scoring information associated with the question data.

[0353] The first scoring information is screened according to the first screening criterion, and the problem data corresponding to the first scoring information that does not meet the first screening criterion is deleted.

[0354] Preferably, the first scoring information includes: a question fluency score, a question relevance score, a question harmfulness score, a complexity score and a personalization degree score.

[0355] The first screening criteria include: the question harmfulness score is 0, and the question fluency score, question relevance score, complexity score and personalization score are not less than half of the total score.

[0356] Preferably, generating a second prompt text according to the question data so that the first language model outputs reference data specifically includes:

[0357] A second scoring text and a second screening criterion are generated according to the reference data, wherein the second screening criterion is set based on the model fine-tuning data.

[0358] The second scoring text is input into the preset third language model so that the third language model outputs second scoring information associated with the reference data.

[0359] Generate a second optimized text, filter the second scoring information according to the second filtering criteria, and input the reference data corresponding to the second scoring information that does not meet the second filtering criteria, the problem data corresponding to the reference data, and the second optimized text into the first language model, so that the first language model optimizes the reference data.

[0360] Preferably, the second scoring information includes: a reference fluency score, a reference relevance score, a reference harmfulness score, a reference accuracy score, a reference rationality score and a specificity score.

[0361] The second screening criteria include: the reference harmfulness score is 0, and the reference fluency score, reference relevance score, reference accuracy score, reference rationality score and specificity score are not less than half of the total score.

[0362] Preferably, generating the third prompt text according to the reference data so that the first language model outputs the answer data specifically includes:

[0363] A third scoring text and a third screening criterion are generated according to the reference data, wherein the third screening criterion is set based on the model fine-tuning data.

[0364] The third scoring text is input into the preset fourth language model so that the fourth language model outputs third scoring information associated with the answer data.

[0365] A third optimized text is generated, the third scoring information is screened according to a third screening criterion, and the answer data corresponding to the third scoring information that does not meet the third screening criterion, the reference data and question data corresponding to the answer data, and the third optimized text are input into the first language model, so that the first language model optimizes the answer data.

[0366] Preferably, the third scoring information includes: an answer fluency score, an answer relevance score, an answer harmfulness score, an answer accuracy score and an answer rationality score.

[0367] The third screening criteria include: the answer harmfulness score is 0, and the answer fluency score, answer relevance score, answer accuracy score and answer rationality score are not less than half of the total score.

[0368] A device for generating large model training data in the medical field, characterized by comprising:

[0369] The case reading module is used to obtain case data and generate a first prompt text according to the case data.

[0370] The question generation module is used to input a first prompt text into a preset first language model so that the first language model outputs question data associated with the case data.

[0371] The reference data generation module is used to generate a second prompt text according to the question data so that the first language model outputs reference data. The reference data is used to express the processing process of the first language model generating the question data.

[0372] The answer generation module is used to generate a third prompt text according to the reference data so that the first language model outputs answer data.

[0373] The fine-tuning data integration module is used to generate model fine-tuning data based on question data, reference data and answer data.

[0374] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the program.

[0375] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute any of the methods described above.

[0376] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0377] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0378] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0379] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating large model training data in the medical field, characterized in that: include: Acquire case data, and generate a first prompt text according to the case data; Inputting the first prompt text into a preset first language model so that the first language model outputs question data associated with the case data; Generate a second prompt text according to the question data, so that the first language model outputs reference data; wherein the reference data is used to express the processing process of the first language model generating the question data; generating a third prompt text according to the reference data so that the first language model outputs answer data; Model fine-tuning data is generated according to the question data, the reference data and the answer data.

2. A method for generating large model training data in the medical field according to claim 1, characterized in that: The step of inputting the first prompt text into a preset first language model so that the first language model outputs question data associated with the case data specifically includes: Generate a first scoring text and a first screening criterion according to the question data; wherein the first screening criterion is set based on the model fine-tuning data; Inputting the first scoring text into a preset second language model so that the second language model outputs first scoring information associated with the question data; The first scoring information is screened according to the first screening criterion, and the problem data corresponding to the first scoring information that does not meet the first screening criterion is deleted.

3. A method for generating large model training data in the medical field according to claim 2, characterized in that: The first scoring information includes: question fluency score, question relevance score, question harmfulness score, complexity score and personalization score; The first screening criterion includes: the question harmfulness score is 0, and the question fluency score, the question relevance score, the complexity score and the personalization degree score are not less than half of the total score.

4. A method for generating large model training data in the medical field according to claim 1, characterized in that: The step of generating a second prompt text according to the question data so that the first language model outputs reference data specifically includes: Generate a second scoring text and a second screening criterion according to the reference data; wherein the second screening criterion is set based on the model fine-tuning data; Inputting the second scoring text into a preset third language model so that the third language model outputs second scoring information associated with the reference data; Generate a second optimized text, filter the second scoring information according to the second filtering criteria, and input the reference data corresponding to the second scoring information that does not meet the second filtering criteria, the problem data corresponding to the reference data, and the second optimized text into the first language model, so that the first language model optimizes the reference data.

5. A method for generating large model training data in the medical field according to claim 4, characterized in that: The second scoring information includes: a reference fluency score, a reference relevance score, a reference harmfulness score, a reference accuracy score, a reference rationality score and a specificity score; The second screening criterion includes: the reference harmfulness score is 0, and the reference fluency score, the reference relevance score, the reference accuracy score, the reference rationality score and the specificity score are not less than half of the total score.

6. A method for generating large model training data in the medical field according to claim 1, characterized in that: Generating a third prompt text according to the reference data so that the first language model outputs answer data specifically includes: Generating a third scoring text and a third screening criterion according to the reference data; wherein the third screening criterion is set based on the model fine-tuning data; Inputting the third scoring text into a preset fourth language model so that the fourth language model outputs third scoring information associated with the answer data; Generate a third optimized text, filter the third scoring information according to the third filtering criteria, and input the answer data corresponding to the third scoring information that does not meet the third filtering criteria, the reference data and question data corresponding to the answer data, and the third optimized text into the first language model, so that the first language model optimizes the answer data.

7. A method for generating large model training data in the medical field according to claim 6, characterized in that: The third scoring information includes: answer fluency score, answer relevance score, answer harmfulness score, answer accuracy score and answer rationality score; The third screening criterion includes: the answer harmfulness score is 0, and the answer fluency score, the answer relevance score, the answer accuracy score and the answer rationality score are not less than half of the total score.

8. A device for generating large model training data in the medical field, characterized in that: include: A case reading module, used for acquiring case data and generating a first prompt text according to the case data; A question generation module, used for inputting the first prompt text into a preset first language model, so that the first language model outputs question data associated with the case data; A reference data generation module, used to generate a second prompt text according to the question data, so that the first language model outputs reference data; wherein the reference data is used to express the processing process of the first language model generating the question data; an answer generation module, configured to generate a third prompt text according to the reference data, so that the first language model outputs answer data; The fine-tuning data integration module is used to generate model fine-tuning data according to the question data, the reference data and the answer data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images