Training method of inquiry large model and inquiry method based on large model

Abstract

The invention provides a training method of an inquiry large model and an inquiry method based on the large model, and relates to the field of artificial intelligence, in particular to the technical field of large models, deep learning and AI medical treatment. Based on the sample chief complaint information, performing retrieval enhancement in medical references to generate RAG so as to generate an inquiry knowledge base; acquiring a virtual patient portrait corresponding to the sample chief complaint information; simulating a patient role according to the virtual patient portrait through a multi-modal large language model, and simulating a doctor role according to the inquiry knowledge base; guiding the doctor role and the patient role to carry out inquiry dialogue interaction according to the inquiry knowledge base through a multi-mode large language model, and obtaining inquiry dialogue data; according to the inquiry dialogue data, supervision fine tuning SFT is carried out on the base large model, and a target inquiry large model is obtained.According to the method, multi-round inquiry and cross-modal reasoning can be carried out on the basis of the target inquiry large model, the inquiry ability in a complex inquiry scene is remarkably improved, and the practicability and accuracy of an inquiry result are improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the fields of large models, deep learning, and AI medical technology, and particularly to a training method for a large diagnostic model and a diagnostic method based on the large model. Background Technology

[0002] In related technologies, rule-based and semantic matching methods identify diseases based on patient complaints by setting rules, keywords, or semantic relationships. However, these methods are poorly adaptable to unstructured inputs, lack robustness, and their rule design and parameter configuration are highly dependent on human experience. They can also perform disease diagnosis based on large-scale interactive reasoning models, but this method requires medical background support from knowledge graphs. The construction and updating of knowledge graphs are costly and difficult to dynamically adapt to new knowledge. When patients input vague descriptions of their symptoms, these methods tend to output general and non-targeted diagnostic suggestions, resulting in poor practicality and accuracy of the consultation results. Summary of the Invention

[0003] This disclosure provides a training method for a large-scale medical history model, a medical history method based on the large-scale model, a training device for the large-scale medical history model, a medical history device based on the large-scale model, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect of this disclosure, a training method for a large-scale medical consultation model is provided, comprising: performing a search-enhanced RAG (Research and Enhancement Aggregator) based on sample chief complaint information in medical references to generate a medical consultation knowledge base; obtaining a virtual patient profile corresponding to the sample chief complaint information; simulating a patient role based on the virtual patient profile and a doctor role based on the medical consultation knowledge base using a multimodal large-scale language model; guiding the doctor role and patient role to conduct a medical consultation dialogue interaction using the multimodal large-scale language model based on the medical consultation knowledge base, and obtaining medical consultation dialogue data; and performing supervised fine-tuning of the base large-scale model using SFT (Self-Functional Theory) based on the medical consultation dialogue data to obtain a target medical consultation large-scale model.

[0005] According to a second aspect of this disclosure, a large-scale model-based consultation method is provided, comprising: acquiring target chief complaint information of a target patient; performing RAG (Research and Analysis) on the target chief complaint information in a consultation knowledge base to obtain target consultation path information corresponding to the target chief complaint information; inputting the target consultation path information into a target consultation large-scale model; acquiring the patient's condition information of the target patient through the target consultation path information using the target consultation large-scale model; and determining the consultation result information of the target patient based on the patient's condition information under the reasoning logic of the thought chain; wherein the target consultation large-scale model is a model trained using the training method described in the first aspect.

[0006] According to a third aspect of this disclosure, a training device for a large-scale medical consultation model is provided, comprising: a first acquisition module, used to perform enhanced RAG generation in medical references based on sample chief complaint information to generate a medical consultation knowledge base; a second acquisition module, used to acquire a virtual patient profile corresponding to the sample chief complaint information; a simulation module, used to simulate a patient role based on the virtual patient profile and a doctor role based on the medical consultation knowledge base using a multimodal large language model; an interaction module, used to guide the doctor role and the patient role to conduct a medical consultation dialogue interaction based on the medical consultation knowledge base using the multimodal large language model and to acquire medical consultation dialogue data; and a fine-tuning module, used to perform supervised fine-tuning of the base large-scale model using SFT based on the medical consultation dialogue data to obtain a target medical consultation large-scale model.

[0007] According to a fourth aspect of this disclosure, a large-scale model-based consultation device is provided, comprising: an acquisition module for acquiring target chief complaint information of a target patient; a generation module for performing RAG (Research and Analysis) on the target chief complaint information in a consultation knowledge base to obtain target consultation path information corresponding to the target chief complaint information; and a consultation module for inputting the target consultation path information into a target consultation large-scale model, acquiring patient condition information of the target patient based on the target consultation path information through the target consultation large-scale model, and determining consultation result information of the target patient based on the patient condition information under the reasoning logic of the thought chain; wherein the target consultation large-scale model is a model trained using the training method described in the first aspect.

[0008] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the large-scale diagnostic model training method of the first aspect of this disclosure or the large-scale diagnostic model-based method of the second aspect of this disclosure.

[0009] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to execute the training method of the large-scale medical consultation model described in the first aspect of this disclosure or the large-scale medical consultation method described in the second aspect.

[0010] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the training method for a large-scale diagnostic model according to a first aspect of this disclosure or the diagnostic method based on a large-scale model according to a second aspect.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart illustrating a method for training a large-scale medical history model according to an embodiment of the present disclosure.

[0014] Figure 2 This is a flowchart illustrating a method for training a large-scale medical history model according to an embodiment of the present disclosure.

[0015] Figure 3 This is a flowchart illustrating a large-model-based diagnostic method according to an embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating a method for training a large-scale medical history model and a medical history method based on a large-scale model, according to an embodiment of this disclosure.

[0017] Figure 5 This is a schematic diagram of the structure of a training device for a large-scale medical history model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram of the structure of a large-model-based medical consultation device according to an embodiment of this disclosure;

[0019] Figure 7 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] The following is a brief description of the technical field involved in the solution disclosed herein:

[0022] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, and related technologies such as deep learning, big data processing, and knowledge graphs.

[0023] Large models refer to machine learning models with a large number of parameters and high complexity. They require a lot of computing resources and storage space for training and storage, and often require distributed computing and special hardware acceleration technologies. Large models have stronger generalization and expressive capabilities.

[0024] Deep learning (DL) is a new research direction in the field of machine learning (ML), bringing it closer to its original goal—artificial intelligence. Deep learning learns the inherent laws and hierarchical representations of sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies.

[0025] AI in healthcare (Artificial Intelligence in Healthcare) refers to the use of technologies such as machine learning, deep learning, and natural language processing to intelligently transform the diagnosis, treatment, management, and research and development of healthcare services. Essentially, it uses algorithms to drive data, improving the efficiency and accuracy of disease prediction, image recognition, and personalized treatment.

[0026] The following describes a method for training a large-scale medical history model and a medical history consultation method based on the large-scale model, according to embodiments of the present disclosure, with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating a training method for a large-scale medical history model according to an embodiment of the present disclosure. It should be noted that the execution entity of the training method for the large-scale medical history model in this embodiment is a training device for the large-scale medical history model, which can specifically be a hardware device or software within a hardware device. The hardware device can be, for example, a terminal device or a server.

[0028] like Figure 1 As shown, the training method for the large-scale medical history model proposed in this embodiment includes the following steps:

[0029] S101. Based on the sample's chief complaint information, perform retrieval and enhancement in medical references to generate a RAG, thereby creating a medical consultation knowledge base.

[0030] The chief complaint information in the sample can include chief complaint information for different disease types. Chief complaint information refers to the most important symptoms or signs described by the patient during the consultation and their duration.

[0031] Optionally, a pre-built chief complaint template can be obtained, and sample chief complaint information can be constructed according to the chief complaint template. The number of sample chief complaint information can be set according to the actual situation.

[0032] It should be noted that this disclosure does not impose any restrictions on the setting of medical references. Optionally, the latest editions of medical textbooks such as "Internal Medicine", "Surgery", "Obstetrics and Gynecology", "Pathology" and "Pediatrics" can be used as medical references.

[0033] In this embodiment of the disclosure, after obtaining the chief complaint information of the sample, a Retrieval-Augmented Generation (RAG) can be performed in medical references based on the chief complaint information to generate a medical knowledge base.

[0034] Optionally, RAG can be performed in medical references based on the sample chief complaint information to obtain the medical description information and default consultation process information corresponding to the sample chief complaint information. Based on the medical description information and consultation process information, consultation path information corresponding to the sample chief complaint information can be generated. Based on the sample chief complaint information and the corresponding consultation path information, a consultation knowledge base can be generated.

[0035] S102. Obtain the virtual patient profile corresponding to the chief complaint information of the sample.

[0036] Among them, the virtual patient profile is used to simulate the patient's basic medical information.

[0037] In this embodiment of the disclosure, a multimodal large language model is used to obtain image and text data pairs corresponding to the chief complaint information of the sample from a medical examination database. The medical examination database includes at least one of an image database and an examination report database. Based on the disease database and the image and text data pairs, the multimodal large language model generates a virtual patient profile corresponding to the chief complaint information of the sample according to a preset patient profile framework.

[0038] S103. Simulate the patient role based on the virtual patient profile using a multimodal large language model, and simulate the doctor role based on the consultation knowledge base.

[0039] It should be noted that a multimodal large language model is used to simulate the patient role based on the virtual patient profile, and a doctor role is simulated based on the consultation knowledge base. The doctor role and the patient role interact and converse through consultation to obtain consultation dialogue data.

[0040] S104. Using a multimodal large language model based on the consultation knowledge base, guide the doctor and patient roles to conduct consultation dialogue interaction and obtain consultation dialogue data.

[0041] In this embodiment of the disclosure, the consultation path information corresponding to the sample chief complaint information can be obtained from the consultation knowledge base through a multimodal large language model. The doctor asks questions according to the consultation path information, and the patient answers the doctor's questions based on a virtual patient profile to obtain consultation dialogue data.

[0042] S105. Based on the consultation dialogue data, perform supervised fine-tuning of the base model using SFT to obtain the target consultation model.

[0043] In this embodiment of the disclosure, after obtaining the consultation dialogue data, the base model is subjected to supervised fine-tuning (SFT) based on the consultation dialogue data. It is then determined whether the base model meets the fine-tuning termination condition. In response to the base model meeting the fine-tuning termination condition, the target consultation model is obtained.

[0044] It should be noted that this disclosure does not limit the type of the base model. Optionally, a large model that supports multimodal input and is suitable for processing consultation data that combines text and images can be selected as the base model.

[0045] According to the training method of the large-scale consultation model in this disclosure, a Retrieval Enhancement (RAG) is generated in medical references based on the sample chief complaint information to create a consultation knowledge base. A virtual patient profile corresponding to the sample chief complaint information is obtained. A multimodal large language model simulates the patient role based on the virtual patient profile and the doctor role based on the consultation knowledge base. The multimodal large language model guides the doctor and patient roles to conduct consultation dialogues based on the consultation knowledge base, obtaining consultation dialogue data. Based on the consultation dialogue data, a supervised fine-tuning SFT is performed on the base large model to obtain the target consultation large model. Thus, this disclosure obtains the target consultation large model by supervising the SFT of the base large model based on consultation dialogue data. The target consultation large model can perform multi-round consultations and cross-modal reasoning, significantly improving the consultation capability in complex consultation scenarios and enhancing the practicality and accuracy of consultation results.

[0046] Figure 2 This is a flowchart illustrating a method for training a large-scale medical history model according to an embodiment of the present disclosure.

[0047] like Figure 2 As shown, the training method for the large-scale medical history model proposed in this embodiment includes the following steps:

[0048] The above embodiment's S101 "based on the sample's chief complaint information, perform retrieval enhancement in medical references to generate a RAG, in order to generate a medical consultation knowledge base" may specifically include S201 and S205.

[0049] S201. Obtain the diagnostic assistance information corresponding to the chief complaint information of the sample.

[0050] The consultation assistance information can be customized. It can be customized according to different sample chief complaints. Consultation assistance information can be understood as consultation template information, which contains a variety of consultation questions.

[0051] S202. Based on the sample's chief complaint information, perform RAG in medical references to obtain the medical description information corresponding to the sample's chief complaint information.

[0052] Optionally, the sample's chief complaint information can be converted into a vector, and target medical reference fragments can be selected from medical references through similarity calculation. Based on the target medical reference fragments, the medical description information corresponding to the sample's chief complaint information can be determined.

[0053] S203. Based on the consultation assistance information, determine the consultation reference information and consultation process information corresponding to the chief complaint information of the sample.

[0054] In this embodiment of the disclosure, after obtaining the consultation assistance information, the consultation reference information and consultation process information corresponding to the chief complaint information of the sample can be determined based on the consultation assistance information.

[0055] S204. Based on the medical description information, consultation reference information, and consultation process information, generate consultation path information corresponding to the sample chief complaint information.

[0056] In this embodiment of the disclosure, after obtaining medical description information, consultation reference information and consultation process information, the medical description information, consultation reference information and consultation process information can be integrated to generate consultation path information corresponding to the sample chief complaint information.

[0057] S205. Generate a consultation knowledge base based on the sample chief complaint information and the corresponding consultation path information.

[0058] In this embodiment of the disclosure, after obtaining the consultation path information, the sample chief complaint information and the corresponding consultation path information can be associated and stored to generate a consultation knowledge base.

[0059] The above embodiment's S102 "obtaining a virtual patient profile corresponding to the sample's chief complaint information" may specifically include S206 and S204.

[0060] S206. Obtain image-text data pairs corresponding to the chief complaint information of the sample from the medical examination database through a multimodal large language model, wherein the medical examination database includes at least one of the imaging database and the examination report database.

[0061] In this embodiment of the disclosure, a multimodal large language model can be used to obtain historical examination data associated with the sample chief complaint information from a medical examination database. The historical examination data includes at least one of the historical images and historical examination reports associated with the sample chief complaint information. The historical examination data associated with the sample chief complaint information is understood and parsed, and based on the understanding and parsing results, descriptive information corresponding to the historical examination data is output. Based on the historical examination data and the corresponding descriptive information, a graphic data pair corresponding to the sample chief complaint information is generated.

[0062] Optionally, historical images associated with the subject complaint information of the sample can be obtained from a historical image database. The historical images can be understood and descriptive information can be obtained through a multimodal large language model.

[0063] For example, by using a multimodal large model to understand skin images, we can obtain the human body parts shown in the skin images and the possible skin problems.

[0064] Optionally, historical examination reports associated with the sample chief complaint information can be obtained from a historical examination report database. These historical examination reports can then be parsed using a multimodal large language model to obtain descriptive information.

[0065] For example, by analyzing a routine blood test report using a multimodal large model, the test indicators and abnormalities displayed on the report can be obtained.

[0066] Optionally, after obtaining the descriptive information corresponding to the historical inspection data, in order to improve the quality of the descriptive information, the descriptive information can be cleaned to obtain the target descriptive information. The historical inspection data and the corresponding descriptive information can then be associated to generate a pair of graphic and textual data corresponding to the sample chief complaint information.

[0067] S207. Using a multimodal large language model, based on the disease database and image / text data pairs, and following a pre-defined patient profile framework, generate a virtual patient profile corresponding to the sample's chief complaint information.

[0068] In this embodiment of the disclosure, patient attribute information is determined based on the patient profile framework, symptom information to be collected corresponding to the sample chief complaint information is obtained from the disease database, examination information to be collected corresponding to the sample chief complaint information is extracted from the image and text data pair based on the patient profile framework, and a virtual patient profile corresponding to the sample chief complaint information is generated based on the symptom information to be collected, the examination information to be collected, and the patient attribute information.

[0069] It should be noted that a patient profile framework can be predefined, which includes, but is not limited to, information such as the patient's age, gender, basic symptoms, present medical history, allergy history, and family history.

[0070] Optionally, a virtual patient profile can be generated randomly using a large model based on the symptom information to be collected, the examination information to be collected, and the patient attribute information.

[0071] S208. Simulate the patient role based on the virtual patient profile using a multimodal large language model, and simulate the doctor role based on the consultation knowledge base.

[0072] The above embodiment's S104 "guides the doctor and patient roles to conduct consultation dialogue interaction based on the consultation knowledge base through a multimodal large language model, and obtains consultation dialogue data" may specifically include S209 and S2010.

[0073] S209. Obtain the consultation path information corresponding to the chief complaint information of the sample from the consultation knowledge base through a multimodal large language model.

[0074] S2010. Doctors ask questions according to the consultation path information, and patients answer the doctor's questions based on a virtual patient profile to obtain consultation dialogue data.

[0075] Optionally, the doctor role conducts a step-by-step consultation and asks questions according to the consultation path information, and the patient role answers the doctor role's consultation and questions based on the virtual patient profile. The consultation path information can be dynamically adjusted based on the patient role's answers until the consultation ends and the dialogue stops, thus obtaining consultation dialogue data.

[0076] S105 in the above embodiment, "Based on the consultation dialogue data, perform supervised fine-tuning of the base model using SFT to obtain the target consultation model," may specifically include S2011 and S2013.

[0077] S2011. Conduct a quality assessment on the consultation dialogue data, and filter the consultation dialogue data based on the quality assessment results to obtain the target consultation dialogue data.

[0078] It should be noted that after obtaining the consultation dialogue data, in order to improve the quality of the consultation dialogue data and ensure the accuracy of the subsequent target consultation model, heuristic rules and the large model can be used simultaneously to evaluate the quality of the consultation dialogue data, and the consultation dialogue data can be filtered based on the quality evaluation results to obtain the target consultation dialogue data.

[0079] Optionally, quality assessment indicators for the consultation dialogue data can be pre-selected. Based on the quality assessment indicators, heuristic rules and large models are used to assess the quality of the consultation dialogue data, obtain the quality assessment score of the consultation dialogue data, remove consultation dialogue data with a quality assessment score less than the quality assessment score threshold, and retain consultation dialogue data with a quality assessment score greater than or equal to the quality assessment score threshold to obtain the target consultation dialogue data.

[0080] S2012. Extract information from the target consultation dialogue data to obtain consultation training corpus.

[0081] In this embodiment of the disclosure, after obtaining the target consultation dialogue data, key information can be extracted from the target consultation dialogue data through a large model, and consultation training corpus can be obtained based on the information extraction results under the reasoning logic of the thought chain.

[0082] S2013. Based on the consultation training corpus, perform SFT on the base large model to obtain the target consultation large model.

[0083] In this embodiment of the disclosure, after obtaining the consultation training data, the base model is subjected to SFT based on the consultation training data, and the model parameters of the base model are continuously adjusted until the base model meets the fine-tuning termination condition, thereby obtaining the target consultation model.

[0084] According to the training method of the large-scale consultation model of this disclosure, the following steps are taken: First, consultation auxiliary information corresponding to the sample chief complaint information is obtained. Then, RAG (Research and Analysis) is performed on medical references based on the sample chief complaint information to obtain medical description information corresponding to the sample chief complaint information. Based on the consultation auxiliary information, consultation reference information and consultation process information corresponding to the sample chief complaint information are determined. Based on the medical description information, consultation reference information, and consultation process information, consultation path information corresponding to the sample chief complaint information is generated. Based on the sample chief complaint information and the corresponding consultation path information, a consultation knowledge base is generated. Then, a multimodal large-scale language model is used to obtain image-text data pairs corresponding to the sample chief complaint information from a medical examination database, wherein the medical examination database includes at least one of an image database and an examination report database. Based on the disease database and image-text data pairs, and according to a preset patient profile framework, a virtual patient profile corresponding to the sample chief complaint information is generated using the multimodal large-scale language model. The multimodal large-scale language model simulates the patient role based on the virtual patient profile and the doctor role based on the consultation knowledge base. Finally, the multimodal large-scale language model obtains the corresponding information from the consultation knowledge base. The process involves obtaining consultation path information. Doctors, acting as consultants, ask questions according to this path, while patients, acting as consultants, respond to these questions based on a virtual patient profile. This generates consultation dialogue data. The quality of this data is assessed, and based on the assessment results, the data is filtered to obtain target consultation dialogue data. Information is extracted from this target data to obtain training corpus. Based on this training corpus, a SFT is performed on the base model to obtain the target consultation model. This disclosure introduces multi-turn consultation dialogue data and multimodal information input from real-world scenarios, significantly improving the target consultation model's ability to understand and adapt to unstructured expressions. It reduces reliance on rule design and manual configuration, giving the target consultation model stronger generalization and transferability. With supervised fine-tuning, the target consultation model can be endowed with multi-turn and cross-modal consultation capabilities, enabling it to understand patients' unstructured expressions and conduct consultations even with incomplete or ambiguous information. This significantly improves consultation capabilities in complex scenarios and enhances the practicality and accuracy of consultation results.

[0085] Figure 3 This is a flowchart illustrating a large-model-based consultation method according to an embodiment of this disclosure. It should be noted that the execution entity of this large-model-based consultation method is a large-model-based consultation device, which can specifically be a hardware device or software within a hardware device. The hardware device can be, for example, a terminal device or a server.

[0086] like Figure 3 As shown, the large-model-based diagnostic method proposed in this embodiment includes the following steps:

[0087] S301. Obtain the target patient's chief complaint information.

[0088] It should be noted that the target patient will provide target chief complaint information based on their own condition, such as: "I have recently had a sore throat" or "What department should I go to for numbness in my arm"?

[0089] It should be noted that this disclosure does not limit the specific method by which the target patient provides the target chief complaint information. Optionally, the target chief complaint information can be input by text, voice, or other means.

[0090] S302. Perform RAG in the consultation knowledge base based on the target chief complaint information to obtain the target consultation path information corresponding to the target chief complaint information.

[0091] Optionally, after obtaining the target chief complaint information, the latest target consultation path information corresponding to the target chief complaint information can be generated by retrieving from the consultation knowledge base based on RAG technology.

[0092] S303. Input the target consultation path information into the target consultation model. The target consultation model obtains the patient's condition information based on the target consultation path information. Under the reasoning logic of the thought chain, the consultation result information of the target patient is determined based on the patient's condition information.

[0093] Among them, the target consultation model is a model obtained using the training method in the first aspect.

[0094] Optionally, after obtaining the target consultation path information, the target consultation model interacts with the target patient for consultation. The consultation ends after the patient's condition information is collected. The target consultation model then organizes the patient's condition information and, based on the reasoning logic of the thought chain, determines the consultation result information for the target patient. The reasoning logic of the thought chain can help the target consultation model improve the accuracy of the consultation result information, while also demonstrating the decision-making process and improving interpretability.

[0095] In this embodiment of the disclosure, during the consultation interaction through the target consultation model, in response to the target chief complaint information involving the target type of illness, the target consultation model generates an inquiry dialogue for examination data, receives the data input information of the target patient, and obtains the examination data of the target patient based on the data input information, wherein the examination data is used to supplement the target patient's condition information.

[0096] Among them, the target disease can be preset. For example, the target disease is a skin-related disease, or the target disease can be a disease that can be combined with examination reports.

[0097] For example, when the target patient's chief complaint involves a target-related illness, the target consultation model generates a dialogue to inquire about examination data. It will proactively ask the target patient if there are any relevant examination data that can be provided, receive the data input information from the target patient, and obtain the target patient's examination data based on the data input information. The target consultation model will then conduct further interactive consultation based on the examination data. If no data input information is received from the target patient, the consultation will continue.

[0098] According to the large-scale model-based consultation method of this disclosure, the target patient's chief complaint information is obtained. Based on this chief complaint information, a Relational Analysis (RAG) is performed in the consultation knowledge base to obtain the target consultation path information corresponding to the chief complaint information. This target consultation path information is then input into a target consultation large-scale model. The target consultation large-scale model obtains the patient's condition information based on the target consultation path information. Under the reasoning logic of the thought chain, the consultation result information of the target patient is determined based on the patient's condition information. Therefore, this disclosure, by performing RAG on the consultation knowledge base based on the chief complaint information, can obtain the latest target consultation path information corresponding to the chief complaint information. This effectively solves the problems of high cost and slow response of knowledge graph updates in traditional methods. By obtaining the patient's condition information based on the target consultation path information through the target consultation large-scale model, and under the reasoning logic of the thought chain, the reasoning process of the target consultation large-scale model becomes clearer and more interpretable, enhancing the user's trust in the consultation result information, improving the efficiency and accuracy of determining the consultation result, and improving the user experience.

[0099] The following explains the specific process of training the large-scale medical consultation model proposed in this publication, as well as the specific process of the medical consultation method based on the large-scale model.

[0100] For example, such as Figure 4As shown, the process involves obtaining auxiliary information for patient consultation corresponding to the sample's chief complaint. Based on this information, a Relationship Analysis (RAG) is performed in medical references to obtain corresponding medical description information. Based on the auxiliary information, reference information and consultation process information are determined. Using a large-scale model, consultation path information corresponding to the sample's chief complaint is generated based on the medical description information, reference information, and consultation process information. A consultation knowledge base is then generated based on the sample's chief complaint and the corresponding consultation path information. A multimodal large-scale language model retrieves image-text pairs corresponding to the sample's chief complaint from a medical examination database (image database and examination report database). Based on the disease database and image-text pairs, and following a pre-defined patient profile framework, a virtual patient profile corresponding to the sample's chief complaint is generated using the multimodal large-scale language model. The model then simulates the patient role based on the virtual patient profile and the doctor role based on the consultation knowledge base. Finally, the multimodal large-scale language model guides the doctor and patient roles based on the consultation knowledge base. The system involves a patient engaging in a consultation dialogue to obtain consultation dialogue data. Information is extracted from this data to obtain consultation training corpus. Based on this training corpus, the base model undergoes SFT to obtain the target consultation model. After obtaining the target consultation model, the system acquires the target patient's chief complaint information. Based on this chief complaint information, the system performs RAG on the consultation knowledge base to obtain the target consultation path information corresponding to the chief complaint information. This target consultation path information is then input into the target consultation model. The target consultation model then retrieves the patient's condition information based on the target consultation path information. During the consultation interaction through the target consultation model, in response to the target chief complaint information involving the target type of condition, the system generates a dialogue to inquire about examination data. It receives the target patient's data input information and, based on this data input information, retrieves the target patient's examination data to supplement the target patient's condition information. Finally, under the reasoning logic of the thought chain, the system determines the consultation result information of the target patient based on the patient's condition information.

[0101] In summary, this disclosure significantly improves the accuracy, knowledge update efficiency, and unstructured input processing capabilities of intelligent consultation in complex scenarios by integrating multimodal large-scale models, thought chain technology, and RAG technology. Compared with traditional methods that rely on rules or static knowledge graphs, the consultation method based on large-scale models has stronger robustness and adaptability, and can handle patients' fuzzy, fragmented, and multimodal descriptions of their conditions, ensuring coverage of real-world consultation scenarios. At the same time, RAG technology enables flexible access to the latest medical knowledge, and the thought chain mechanism improves the clarity and interpretability of the reasoning process of the target consultation large-scale model, which helps to enhance users' trust in the consultation results, improve consultation efficiency, and enhance user experience and satisfaction.

[0102] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0103] Corresponding to the training methods of the large-scale medical history model provided in the above embodiments, an embodiment of this disclosure also provides a training device for the large-scale medical history model. Since the training device for the large-scale medical history model provided in this embodiment corresponds to the training methods of the large-scale medical history model provided in the above embodiments, the implementation methods of the training methods for the large-scale medical history model are also applicable to the training device for the large-scale medical history model provided in this embodiment, and will not be described in detail in this embodiment.

[0104] Figure 5 This is a schematic diagram of the structure of a training device for a large-scale medical history model according to an embodiment of the present disclosure.

[0105] like Figure 5 As shown, the training device 500 for the large-scale medical consultation model includes: a first acquisition module 510, a second acquisition module 520, a simulation module 530, an interaction module 540, and a fine-tuning module 540.

[0106] The first acquisition module 510 is used to perform enhanced RAG generation in medical references based on the sample chief complaint information to generate a medical consultation knowledge base;

[0107] The second acquisition module 520 is used to acquire a virtual patient profile corresponding to the sample chief complaint information;

[0108] The simulation module 530 is used to simulate the role of a patient based on a virtual patient profile using a multimodal large language model, and to simulate the role of a doctor based on the consultation knowledge base.

[0109] Interaction module 540 is used to guide the doctor and patient roles to conduct consultation dialogue interaction based on the consultation knowledge base through the multimodal large language model, and to obtain consultation dialogue data;

[0110] The fine-tuning module 550 is used to perform supervised fine-tuning of the SFT on the base model based on the consultation dialogue data to obtain the target consultation model.

[0111] The first acquisition module 510 is further configured to: acquire consultation assistance information corresponding to the sample chief complaint information; perform RAG analysis on the medical references based on the sample chief complaint information to acquire medical description information corresponding to the sample chief complaint information; determine consultation reference information and consultation process information corresponding to the sample chief complaint information based on the consultation assistance information; generate consultation path information corresponding to the sample chief complaint information based on the medical description information, the consultation reference information, and the consultation process information; and generate the consultation knowledge base based on the sample chief complaint information and the corresponding consultation path information.

[0112] The second acquisition module 520 is further configured to: acquire the image-text data pair corresponding to the sample chief complaint information from the medical examination database through the multimodal large language model, wherein the medical examination database includes at least one of an image database and an examination report database; and generate a virtual patient profile corresponding to the sample chief complaint information according to the disease database and the image-text data pair through the multimodal large language model and a preset patient profile framework.

[0113] The second acquisition module 520 is further configured to: acquire historical examination data associated with the sample chief complaint information from the medical examination database through the multimodal large language model, wherein the historical examination data includes at least one of historical images and historical examination reports associated with the sample chief complaint information; understand and parse the historical examination data associated with the sample chief complaint information, and output descriptive information corresponding to the historical examination data based on the understanding and parsing results; and generate image-text data pairs corresponding to the sample chief complaint information based on the historical examination data and the corresponding descriptive information.

[0114] The second acquisition module 520 is further configured to: determine patient attribute information based on the patient profile framework; acquire symptom information to be collected corresponding to the sample chief complaint information from the disease database; extract examination information to be collected corresponding to the sample chief complaint information from the image and text data pair based on the patient profile framework; and generate a virtual patient profile corresponding to the sample chief complaint information based on the symptom information to be collected, the examination information to be collected, and the patient attribute information.

[0115] The interaction module 540 is further configured to: obtain consultation path information corresponding to the sample chief complaint information from the consultation knowledge base through the multimodal large language model; ask consultation questions through the doctor role according to the consultation path information, and answer the consultation questions of the doctor role based on the virtual patient profile through the patient role, so as to obtain the consultation dialogue data.

[0116] The fine-tuning module 550 is further configured to: perform quality assessment on the consultation dialogue data, and filter the consultation dialogue data based on the quality assessment results to obtain target consultation dialogue data; extract information from the target consultation dialogue data to obtain consultation training corpus; and perform SFT on the base large model based on the consultation training corpus to obtain the target consultation large model.

[0117] According to the training device of the large-scale consultation model of this disclosure, the RAG is generated by searching and enhancing medical references based on the chief complaint information of the samples to generate a consultation knowledge base. The virtual patient profile corresponding to the chief complaint information is obtained. The patient role is simulated according to the virtual patient profile by a multimodal large language model, and the doctor role is simulated according to the consultation knowledge base. The doctor role and the patient role are guided to conduct consultation dialogue interaction by the multimodal large language model according to the consultation knowledge base to obtain consultation dialogue data. Based on the consultation dialogue data, the base large model is subjected to supervised fine-tuning of SFT to obtain the target consultation large model. Thus, this disclosure obtains the target consultation large model by supervising fine-tuning of SFT of the base large model according to consultation dialogue data. The target consultation large model can perform multi-round consultation and cross-modal reasoning, which significantly improves the consultation ability in complex consultation scenarios and improves the practicality and accuracy of consultation results.

[0118] Corresponding to the large-model-based consultation methods provided in the above embodiments, one embodiment of this disclosure also provides a large-model-based consultation device. Since the large-model-based consultation device provided in this embodiment corresponds to the large-model-based consultation methods provided in the above embodiments, the implementation methods of the large-model-based consultation methods are also applicable to the large-model-based consultation device provided in this embodiment, and will not be described in detail in this embodiment.

[0119] Figure 6 This is a schematic diagram of a large-model-based diagnostic device according to an embodiment of the present disclosure.

[0120] like Figure 6 As shown, the large-model-based consultation device 600 includes: an acquisition module 610, a generation module 620, and a consultation module 630. Wherein:

[0121] The acquisition module 610 is used to acquire the target chief complaint information of the target patient;

[0122] The generation module 620 is used to perform RAG in the consultation knowledge base based on the target chief complaint information to obtain the target consultation path information corresponding to the target chief complaint information;

[0123] The consultation module 630 is used to input the target consultation path information into the target consultation model, obtain the patient condition information of the target patient through the target consultation path information, and determine the consultation result information of the target patient based on the patient condition information under the reasoning logic of the thinking chain.

[0124] Among them, the target consultation model is a model obtained using the training method in the first aspect.

[0125] The device 600 is further configured to: during the consultation interaction through the target consultation model, in response to the target chief complaint information involving a target type of illness, generate an inquiry dialogue for examination data through the target consultation model; receive the data input information of the target patient, and obtain the examination data of the target patient based on the data input information, wherein the examination data is used to supplement the medical condition information of the target patient.

[0126] According to the large-model-based consultation device of this disclosure, the target patient's chief complaint information is obtained, and RAG is performed in the consultation knowledge base based on the chief complaint information to obtain the target consultation path information corresponding to the chief complaint information. The target consultation path information is input into the target consultation large model, and the target consultation large model obtains the patient's condition information based on the target consultation path information. Under the reasoning logic of the thought chain, the consultation result information of the target patient is determined based on the patient's condition information. Thus, this disclosure obtains the patient's condition information based on the target consultation path information through the target consultation large model, and under the reasoning logic of the thought chain, the reasoning process of the target consultation large model can be made clearer and more interpretable, enhancing the user's trust in the consultation result information, improving the efficiency and accuracy of determining the consultation result, and improving the user experience.

[0127] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0128] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0130] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods for training large diagnostic models or methods based on large models for diagnosis. For example, in some embodiments, the training of large diagnostic models or methods based on large models for diagnosis can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the training of large diagnostic models or methods based on large models for diagnosis described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform a training method for a large diagnostic model or a diagnostic method based on a large model.

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0138] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the training method for a large-scale medical consultation model or the medical consultation method based on the large-scale model as described above.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a large-scale medical history taking model, wherein, The method includes: Based on the chief complaint information of the samples, a search enhancement RAG is performed in medical references to generate a medical consultation knowledge base; Obtain the virtual patient profile corresponding to the chief complaint information of the sample; The patient role is simulated based on a virtual patient profile using a multimodal large language model, and the doctor role is simulated based on the aforementioned consultation knowledge base; The multimodal large language model guides the doctor and patient roles to conduct consultation dialogues based on the consultation knowledge base, thereby obtaining consultation dialogue data. Based on the consultation dialogue data, the base model is subjected to supervised fine-tuning of SFT to obtain the target consultation model.

2. The method according to claim 1, wherein, The method of enhancing and generating a RAG based on the sample's chief complaint information in medical references to create a medical history knowledge base includes: Obtain the consultation assistance information corresponding to the chief complaint information of the sample; Based on the sample's chief complaint information, RAG was performed in the medical references to obtain the medical description information corresponding to the sample's chief complaint information; Based on the consultation assistance information, determine the consultation reference information and consultation process information corresponding to the chief complaint information of the sample; Based on the medical description information, the consultation reference information, and the consultation process information, the consultation path information corresponding to the sample chief complaint information is generated; The consultation knowledge base is generated based on the chief complaint information of the sample and the corresponding consultation path information.

3. The method according to claim 2, wherein, The process of obtaining the virtual patient profile corresponding to the chief complaint information of the sample includes: The multimodal large language model is used to obtain the image and text data pairs corresponding to the chief complaint information of the sample from the medical examination database, wherein the medical examination database includes at least one of the imaging database and the examination report database. Based on the disease database and the image-text data pair, the multimodal large language model generates a virtual patient profile corresponding to the sample chief complaint information according to a preset patient profile framework.

4. The method according to claim 3, wherein, The step of obtaining the image-text data pair corresponding to the chief complaint information of the sample from the medical examination database through the multimodal large language model includes: The multimodal large language model is used to obtain historical examination data associated with the sample's chief complaint information from the medical examination database. The historical examination data includes at least one of the historical images and historical examination reports associated with the sample's chief complaint information. The historical examination data associated with the chief complaint information of the sample are understood and parsed, and based on the understanding and parsing results, the descriptive information corresponding to the historical examination data is output. Based on the historical examination data and corresponding descriptive information, generate image and text data pairs corresponding to the sample chief complaint information.

5. The method according to claim 4, wherein, The step of generating a virtual patient profile corresponding to the sample chief complaint information using the multimodal large language model based on the disease database and the image-text data pair, according to a preset patient profile framework, includes: Based on the patient profile framework, determine the patient attribute information; Obtain the symptom information to be collected corresponding to the chief complaint information of the sample from the disease database; Based on the patient profile framework, extract the examination information to be collected corresponding to the chief complaint information of the sample from the image and text data pair; Based on the symptom information to be collected, the examination information to be collected, and the patient attribute information, a virtual patient profile corresponding to the sample chief complaint information is generated.

6. The method according to any one of claims 1-5, wherein, The process of guiding the doctor and patient roles to engage in consultation dialogue interaction through the multimodal large language model based on the consultation knowledge base, and obtaining consultation dialogue data, includes: The multimodal large language model is used to obtain the consultation path information corresponding to the chief complaint information of the sample from the consultation knowledge base; The doctor asks questions according to the consultation path information, and the patient answers the doctor's questions based on the virtual patient profile to obtain the consultation dialogue data.

7. The method according to claim 6, wherein, The step of performing supervised fine-tuning of the base model using the consultation dialogue data to obtain the target consultation model includes: The quality of the consultation dialogue data is assessed, and the consultation dialogue data is filtered based on the quality assessment results to obtain the target consultation dialogue data; Information is extracted from the target consultation dialogue data to obtain consultation training corpus; Based on the aforementioned consultation training corpus, SFT is performed on the base large model to obtain the target consultation large model.

8. A diagnostic method based on a large model, wherein, The method includes: Obtain the target patient's chief complaint information; Based on the target chief complaint information, a RAG is performed in the consultation knowledge base to obtain the target consultation path information corresponding to the target chief complaint information; The target consultation path information is input into the target consultation model. The target consultation model obtains the patient's condition information based on the target consultation path information. Under the reasoning logic of the thought chain, the consultation result information of the target patient is determined based on the patient's condition information. The target diagnostic model is a model obtained by the training method described in any one of claims 1-7.

9. The method according to claim 8, wherein, The method further includes: During the consultation interaction through the target consultation model, in response to the target chief complaint information involving the target type of illness, the target consultation model generates a dialogue to inquire about examination data. The system receives data input information from the target patient and, based on the data input information, obtains the target patient's examination data, wherein the examination data is used to supplement the target patient's medical condition information.

10. A training device for a large-scale medical history taking model, wherein, The device includes: The first acquisition module is used to perform search enhancement and RAG generation in medical references based on the sample chief complaint information in order to generate a medical consultation knowledge base; The second acquisition module is used to acquire a virtual patient profile corresponding to the chief complaint information of the sample. The simulation module is used to simulate the role of a patient based on a virtual patient profile using a multimodal large language model, and to simulate the role of a doctor based on the consultation knowledge base. The interaction module is used to guide the doctor and patient roles to conduct consultation dialogues based on the consultation knowledge base using the multimodal large language model, and to obtain consultation dialogue data. The fine-tuning module is used to perform supervised fine-tuning of the SFT on the base model based on the consultation dialogue data to obtain the target consultation model.

11. A diagnostic device based on a large model, wherein, The device includes: The acquisition module is used to acquire the target patient's chief complaint information; The generation module is used to perform RAG in the consultation knowledge base based on the target chief complaint information to obtain the target consultation path information corresponding to the target chief complaint information; The consultation module is used to input the target consultation path information into the target consultation model, obtain the patient's condition information based on the target consultation path information through the target consultation model, and determine the consultation result information of the target patient based on the patient's condition information under the reasoning logic of the thought chain. The target diagnostic model is a model obtained by the training method described in any one of claims 1-7.

12. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7 or 8-9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7 or 8-9.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7 or 8-9.

Images