Intelligent medical pre-inquiry method and system based on multi-modal large model, and medium

Abstract

The invention discloses an intelligent medical pre-inquiry method and system based on a multi-modal large model and a medium, and the method comprises the steps: obtaining user input data to recognize a target intention, and when a knowledge question and answer intention is recognized, obtaining a knowledge question and answer module; based on the medical knowledge graph, combining with knowledge keywords in the user input data and utilizing a multi-modal large model to obtain medical answers; when the pre-inquiry intention is recognized, inquiry keywords in the user input data are extracted for slot filling to obtain slot information, and a slot filling strategy is dynamically adjusted based on the slot information and user feedback data; and performing visual display on the medical answers and the health management suggestions based on the user interaction interface, and obtaining user interaction data for human-computer interaction. According to the invention, a natural language processing and multi-modal fusion mode is adopted, so that the information input process of a patient is more convenient; and meanwhile, by introducing dynamic medical history information acquisition and interaction, real-time interaction with the patient is realized, and the user experience of pre-inquiry is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to an intelligent medical pre-diagnosis method, system and medium based on a multimodal large model. Background Art

[0002] Pre-consultation systems are becoming increasingly important in modern medicine. With the development of the Internet and information technology, many hospitals and medical institutions have begun to optimize the patient medical process through pre-consultation systems. At present, pre-consultation systems mainly include two parts: registration and medical history collection. Existing pre-consultation systems usually rely on traditional online registration platforms, and patients need to fill in their personal information, choose doctors, and make appointments on the platform.

[0003] Through the above process, medical institutions can obtain some basic information of patients before they arrive at the hospital, thereby improving the hospital's work efficiency and the rationality of resource allocation to a certain extent. The existing pre-consultation system has many deficiencies in information input and interaction logic, as follows:

[0004] (1) Limitations of information input methods. Existing pre-consultation systems mainly rely on patients to input their medical information through text input or by checking questionnaires. Patients are required to manually input or select information for initial or follow-up consultations. This method can easily lead to a decline in patient experience when faced with a large amount of input content. This is mainly manifested in low input efficiency: patients need to fill in a variety of information item by item, such as symptom descriptions, past medical history, etc., especially when filling in complex medical conditions or a large amount of medical record information. The efficiency is extremely low, affecting the overall speed of information collection before the visit; and it is prone to errors: the manual information input process is prone to spelling errors or omissions of important details, especially for complex medical terms and descriptions of past treatment processes, which are more prone to errors and affect the doctor's judgment.

[0005] (2) Lack of dynamic interaction capabilities. Existing systems usually generate fixed questions based on the patient's basic information (such as gender, age, and department visited). They lack the ability to dynamically adjust based on the patient's real-time response and previous visit information. Some follow-up units mainly simply process and inquire about the medical record pictures or text uploaded by the patient, and cannot flexibly adjust based on the patient's latest input. This leads to insufficient pertinence in the pre-consultation process: the system cannot adjust questions based on the patient's real-time answers, lacks personalized questioning and interaction mechanisms, and results in a lot of inaccurate collection of consultation information, affecting the doctor's accurate judgment of the patient's condition; and it is difficult to effectively respond to complex changes in the patient's condition: when the patient's condition is complex and the patient has visited the doctor multiple times, it is difficult for the system to flexibly adjust the pre-consultation strategy based on different disease development histories and changes, and lacks sufficient response capabilities. Summary of the invention

[0006] The purpose of the present invention is to provide an intelligent medical pre-consultation method, system and medium based on a multimodal large model, which adopts natural language processing and multimodal fusion, supports multiple input methods such as voice, picture, text, etc., improves the flexibility of information input, enables patients to provide health information in the most natural way, and reduces the difficulty and error rate of information entry.

[0007] The first aspect of the present invention provides an intelligent medical pre-diagnosis method based on a multimodal large model, comprising the following steps:

[0008] Obtain user input data to identify target intent, wherein the target intent includes knowledge question answering intent and pre-diagnosis intention, wherein:

[0009] When the knowledge question-answering intention is identified, a medical answer is obtained using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data;

[0010] When the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data;

[0011] The medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction.

[0012] In this solution, obtaining user input data to identify target intent specifically includes:

[0013] Obtain user input data of the information processing interface;

[0014] The user input data is analyzed and processed based on a preset model to obtain different target intentions.

[0015] In this solution, when the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data, specifically including:

[0016] When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword;

[0017] Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results;

[0018] The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

[0019] In this solution, when the pre-inquiry intention is identified, the inquiry keywords in the user input data are extracted to fill the slots to obtain slot information, which specifically includes:

[0020] When the pre-consultation intention is identified, extracting the consultation type field in the user input data to obtain the consultation keyword;

[0021] Perform named entity recognition and relationship extraction on the medical inquiry keywords, so as to match the slots corresponding to the medical inquiry keywords of different dimensions;

[0022] The matching medical consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots at least include a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0023] In this solution, the strategy of dynamically adjusting slot filling based on the slot information and user feedback data specifically includes:

[0024] Acquiring user feedback data, wherein the user feedback data at least includes voice data, answer speed, and emotional feedback data;

[0025] Obtaining consultation strategies for different slots based on the user feedback data combined with corresponding slot information;

[0026] Based on the preset retrieval technology combined with the consultation strategy, target data is obtained from the preset database to obtain health management suggestions, wherein the preset database includes medical literature and clinical guidelines, and the target data includes various health management measures that match the current consultation strategy.

[0027] In this solution, the medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction, specifically including:

[0028] Obtaining the medical answer and the health management suggestion, performing data conversion, and visually displaying them using the user interaction interface;

[0029] During the display process, user interaction data on the user interaction interface is obtained for human-computer interaction, wherein the user interaction data is transmitted based on a preset encryption authority.

[0030] The second aspect of the present invention further provides an intelligent medical pre-diagnosis system based on a multimodal large model, comprising a memory and a processor, wherein the memory comprises an intelligent medical pre-diagnosis method program based on a multimodal large model, and when the intelligent medical pre-diagnosis method program based on a multimodal large model is executed by the processor, the following steps are implemented:

[0031] Obtain user input data to identify target intent, wherein the target intent includes knowledge question answering intent and pre-diagnosis intention, wherein:

[0032] When the knowledge question-answering intention is identified, a medical answer is obtained using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data;

[0033] When the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data;

[0034] The medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction.

[0035] In this solution, obtaining user input data to identify target intent specifically includes:

[0036] Obtain user input data of the information processing interface;

[0037] The user input data is analyzed and processed based on a preset model to obtain different target intentions.

[0038] In this solution, when the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data, specifically including:

[0039] When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword;

[0040] Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results;

[0041] The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

[0042] In this solution, when the pre-inquiry intention is identified, the inquiry keywords in the user input data are extracted to fill the slots to obtain slot information, which specifically includes:

[0043] When the pre-consultation intention is identified, extracting the consultation type field in the user input data to obtain the consultation keyword;

[0044] Perform named entity recognition and relationship extraction on the medical inquiry keywords, so as to match the slots corresponding to the medical inquiry keywords of different dimensions;

[0045] The matching medical consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots at least include a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0046] In this solution, the strategy of dynamically adjusting slot filling based on the slot information and user feedback data specifically includes:

[0047] Acquiring user feedback data, wherein the user feedback data at least includes voice data, answer speed, and emotional feedback data;

[0048] Obtaining consultation strategies for different slots based on the user feedback data combined with corresponding slot information;

[0049] Based on the preset retrieval technology combined with the consultation strategy, target data is obtained from the preset database to obtain health management suggestions, wherein the preset database includes medical literature and clinical guidelines, and the target data includes various health management measures that match the current consultation strategy.

[0050] In this solution, the medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction, specifically including:

[0051] Obtaining the medical answer and the health management suggestion, performing data conversion, and visually displaying them using the user interaction interface;

[0052] During the display process, user interaction data on the user interaction interface is obtained for human-computer interaction, wherein the user interaction data is transmitted based on a preset encryption authority.

[0053] The third aspect of the present invention provides a computer-readable storage medium, which includes a machine program for an intelligent medical pre-diagnosis method based on a multimodal large model. When the program for the intelligent medical pre-diagnosis method based on a multimodal large model is executed by a processor, the steps of an intelligent medical pre-diagnosis method based on a multimodal large model as described in any one of the above items are implemented.

[0054] The present invention discloses an intelligent medical pre-diagnosis method, system and medium based on a multimodal large model, which adopts a natural language processing and multimodal fusion method, supports multiple input methods such as voice, picture, text, etc., improves the flexibility of information input, enables patients to provide health information in the most natural way, and reduces the difficulty and error rate of information entry, wherein the specific beneficial effects include the following:

[0055] (1) A more efficient multimodal input mechanism, which effectively improves the input mechanism in the existing technology by integrating advanced technologies such as natural language processing, image recognition and generative large models. Specifically, it includes: comprehensive information integration. By combining multiple input methods such as text input, voice recognition and medical image processing, the system can extract information from the patient's oral statement, uploaded medical records and historical medical records, automatically generate structured data, and greatly improve the completeness and accuracy of information collection; and reduce the input burden on patients. Compared with the existing technology, through large-scale pre-training and automated information extraction and slot filling mechanism, the manual input steps are reduced, making it more convenient for patients to enter information and improving the user experience of pre-consultation.

[0056] (2) Dynamically adjusted consultation interaction: By introducing dynamic medical history information collection and interaction optimization modules, real-time interaction optimization with patients is achieved, including: personalization and real-time adjustment, which can instantly adjust the subsequent consultation content and question order according to the patient's answers during the interaction. For example, when it is detected that the patient mentions a specific symptom, the system will immediately delve into the relevant details to ensure that the doctor obtains sufficient effective information to make accurate clinical judgments. And intelligent rhythm adjustment: by analyzing the patient's answer speed, emotional feedback and other data, the system can fine-tune the rhythm of the interaction, which not only ensures the efficiency of information collection, but also allows patients to experience a more humane consultation process, thereby improving the user experience.

[0057] (3) Medical knowledge integration supported by large models, through the integration of large-scale medical knowledge bases and the latest large model technology, can provide reliable support in a wide range of clinical scenarios. The specific advantages are as follows: stronger knowledge coverage and immediate support, based on the retrieval-enhanced generation model, it can instantly extract relevant information from a large amount of medical literature and clinical guidelines, and provide patients with the latest and most reliable medical advice, thereby improving the system's ability to respond to complex conditions; and accurate intention recognition and decision support: through deeply integrated natural language processing technology, the system can more accurately identify the patient's real needs, thereby providing accurate auxiliary decision-making for subsequent diagnosis and treatment processes.

[0058] (4) Intelligent slot filling and information framework construction. Named entity recognition and relationship extraction are performed on the keywords input by the patient to structure the patient's symptom description, medical history, living habits and other multi-dimensional information, making the entire information framework more comprehensive and improving the accuracy of information collection. Even if the patient's expression is vague or unclear, it can still be accurately captured by the deep learning model and filled into the appropriate slot, thereby ensuring that the information is comprehensive and complete; and providing a solid foundation for subsequent personalized treatment. Structured information not only facilitates doctors to quickly browse and understand, but also lays the foundation for subsequent personalized health management recommendations, which can better meet the unique needs of each patient.

[0059] (5) Automatic generation and delivery of pre-consultation reports. Specifically, the generation and delivery mechanism of pre-consultation reports has been further improved. Information is automatically delivered to doctors and patients through automatically generated pre-consultation reports. The advantage is that it improves the efficiency of diagnosis and treatment. Doctors can view detailed pre-consultation reports before formal consultation, which greatly shortens the time of on-site consultation and improves the efficiency of the overall medical system. It also reduces the psychological burden of patients, allowing them to understand their own symptom analysis before diagnosis and treatment, reducing tension during consultation and making doctor-patient communication smoother.

[0060] To sum up, based on the existing pre-consultation technology, the present invention not only improves the efficiency and accuracy of information input, but also enhances the adaptability of the system in complex medical scenarios through the integration and application of multimodal large models. Compared with the existing technology, its dynamic, intelligent and comprehensive consultation mechanism has brought the practicality and reliability of the pre-consultation system to a new level. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flowchart of an intelligent medical pre-diagnosis method based on a multimodal large model of the present invention is shown;

[0062] Figure 2 A schematic diagram of a pre-diagnosis dialogue of an intelligent medical pre-diagnosis method based on a multimodal large model of the present invention is shown;

[0063] Figure 3 A health management schematic diagram of an intelligent medical pre-diagnosis method based on a multi-modal large model of the present invention is shown;

[0064] Figure 4 A block diagram of an intelligent medical pre-diagnosis system based on a multimodal large model of the present invention is shown. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0067] Figure 1 A flowchart of an intelligent medical pre-diagnosis method based on a multimodal large model of the present application is shown.

[0068] like Figure 1 As shown, the present application discloses an intelligent medical pre-diagnosis method based on a multimodal large model, comprising the following steps:

[0069] S102, obtaining user input data to identify target intent, wherein the target intent includes knowledge question answering intent and pre-diagnosis inquiry intent;

[0070] S104, when the knowledge question-answering intention is identified, a medical answer is obtained using a multimodal large model based on the medical knowledge graph and the knowledge keywords in the user input data;

[0071] S106, when the pre-consultation intention is identified, extract the consultation keywords in the user input data to fill the slots to obtain slot information, and dynamically adjust the slot filling strategy based on the slot information and user feedback data;

[0072] S108, visually displaying the medical answers and health management suggestions based on a preset user interaction interface, and acquiring user interaction data for human-computer interaction.

[0073] It should be noted that in this embodiment, the user's demand intention must be identified first before a pre-consultation can be conducted, wherein it is determined whether the intention input by the user is a pre-consultation. If it is other intentions such as knowledge questions and answers, a prompt is generated to provide an explainable answer. For the pre-consultation intention, the slot is filled, and the medical history information is dynamically collected based on the slot information and the patient's condition. The patient's specific symptoms, medical history and other personal information are used to personalize the interaction process, and the order and content of the questions are dynamically adjusted according to the patient's answers, so as to more accurately capture relevant information and generate personalized health management recommendations.

[0074] Specifically, the user input data is first obtained to identify the target intent, so as to identify whether it is a knowledge question and answer intent or a pre-diagnosis intention. When the knowledge question and answer intent is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data.

[0075] Furthermore, when the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data.

[0076] In the process of dynamically adjusting slot filling, the target data can be obtained from the preset database based on the preset retrieval technology combined with the consultation strategy to obtain health management suggestions, so as to visualize the medical answers and the health management suggestions based on the preset user interaction interface, and obtain user interaction data for human-computer interaction.

[0077] According to an embodiment of the present invention, obtaining user input data to identify target intent specifically includes:

[0078] Obtain user input data of the information processing interface;

[0079] The user input data is analyzed and processed based on a preset model to obtain different target intentions.

[0080] It should be noted that in this embodiment, intent recognition plays a vital role. It is not only the entrance to information processing, but also the cornerstone to ensure efficient and accurate subsequent interactions. It specifically adopts advanced natural language processing technology, including but not limited to deep learning models. For example, the preset model used in this embodiment can be a BERT model or a Transformers model, so as to achieve a deep understanding of user input. Among them, by learning from a large number of medical annotated corpora, the model can effectively distinguish between pre-consultation requests and other types of medical consultations, such as general health knowledge inquiries, drug information needs, etc.

[0081] According to an embodiment of the present invention, when the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on a medical knowledge graph combined with knowledge keywords in the user input data, specifically including:

[0082] When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword;

[0083] Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results;

[0084] The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

[0085] It should be noted that in this embodiment, when the user's intention is for medical consultation (such as health knowledge questions and answers or drug information inquiries), a huge medical knowledge graph and advanced generation models are used to provide users with scientific, accurate, and easy-to-understand answers to meet users' instant information needs and improve user experience. This not only meets users' instant information needs and enhances user experience, but also reduces the burden on professional medical resources.

[0086] Specifically, when the knowledge question and answer intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keywords, such as "liver health", "drug query", etc., and retrieval matching is performed based on the knowledge keywords and the medical knowledge graph to obtain matching results. Finally, a multimodal large model is used to analyze and process the matching results and user input data to obtain the medical answer. During the analysis and processing, retrieval enhancement generation technology is integrated to generate the medical answer.

[0087] According to an embodiment of the present invention, when the pre-consultation intention is identified, extracting the consultation keywords in the user input data to fill the slots to obtain the slot information specifically includes:

[0088] When the pre-consultation intention is identified, extracting the consultation type field in the user input data to obtain the consultation keyword;

[0089] Perform named entity recognition and relationship extraction on the medical inquiry keywords, so as to match the slots corresponding to the medical inquiry keywords of different dimensions;

[0090] The matching medical consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots at least include a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0091] It should be noted that in this embodiment, the slot filling mechanism is introduced to further deepen the understanding of the user's pre-consultation intention, which is achieved by accurately matching the keywords in the user input with the predefined slot templates. The specific slots cover multiple dimensions such as symptom description, duration, past medical history, and living habits. The purpose is to build a comprehensive and detailed patient information framework, so that the user's pre-consultation intention can be analyzed in detail. Among them, by integrating named entity recognition, relationship extraction and other technologies, even complex or ambiguous user expressions can be accurately captured. After completing the slot filling, not only does it have a structured information foundation, but it also provides solid support for subsequent personalized user interactions.

[0092] Specifically, when the pre-consultation intention is identified, the consultation type field in the user input data is extracted to obtain the consultation keyword, such as "yellowing eyes", "abdominal pain and nausea", "multiple nausea and dizziness", etc., and the consultation keyword is subjected to named entity recognition and relationship extraction processing, so as to match the slots corresponding to the consultation keywords of different dimensions, such as Figure 2 As shown, a dialogue diagram of pre-consultation is displayed, and matching consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots include at least a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0093] According to an embodiment of the present invention, a strategy for dynamically adjusting slot filling based on the slot information and user feedback data specifically includes:

[0094] Acquiring user feedback data, wherein the user feedback data at least includes voice data, answer speed, and emotional feedback data;

[0095] Obtaining consultation strategies for different slots based on the user feedback data combined with corresponding slot information;

[0096] Based on the preset retrieval technology combined with the consultation strategy, target data is obtained from the preset database to obtain health management suggestions, wherein the preset database includes medical literature and clinical guidelines, and the target data includes various health management measures that match the current consultation strategy.

[0097] It should be noted that in this embodiment, according to the information of the filled slots and the real-time feedback of the patients, the information collection strategy is dynamically adjusted, the depth and breadth of the questions are flexibly controlled, and the order of questions is optimized, such as Figure 3 As shown in the figure, it is a health management diagram. When a patient mentions a specific symptom, the system will ask for relevant details in depth. At the same time, according to the patient's answer speed and emotional feedback, the interaction rhythm is fine-tuned to ensure that the interaction process is efficient and humane. When applied, the retrieval enhancement generation technology is integrated to obtain the latest and most relevant knowledge from medical literature and clinical guidelines in real time to ensure the timeliness and scientificity of the recommendations. Specifically, Figure 3 As shown, in this embodiment, the health management suggestions obtained based on the medical symptoms proposed by the patient include four health management measures, specifically:

[0098] 1. Regular work and rest: Maintain a regular work and rest schedule, avoid sitting for long periods of time and monotonous activities to reduce symptoms of sleepiness;

[0099] 2. Healthy diet: Pay attention to balanced nutrition, avoid picky eating and partial eating, and ensure adequate intake of vitamins and minerals;

[0100] 3. Psychological support: If you have a large mood swing, it is recommended to seek psychological counseling or psychological support to relieve stress and anxiety;

[0101] 4. Seek medical attention: If symptoms persist or worsen, it is recommended to seek medical attention promptly for detailed examination and diagnosis to rule out potential neurological diseases or other health problems.

[0102] According to an embodiment of the present invention, the visual display of the medical answers and health management suggestions based on a preset user interaction interface and the acquisition of user interaction data for human-computer interaction specifically include:

[0103] Obtaining the medical answer and the health management suggestion, performing data conversion, and visually displaying them using the user interaction interface;

[0104] During the display process, user interaction data on the user interaction interface is obtained for human-computer interaction, wherein the user interaction data is transmitted based on a preset encryption authority.

[0105] It should be noted that in this embodiment, the user interaction interface described supports multiple interaction methods such as text and voice, and is suitable for multiple platforms such as mobile devices and web clients. The user interaction interface focuses on simplicity and intuitiveness, and can guide users to smoothly complete information input and obtain feedback. Therefore, when applied, not only can the medical answers and the health management suggestions be displayed visually, but also during the display process, the user interaction data on the user interaction interface can be obtained for human-computer interaction, wherein the user interaction data is transmitted based on preset encryption permissions, especially using encryption technology and strict permission control, so as to protect the user's personal privacy and comply with relevant laws, regulations and industry standards.

[0106] Figure 4 A block diagram of an intelligent medical pre-diagnosis system based on a multimodal large model of the present invention is shown.

[0107] like Figure 4 As shown, the present invention discloses an intelligent medical pre-diagnosis system based on a multimodal large model, including a memory and a processor, wherein the memory includes an intelligent medical pre-diagnosis method program based on a multimodal large model, and when the intelligent medical pre-diagnosis method program based on a multimodal large model is executed by the processor, the following steps are implemented:

[0108] It should be noted that in this embodiment, the user's demand intention must be identified first before a pre-consultation can be conducted, wherein it is determined whether the intention input by the user is a pre-consultation. If it is other intentions such as knowledge questions and answers, a prompt is generated to provide an explainable answer. For the pre-consultation intention, the slot is filled, and the medical history information is dynamically collected based on the slot information and the patient's condition. The patient's specific symptoms, medical history and other personal information are used to personalize the interaction process, and the order and content of the questions are dynamically adjusted according to the patient's answers, so as to more accurately capture relevant information and generate personalized health management recommendations.

[0109] Specifically, the user input data is first obtained to identify the target intent, so as to identify whether it is a knowledge question and answer intent or a pre-diagnosis intention. When the knowledge question and answer intent is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data.

[0110] Furthermore, when the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data.

[0111] In the process of dynamically adjusting slot filling, the target data can be obtained from the preset database based on the preset retrieval technology combined with the consultation strategy to obtain health management suggestions, so as to visualize the medical answers and the health management suggestions based on the preset user interaction interface, and obtain user interaction data for human-computer interaction.

[0112] According to an embodiment of the present invention, obtaining user input data to identify target intent specifically includes:

[0113] Obtain user input data of the information processing interface;

[0114] The user input data is analyzed and processed based on a preset model to obtain different target intentions.

[0115] It should be noted that in this embodiment, intent recognition plays a vital role. It is not only the entrance to information processing, but also the cornerstone to ensure efficient and accurate subsequent interactions. It specifically adopts advanced natural language processing technology, including but not limited to deep learning models. For example, the preset model used in this embodiment can be a BERT model or a Transformers model, so as to achieve a deep understanding of user input. Among them, by learning from a large number of medical annotated corpora, the model can effectively distinguish between pre-consultation requests and other types of medical consultations, such as general health knowledge inquiries, drug information needs, etc.

[0116] According to an embodiment of the present invention, when the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on a medical knowledge graph combined with knowledge keywords in the user input data, specifically including:

[0117] When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword;

[0118] Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results;

[0119] The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

[0120] It should be noted that in this embodiment, when the user's intention is for medical consultation (such as health knowledge questions and answers or drug information inquiries), a huge medical knowledge graph and advanced generation models are used to provide users with scientific, accurate, and easy-to-understand answers to meet users' instant information needs and improve user experience. This not only meets users' instant information needs and enhances user experience, but also reduces the burden on professional medical resources.

[0121] Specifically, when the knowledge question and answer intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keywords, such as "liver health", "drug query", etc., and retrieval matching is performed based on the knowledge keywords and the medical knowledge graph to obtain matching results. Finally, a multimodal large model is used to analyze and process the matching results and user input data to obtain the medical answer. During the analysis and processing, retrieval enhancement generation technology is integrated to generate the medical answer.

[0122] According to an embodiment of the present invention, when the pre-consultation intention is identified, extracting the consultation keywords in the user input data to fill the slots to obtain the slot information specifically includes:

[0123] When the pre-consultation intention is identified, extracting the consultation type field in the user input data to obtain the consultation keyword;

[0124] Perform named entity recognition and relationship extraction on the medical inquiry keywords, so as to match the slots corresponding to the medical inquiry keywords of different dimensions;

[0125] The matching medical consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots at least include a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0126] It should be noted that in this embodiment, the slot filling mechanism is introduced to further deepen the understanding of the user's pre-consultation intention, which is achieved by accurately matching the keywords in the user input with the predefined slot templates. The specific slots cover multiple dimensions such as symptom description, duration, past medical history, and living habits. The purpose is to build a comprehensive and detailed patient information framework, so that the user's pre-consultation intention can be analyzed in detail. Among them, by integrating named entity recognition, relationship extraction and other technologies, even complex or ambiguous user expressions can be accurately captured. After completing the slot filling, not only does it have a structured information foundation, but it also provides solid support for subsequent personalized user interactions.

[0127] Specifically, when the pre-consultation intention is identified, the consultation type field in the user input data is extracted to obtain the consultation keyword, such as "yellowing eyes", "abdominal pain and nausea", "multiple nausea and dizziness", etc., and the consultation keyword is subjected to named entity recognition and relationship extraction processing, so as to match the slots corresponding to the consultation keywords of different dimensions, such as Figure 2 As shown, a dialogue diagram of pre-consultation is displayed, and matching consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots include at least a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

[0128] According to an embodiment of the present invention, a strategy for dynamically adjusting slot filling based on the slot information and user feedback data specifically includes:

[0129] Acquiring user feedback data, wherein the user feedback data at least includes voice data, answer speed, and emotional feedback data;

[0130] Obtaining consultation strategies for different slots based on the user feedback data combined with corresponding slot information;

[0131] Based on the preset retrieval technology combined with the consultation strategy, target data is obtained from the preset database to obtain health management suggestions, wherein the preset database includes medical literature and clinical guidelines, and the target data includes various health management measures that match the current consultation strategy.

[0132] It should be noted that in this embodiment, according to the information of the filled slots and the real-time feedback of the patients, the information collection strategy is dynamically adjusted, the depth and breadth of the questions are flexibly controlled, and the order of questions is optimized, such as Figure 3As shown in the figure, it is a health management diagram. When a patient mentions a specific symptom, the system will ask for relevant details in depth. At the same time, according to the patient's answer speed and emotional feedback, the interaction rhythm is fine-tuned to ensure that the interaction process is efficient and humane. When applied, the retrieval enhancement generation technology is integrated to obtain the latest and most relevant knowledge from medical literature and clinical guidelines in real time to ensure the timeliness and scientificity of the recommendations. Specifically, Figure 3 As shown, in this embodiment, the health management suggestions obtained based on the medical symptoms proposed by the patient include four health management measures, specifically:

[0133] 1. Regular work and rest: Maintain a regular work and rest schedule, avoid sitting for long periods of time and monotonous activities to reduce symptoms of sleepiness;

[0134] 2. Healthy diet: Pay attention to balanced nutrition, avoid picky eating and partial eating, and ensure adequate intake of vitamins and minerals;

[0135] 3. Psychological support: If you have a large mood swing, it is recommended to seek psychological counseling or psychological support to relieve stress and anxiety;

[0136] 4. Seek medical attention: If symptoms persist or worsen, it is recommended to seek medical attention promptly for detailed examination and diagnosis to rule out potential neurological diseases or other health problems.

[0137] According to an embodiment of the present invention, the visual display of the medical answers and health management suggestions based on a preset user interaction interface and the acquisition of user interaction data for human-computer interaction specifically include:

[0138] Obtaining the medical answer and the health management suggestion, performing data conversion, and visually displaying them using the user interaction interface;

[0139] During the display process, user interaction data on the user interaction interface is obtained for human-computer interaction, wherein the user interaction data is transmitted based on a preset encryption authority.

[0140] It should be noted that in this embodiment, the user interaction interface described supports multiple interaction methods such as text and voice, and is suitable for multiple platforms such as mobile devices and web clients. The user interaction interface focuses on simplicity and intuitiveness, and can guide users to smoothly complete information input and obtain feedback. Therefore, when applied, not only can the medical answers and the health management suggestions be displayed visually, but also during the display process, the user interaction data on the user interaction interface can be obtained for human-computer interaction, wherein the user interaction data is transmitted based on preset encryption permissions, especially using encryption technology and strict permission control, so as to protect the user's personal privacy and comply with relevant laws, regulations and industry standards.

[0141] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for an intelligent medical pre-diagnosis method based on a multimodal large model. When the program for an intelligent medical pre-diagnosis method based on a multimodal large model is executed by a processor, the steps of an intelligent medical pre-diagnosis method based on a multimodal large model as described in any one of the above items are implemented.

[0142] The present invention discloses an intelligent medical pre-consultation method, system and medium based on a multimodal large model, which adopts natural language processing and multimodal fusion, supports multiple input methods such as voice, picture, text, etc., improves the flexibility of information input, enables patients to provide health information in the most natural way, and reduces the difficulty and error rate of information entry.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0146] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and other media that can store program codes.

[0147] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. An intelligent medical pre-diagnosis method based on a multimodal large model, characterized in that: The following steps are involved: Obtain user input data to identify target intent, wherein the target intent includes knowledge question answering intent and pre-diagnosis intention, wherein: When the knowledge question-answering intention is identified, a medical answer is obtained using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data; When the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data; The medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction.

2. According to claim 1, the intelligent medical pre-diagnosis method based on multimodal large model is characterized in that: The obtaining of user input data to identify the target intention specifically includes: Obtain user input data of the information processing interface; The user input data is analyzed and processed based on a preset model to obtain different target intentions.

3. According to claim 2, the intelligent medical pre-diagnosis method based on a multimodal large model is characterized in that: When the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data, specifically including: When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword; Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results; The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

4. The intelligent medical pre-diagnosis method based on a multimodal large model according to claim 3 is characterized in that: When the pre-consultation intention is identified, extracting the consultation keywords in the user input data to fill the slots to obtain the slot information specifically includes: When the pre-consultation intention is identified, extracting the consultation type field in the user input data to obtain the consultation keyword; Perform named entity recognition and relationship extraction on the medical inquiry keywords, so as to match the slots corresponding to the medical inquiry keywords of different dimensions; The matching medical consultation keywords are filled into corresponding slots to obtain the slot information, wherein the slots at least include a symptom description slot, a duration slot, a past medical history slot, and a living habit slot.

5. The intelligent medical pre-diagnosis method based on a multimodal large model according to claim 4 is characterized in that: The strategy for dynamically adjusting slot filling based on the slot information and user feedback data specifically includes: Acquiring user feedback data, wherein the user feedback data at least includes voice data, answer speed, and emotional feedback data; Obtaining consultation strategies for different slots based on the user feedback data combined with corresponding slot information; Based on the preset retrieval technology combined with the consultation strategy, target data is obtained from the preset database to obtain health management suggestions, wherein the preset database includes medical literature and clinical guidelines, and the target data includes various health management measures that match the current consultation strategy.

6. The intelligent medical pre-diagnosis method based on a multimodal large model according to claim 5, characterized in that: The method of visually displaying the medical answers and health management suggestions based on a preset user interaction interface and acquiring user interaction data for human-computer interaction specifically includes: Obtaining the medical answer and the health management suggestion, performing data conversion, and visually displaying them using the user interaction interface; During the display process, user interaction data on the user interaction interface is obtained for human-computer interaction, wherein the user interaction data is transmitted based on a preset encryption authority.

7. An intelligent medical pre-diagnosis system based on a multimodal large model, characterized in that: The system comprises a memory and a processor, wherein the memory comprises an intelligent medical pre-diagnosis method program based on a multimodal large model, and when the intelligent medical pre-diagnosis method program based on a multimodal large model is executed by the processor, the following steps are implemented: Obtain user input data to identify target intent, wherein the target intent includes knowledge question answering intent and pre-diagnosis intention, wherein: When the knowledge question-answering intention is identified, a medical answer is obtained using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data; When the pre-consultation intention is identified, the consultation keywords in the user input data are extracted to fill the slots to obtain slot information, and the slot filling strategy is dynamically adjusted based on the slot information and user feedback data; The medical answers and health management suggestions are visualized based on a preset user interaction interface, and user interaction data is obtained for human-computer interaction.

8. The intelligent medical pre-diagnosis system based on multimodal large model according to claim 7 is characterized in that: The obtaining of user input data to identify the target intention specifically includes: Obtain user input data of the information processing interface; The user input data is analyzed and processed based on a preset model to obtain different target intentions.

9. The intelligent medical pre-diagnosis system based on multimodal large model according to claim 8, characterized in that: When the knowledge question-answering intention is identified, a medical answer is obtained by using a multimodal large model based on the medical knowledge graph combined with the knowledge keywords in the user input data, specifically including: When the knowledge question-answering intention is identified, the knowledge type field in the user input data is extracted to obtain the knowledge keyword; Performing search and matching based on the knowledge keywords and the medical knowledge graph to obtain matching results; The matching results and user input data are analyzed and processed using a multimodal large model to obtain the medical answer.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program for an intelligent medical pre-diagnosis method based on a multimodal large model. When the program for an intelligent medical pre-diagnosis method based on a multimodal large model is executed by a processor, the steps of an intelligent medical pre-diagnosis method based on a multimodal large model as described in any one of claims 1 to 6 are implemented.

Images