Intelligent medical auxiliary system and method based on multi-modal large model

Abstract

The invention relates to the technical field of industrial intelligence, and discloses an intelligent medical auxiliary system and method based on a multi-modal large model. The invention discloses an intelligent medical auxiliary system based on a multi-modal large model, and the system comprises an identity verification module which is used for carrying out the identity verification of a login user, and obtaining an identity feature; the information input and processing module is used for acquiring input data of a user and preprocessing the input data to obtain query features; the information retrieval module is used for performing retrieval in the trained large model based on query features to obtain multi-modal associated data; the result generation module is used for performing intention recognition on the user input data based on semantic analysis, and sorting the associated data according to an intention recognition result to obtain a retrieval result; and the display module is used for displaying the retrieval result according to the identity characteristics. According to the invention, full-process intelligent medical auxiliary processing from user input to result display is realized through multi-module cooperation, and the adaptability of the auxiliary system in a complex medical environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly relates to an intelligent medical assistance system and method based on a multimodal large model. Background Art

[0002] Introducing artificial intelligence technology into the medical field can significantly enhance the diagnosis and treatment capabilities of primary care doctors, improve the efficiency and accuracy of medical services, help doctors master new knowledge and promote the development of professional skills. Artificial intelligence can process and analyze a large amount of medical data, reduce the missed diagnosis and misdiagnosis of rare diseases. As an intelligent assistant, it can provide knowledge and decision-making support for doctors, reduce the workload of work and learning, and also solve the problem of uneven distribution of grass-roots medical resources, help doctors quickly supplement and update knowledge, assist in clinical decision-making, and provide simulation training and case learning functions.

[0003] With the rapid development of medical informatization and artificial intelligence technology, intelligent medical assistance systems have shown great potential in improving medical service efficiency, optimizing the patient experience, and promoting precision medicine. However, in complex medical scenarios, traditional single-modal or rule-driven medical assistance systems often struggle to meet diverse needs, especially when dealing with multi-source heterogeneous data, showing obvious limitations. For this reason, intelligent medical assistance systems based on multimodal large models have emerged as a key technical direction to solve this problem. Multimodal large models are deep learning models that can simultaneously process multiple types of data such as text, images, and audio. Their core advantages lie in their powerful representation ability and cross-modal alignment ability. For example, the CLIP model achieves semantic alignment between text and images through contrastive learning, while the M6 model further extends to the joint representation of text, images, and other modalities. This technology provides new possibilities for the medical field, enabling the system to more comprehensively understand the user's input information and quickly retrieve relevant data from a large-scale medical knowledge base. In practical applications, medical assistance systems still face challenges in terms of intelligence and adaptability. Summary of the Invention

[0004] The present invention aims to provide an intelligent medical assistance system and method based on a multimodal large model, which realizes the full-process intelligent medical assistance processing from user input to result display through multi-module collaboration, and improves the adaptability of the assistance system in complex medical environments.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] An intelligent medical assistance system based on a multimodal large model, comprising:

[0007] An identity verification module for verifying the identity of the logged-in user and obtaining identity characteristics;

[0008] An information input and processing module, which is used to obtain the input data of the user and preprocess the input data to obtain query features;

[0009] An information retrieval module, which is used to retrieve in the trained multi-modal large model based on the query features to obtain multi-modal associated data;

[0010] A result generation module, which is used to identify the intention of the user input data based on semantic analysis, and sort out the associated data according to the intention recognition result to obtain the retrieval result;

[0011] A display module, which is used to display the retrieval result according to the identity characteristics.

[0012] The principle and advantages of this solution are as follows: In practical applications, on the one hand, identity authentication is convenient for ensuring system security, and on the other hand, it is convenient for providing personalized data support and providing user background information for subsequent module processing; obtaining and processing the user's input data is convenient for improving data universality and generating high-quality query features, providing a basis for subsequent retrieval; using the powerful representation ability of the multi-modal large model to match the query features with the knowledge stored in the model to quickly find data highly relevant to the user input from a large-scale medical knowledge base; screening and sorting out the associated data according to the intention to generate a structured retrieval result, optimizing the personalization degree of the result, so as to accurately understand the user's needs, generate results that meet the user's intention, improve the intelligent level of the system, and improve the user experience; finally, displaying the retrieval result according to the identity characteristics to improve the readability and usability of the result. Through the collaborative work of the above multi-modules, the system can provide accurate and efficient services in complex medical scenarios and promote the development of intelligent healthcare.

[0013] Preferably, as an improvement, the information input and processing module includes:

[0014] An input receiving sub-module, which is used to receive multi-modal input data;

[0015] A data processing sub-module, which is used for data cleaning, annotation and encoding;

[0016] A feature fusion sub-module, which is used to integrate data features of different modalities into a unified representation space and generate the final query features.

[0017] Technical effect: It is convenient to improve the quality of the input data, thereby improving the accuracy of subsequent retrieval.

[0018] Preferably, as an improvement, the result generation module includes:

[0019] The intention recognition sub-module performs semantic analysis on the text input by the user through natural language processing technology, extracts the core meaning, and maps the core meaning to predefined intention categories based on an intention classification model pre-trained by a supervised learning method;

[0020] The associated data sorting sub-module filters the associated data according to the intention recognition result, sorts the filtered data according to the relevance, obtains the confidence score according to the confidence score model, adjusts the sorting weight according to the confidence score, optimizes the sorting, and integrates the scattered data segments in each ranking into complete knowledge units to obtain the retrieval result.

[0021] Technical effect: By comprehensively and completely obtaining the data associated with the query features through the associated data, and then sorting out the associated data through intention recognition, the accuracy, understanding depth and adaptability of knowledge recognition are improved.

[0022] Preferably, as an improvement, the confidence score model is:

[0023] p n = w1q n + w2l n + w3f n + w4t n

[0024] Z x = Normal Min-Max (p n )

[0025] Wherein, w1, w2, w3, and w4 are weight parameters, q n is the regional influence parameter value, l n is the data source influence parameter value, f n is the user feedback influence parameter value, t n is the time influence parameter value, p n is the initial confidence score, Normal Min-Max () represents the min-max normalization processing function, and Z x is the confidence score.

[0026] Technical effect: The confidence score model can dynamically adjust the model parameters in real time, improving the pertinence of the output results.

[0027] Preferably, as an improvement, it further includes a related field feature fusion module, which is used to match the associated field of the input information from a preset mapping table, fuse the associated field with the query features, retrieve the field-associated data through the information retrieval module, and sort out the field-associated result based on the intention recognition result.

[0028] Technical effect: Facilitate the provision of extended resources, assist users in knowledge expansion, and improve the user experience effect.

[0029] Preferably, as an improvement, it further includes a modality verification module for verifying the fused data through unimodal data and optimizing the associated data based on the verification result; the modality verification module includes:

[0030] A unimodal data retrieval sub-module for retrieving unimodal associated data in a trained large model based on query features;

[0031] A comparison and analysis sub-module for comparing the unimodal associated data with the multimodal associated data to determine whether the error exceeds a threshold;

[0032] An optimization sub-module for obtaining the confidence of the error content when the comparison error exceeds the threshold, and when the confidence meets the requirements, using the unimodal associated data as supplementary data and sending the supplementary data to the result generation module for processing.

[0033] Technical effect: Facilitate improving the integrity of knowledge acquisition.

[0034] Preferably, as an improvement, the display module further includes:

[0035] A requirements analysis sub-module for obtaining detailed identity information from the identity verification module, classifying the identity information, and determining the display requirements according to the identity category;

[0036] A sensitive information filtering sub-module for filtering sensitive information based on the identity information;

[0037] A result adaptation and optimization sub-module for adapting and optimizing according to the display requirements and the retrieval results.

[0038] Technical effect: Facilitate significantly enhancing the user experience and practicality of the system, and at the same time meet the needs of different user groups.

[0039] Preferably, as an improvement, the intention category is also weighted according to the identity category.

[0040] Technical effect: Facilitate improving the accuracy of intention recognition.

[0041] Preferably, as an improvement, the multimodality includes text, image, audio, video, and animation modalities.

[0042] Technical effect: Facilitate ensuring the comprehensiveness and diversity of retrieval results.

[0043] It also includes an intelligent medical assistance method based on a multimodal large model, which is applied to the above-mentioned intelligent medical assistance system based on a multimodal large model. Brief Description of the Drawings

[0044] Figure 1 This is a schematic structural diagram of an embodiment of the present invention. Detailed Embodiment

[0045] The following is a more detailed description through specific embodiments:

[0046] The embodiment is basically as shown in the attached Figure 1 drawing:

[0047] An intelligent medical assistance system based on a multimodal large model includes an identity authentication module, an information input and processing module, an information retrieval module, a result generation module, and a display module. Through the cooperation of the above-mentioned multiple modules, it realizes the full-process intelligent medical assistance processing from user input to result display, and improves the adaptability of the assistance system in a complex medical environment.

[0048] The identity authentication module is used to authenticate the logged-in user and obtain identity characteristics. By authenticating the identity, it ensures the security of the system, prevents unauthorized access, and at the same time provides personalized data support to provide user background information for subsequent modules. Specifically, the identity authentication includes initial authentication and secondary authentication. The initial authentication is password authentication, and identity characteristics such as age, gender, occupation, and medical history are obtained during login; the secondary authentication is based on the user's behavior pattern for identity authentication. The behavior pattern includes typing habits, mouse movement trajectories, touch screen operations, etc. The historical behavior data of the user is collected, a behavior feature model is established, and data is collected in real time during the operation and compared with the model to ensure the authenticity of the identity.

[0049] The information input and processing module is used to obtain the input data of the user and preprocess the input data to obtain query characteristics; the information input and processing module includes an input receiving sub-module, a data processing sub-module, and a feature fusion sub-module. The input receiving sub-module is used to receive multimodal input data, including text, image, audio, video, and animation modalities, and extract the features of the input data through a multimodal encoder; the data processing sub-module is used for data cleaning, annotation, and encoding, and through standardization processing, it is convenient to ensure the data quality; the feature fusion sub-module is used to integrate the data features of different modalities into a unified representation space and generate the final query characteristics; through the information input and processing module, it is convenient to generate high-quality query characteristics, provide a basis for subsequent retrieval, and at the same time realize the efficient processing of multimodal data and improve the applicability of the system.

[0050] An information retrieval module for retrieving in a trained multi-modal large model based on query features to obtain multi-modal associated data. Specifically, in the model training stage, feature vectors are generated for each stored knowledge point, and an efficient index is established. The similarity between the query features and the features of each knowledge point in the knowledge base is calculated, and the candidate knowledge points are sorted according to the similarity scores. Several knowledge points with the highest scores are selected as associated data. The priority of the retrieval results is adjusted through context awareness, duplicate information is removed to ensure the diversity of the results, and the optimized multi-modal associated data is passed to the subsequent module. Metadata (including source, user feedback, timestamp) is attached to each retrieval result for subsequent analysis and verification.

[0051] A result generation module for intent recognition of user input data based on semantic analysis and organizing the associated data according to the intent recognition results to obtain retrieval results. The result generation module includes an intent recognition sub-module and an associated data organizing sub-module. The intent recognition sub-module performs semantic analysis on the text input by the user through natural language processing technology, extracts the core meaning, converts the text into a vector that can be understood by the machine, and maps the core meaning to predefined intent categories based on an intent classification model pre-trained by a supervised learning method. Specifically, irrelevant characters such as punctuation marks and special symbols and stop words such as "de" and "shi" are removed through noise processing, the text is segmented into words or phrases, and the part of speech of each word is labeled to unify the text format. The words are restored to their basic forms through stemming and lemmatization. Key entities or topic words in the text are identified through TF-IDF, specific entities in the text (such as disease names and drug names) are detected, and the sentiment tendency of the text (such as positive or negative) is judged through sentiment analysis for supplementary intent recognition. The core meaning of the text is mapped to predefined intent categories. Training data with labels is collected, and each sample includes the user input text and the corresponding intent category. The intent categories are such as diagnosis request, treatment advice, health consultation, drug query, etc. A supervised learning method is used to build an intent classification model, and the intent classification model uses Transformer. The feature vector of the input text enters the model, and the model outputs the probability distribution of each intent category. The final intent category is determined according to the maximum probability. For example, if the probability of a diagnosis request is 0.8 and the probability of treatment advice is 0.2, then "diagnosis request" is returned as the final intent. To improve the credibility of the results, a threshold filtering sub-module is also included. A confidence threshold is set, and when it is lower than the threshold, "unknown intent" is returned. A multi-intent threshold is set, and when it is higher than the threshold, multiple intents can be recognized simultaneously, and an intent list sorted by probability is returned.

[0052] The associated data sorting sub-module filters the associated data according to the intention recognition result, sorts the filtered data according to the relevance, obtains the confidence score according to the confidence score model, adjusts the sorting weight according to the confidence score, and optimizes the sorting based on the preset optimization strategy. The preferred optimization strategy in this embodiment includes: determining whether the confidence scores of the data at each position exceed the preset adjustment threshold. When it exceeds, the sorting weight is mapped according to the confidence score in the preset query table. The higher the confidence score, the higher the sorting weight; when it does not exceed, the sorting weight is assigned 0; compare the sorting weight values of each position in turn. When the weight value of the latter position is greater than that of the previous position, the positions are swapped. Through the above optimization of sorting, both the influence of relevance and the influence of confidence are retained. Integrate the scattered data fragments at each position into complete knowledge units. In this embodiment, a structured reorganization method is used for integration, and the data is reorganized according to the standard structure of medical knowledge (such as etiology, symptoms, diagnosis, treatment, etc.) to obtain the retrieval result, and the retrieval result includes multi-modal data. The confidence score model is:

[0053] p n = w1q n + w2l n + w3f n + w4t n

[0054] Z x = Normal Min-Max (p n )

[0055] Among them, w1, w2, w3, and w4 are weight parameters, q n is the regional influence parameter value. Through the regional influence parameter value, it is convenient to automatically adjust the model to adapt to the new medical data distribution when the system detects changes in the incidence or symptom characteristics of a certain disease in a certain area, and improve the pertinence of diagnosis and suggestions; l n is the data source influence parameter value, f n is the user feedback influence parameter value, that is, the user feedback score; t n is the time influence parameter value, q n , l n , f n and t n are obtained through the preset query table, p n is the initial confidence score, Normal Min-Max () represents the min-max normalization processing function, and Z x is the confidence score

[0056] The calculation model of the regional influence parameter value is:

[0057]

[0058] D c,m is the distribution of disease-related incidence and symptom characteristic data for the current region, and m is the number of characteristics; D b,m is the reference data distribution. In this embodiment, the reference data is the global average data; μ m is the adjustment constant for the preset regional influence parameter value.

[0059] The calculation model for the data source influence parameter value is:

[0060]

[0061] where z l is the quality score of the i-th data source, and the quality score is obtained according to the preset scoring engine. F(q i , q j ) represents the distribution difference of the data source q i , q j , b is the total number of data sources, and β i is the adjustment constant for the data source influence parameter value.

[0062] The calculation model for the time influence parameter value is:

[0063] t n = ε t * t0 * e -k(t0-t)

[0064] where t n is the current time influence parameter value, t0 is the initial time influence parameter value, k is the cooling coefficient, t0 - t is the time interval, and ε t is the adjustment constant for the time influence parameter value.

[0065] This embodiment further includes a related field feature fusion module, which is used to match the associated field of the input information from the preset mapping table, fuse the associated field with the query feature, retrieve the field-associated data through the information retrieval module, and organize the field-associated results based on the intent recognition result. Fuse with related fields, such as combining with the public health field to analyze the disease transmission trend and group characteristics, and provide more macroscopic disease prevention and control suggestions for primary healthcare; combine with the drug R & D field to timely obtain the latest drug R & D achievements and clinical application information, and assist doctors in drug selection and treatment plan formulation.

[0066] It also includes a modality verification module for verifying the fused data through unimodal data and optimizing the associated data based on the verification results; the modality verification module includes a unimodal data retrieval sub-module, a comparative analysis sub-module, and an optimization sub-module. The unimodal data retrieval sub-module is used to retrieve unimodal associated data in the trained large model based on query features, and the query features are multiple unimodal features without feature fusion; the comparative analysis sub-module is used to compare the associated data of each unimodal with the associated data obtained by querying the multimodal fusion features to determine whether the error exceeds the threshold; specifically, by extracting the core keywords of the unimodal data and the multimodal data, checking the coincidence degree, aligning the unimodal data with the multimodal data to ensure that the described topics are the same; defining an error index to measure the difference between the unimodal data and the multimodal data, and the error index includes content difference, structural difference, and confidence difference. The content difference is the overlap rate or complementarity of the unimodal data and the multimodal data in content, and the structural difference is to compare the structural representation errors of the unimodal data and the multimodal data, such as nodes and edges in a knowledge graph. If the error exceeds the threshold, it is considered that there are omissions or deviations in the multimodal data. The optimization sub-module is used to obtain the confidence of the error content when the comparison error exceeds the threshold. When the confidence meets the requirements, the unimodal associated data is used as supplementary data and sent to the result generation module for processing.

[0067] A display module for displaying the retrieval results according to the identity features. According to the needs and usage habits of different user roles, such as primary care doctors, medical students, and specialist doctors, a personalized interaction interface is set up to provide customized function modules and information display methods, improving the convenience and efficiency of user use. The display module includes a requirements analysis sub-module, a sensitive information filtering sub-module, and a result adaptation and optimization sub-module. The requirements analysis sub-module is used to obtain detailed identity information from the identity verification module, classify the identity information, and determine the display requirements according to the identity category; the sensitive information filtering sub-module is used to filter sensitive information based on the identity information, set different access permissions according to the user identity, such as primary care doctors can only view de-identified patient data, and specialist doctors can access complete patient medical records and research data, and desensitize sensitive information, such as data anonymization and partial hiding, to effectively prevent the leakage of sensitive information and protect patient privacy and data security. The result adaptation and optimization sub-module is used to adapt and optimize according to the display requirements and the retrieval results, such as providing concise diagnostic suggestions and standardized treatment plans for primary care doctors, providing detailed theoretical backgrounds and case analyses for medical students, and providing complex statistical data and cutting-edge research results for specialist doctors. The intention category is also weighted according to the identity category, such as the intentions of specialist doctors and primary care doctors tend to be diagnosis and treatment plans; the intention of medical students tends to be display and analysis.

[0068] It further includes an intelligent medical assistance method based on a multimodal large model, which is applied to the intelligent medical assistance system based on the multimodal large model described above.

[0069] The above are only embodiments of the present invention. Specific technical solutions and / or common knowledge such as characteristics well known in the art are not described in detail herein. It should be noted that for those skilled in the art, without departing from the technical solution of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent. The protection scope claimed in this application shall be subject to the content of its claims, and the specific implementation manners described in the specification can be used to interpret the content of the claims.

Claims

1. An intelligent medical assistance system based on a multimodal large model, characterized in that: include: The authentication module is used to authenticate the logged-in user and obtain identity characteristics; The information input and processing module is used to obtain the user's input data and pre-process the input data to obtain the query features; The information retrieval module is used to search in the trained multimodal large model based on the query features to obtain multimodal related data; The result generation module is used to identify the intent of the user input data based on semantic analysis, and to organize the related data according to the intent identification results to obtain the search results; The display module is used to display the search results according to identity features.

2. According to claim 1, the intelligent medical assistance system based on multimodal large model is characterized in that: The information input and processing module includes: An input receiving submodule, used for receiving multi-modal input data; Data processing submodule, used for data cleaning, labeling and coding; The feature fusion submodule is used to integrate data features of different modalities into a unified representation space and generate the final query features.

3. The intelligent medical assistance system based on a multimodal large model according to claim 1, characterized in that: The result generation module comprises: The intent recognition submodule uses natural language processing technology to perform semantic analysis on the text input by the user, extract the core meaning, and map the core meaning to the predefined intent category based on the intent classification model pre-trained by the supervised learning method; The associated data sorting submodule filters the associated data according to the intent recognition results, sorts the filtered data according to relevance, obtains the confidence score according to the confidence scoring model, adjusts the sorting weight according to the confidence score, optimizes the sorting, integrates the scattered data fragments in each order into a complete knowledge unit, and obtains the retrieval results.

4. The intelligent medical assistance system based on a multimodal large model according to claim 3 is characterized in that: The confidence scoring model is: p n =w1q n +w2l n +w3f n +w4t n WITH x =Normal Min-Max (p n ) Among them, w1, w2, w3, w4 are weight parameters, q n is the regional influence parameter value, l n is the parameter value that affects the data source, f n is the parameter value affected by user feedback, t n is the time-influence parameter value, p n is the initial confidence score, Normal Min-Max () represents the minimum-maximum normalization function, Z x Rate the confidence level.

5. The intelligent medical assistance system based on a multimodal large model according to claim 1, characterized in that: It also includes a related domain feature fusion module, which is used to match the related domains of the input information from the preset mapping table, and fuse the related domains with the query features, retrieve the domain related data through the information retrieval module, and obtain the domain related results based on the intent recognition results.

6. The intelligent medical assistance system based on a multimodal large model according to claim 4, characterized in that: It also includes a modal verification module, which is used to verify the fused data through single modal data and optimize the associated data based on the verification results; the modal verification module includes: The unimodal data retrieval submodule is used to retrieve the trained large model based on the query features to obtain the unimodal related data; The comparison and analysis submodule is used to compare the associated data of each single mode with the associated data of the multi-mode to determine whether the error exceeds the threshold; The optimization submodule is used to obtain the confidence of the error content when the comparison error exceeds the threshold. When the confidence meets the requirements, the single-modal associated data is used as supplementary data and the supplementary data is sent to the result generation module for processing.

7. The intelligent medical assistance system based on a multimodal large model according to claim 6 is characterized in that: The display module also includes: The demand analysis submodule is used to obtain detailed identity information from the identity authentication module, classify the identity information, and determine the display requirements according to the identity category; Sensitive information filtering submodule, used to filter sensitive information based on identity information; The result adaptation and optimization submodule is used to adapt and optimize the search results according to display requirements.

8. The intelligent medical assistance system based on a multimodal large model according to claim 3 is characterized in that: The intent categories are also weighted according to the identity categories.

9. The intelligent medical assistance system based on a multimodal large model according to claim 1, characterized in that: The multimodality includes text, image, audio, video, and animation modalities.

10. An intelligent medical assistance method based on a multimodal large model, characterized in that: Applied to an intelligent medical assistance system based on a multimodal large model as described in any one of claims 1-8.

Images