Medical image analysis system based on multi-modal large model

Abstract

The invention relates to medical image analysis, in particular to a medical image analysis system based on a multi-modal large model, and the system comprises a multi-modal feature extractor which is used for extracting visual feature information of a medical image in a current case and text feature information of patient information in the current case; fusing the visual feature information and the text feature information to obtain multi-modal features of the current case; the multi-modal RAG module is used for respectively retrieving specialized clinical knowledge and typical case data related to the current case from a specialized clinical database and a typical case knowledge base according to the multi-modal characteristics of the current case, and sending the specialized clinical knowledge and the typical case data to the medical image multi-modal large model as retrieval enhancement information; the medical image multi-modal large model is used for providing a medical image analysis result and related services thereof according to the multi-modal features of the current case, retrieval enhancement information and feedback and correction suggestions; according to the technical scheme provided by the invention, the defect that the medical image is difficult to accurately and efficiently analyze in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to medical image analysis, and in particular to a medical image analysis system based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence technology, the medical field has gradually introduced and applied artificial intelligence technology to improve the accuracy and efficiency of diagnosis and treatment. The existing medical image analysis systems mainly include single-modality / multi-modality fusion medical image analysis systems, single-modality / multi-modality RAG medical image analysis systems, small-model single-task medical image analysis systems, and large-model multi-task medical image analysis systems.

[0003] The above existing medical image analysis systems mainly have the following problems:

[0004] 1) There is a lack of medical images formed by all imaging modalities including X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscope, and multimodal data of medical images with corresponding text data, which makes it difficult to apply to a wide range of medical images;

[0005] 2) The multimodal RAG module lacks a specialized clinical database and a typical case knowledge base, which cannot provide professional clinical knowledge and imaging support for the system;

[0006] 3) It lacks extensive patient-oriented and doctor-oriented functions, making it difficult to universally apply to multimodal medical imaging scenarios;

[0007] 4) The lack of feedback and correction suggestions from experts makes it difficult to ensure the accuracy of medical image analysis results, and it is impossible to iterate and update professional knowledge. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the above-mentioned shortcomings of the prior art, the present invention provides a medical image analysis system based on a multimodal large model, which can effectively overcome the defect of the prior art that it is difficult to accurately and efficiently analyze medical images.

[0010] (II) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0012] Medical image analysis system based on multimodal large model, including specialized clinical database and typical case knowledge base, multimodal feature extractor, multimodal RAG module, medical image multimodal large model, expert feedback and correction module;

[0013] The multimodal feature extractor extracts the visual feature information of the medical image in the current case and the text feature information of the patient information in the current case, fuses the visual feature information and the text feature information to obtain the multimodal features of the current case, and sends them to the multimodal RAG module and the medical image multimodal large model;

[0014] The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and typical case knowledge base according to the multimodal characteristics of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information;

[0015] The multimodal large model of medical imaging provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal characteristics of the current case, retrieval enhancement information, and feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation;

[0016] The expert feedback and correction module provides feedback and correction suggestions from experts on the information stored in the specialist clinical database and typical case knowledge base, as well as the output information of the medical imaging multimodal large model, based on the medical images and patient information in the current case, so as to dynamically update the specialist clinical database and typical case knowledge base and improve the accuracy of the medical imaging multimodal large model.

[0017] Preferably, the specialist clinical database and typical case knowledge base are constructed based on multimodal medical imaging data;

[0018] Among them, multimodal medical imaging data include massive medical images formed by all imaging modalities including X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscopy, and corresponding text data.

[0019] Preferably, the multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base respectively according to the multimodal features of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information, including:

[0020] According to the multimodal characteristics of the current case, the multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base through knowledge retrieval based on keywords or vector similarity, image retrieval based on vector similarity, and retrieval based on a neural network model, and sends them to the medical imaging multimodal large model as retrieval enhancement information.

[0021] Preferably, the medical imaging multimodal large model provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal features of the current case, retrieval enhancement information, and feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation, including:

[0022] Preventive health care: Provides information on the causes of various diseases, clinical prevention, and suggestions for improving health status;

[0023] Similar case recommendation: recommend similar or typical cases for the current case to better analyze the current case;

[0024] Interpretation of imaging data: providing interpretation and analysis of medical images, diagnostic reports, examination reports and test reports in various scenarios;

[0025] Medical image analysis: analysis of cell, tissue, organ, system conditions and related diseases, as well as lesion location and analysis report generation;

[0026] Treatment planning: providing examinations and tests, medication, physical therapy, psychotherapy, surgical treatment, nutritional guidance and rehabilitation plans for specific diseases or health problems;

[0027] Prognostic analysis: Provide personalized rehabilitation plans, disease risk assessment and lifestyle guidance based on disease severity, health status and treatment effects;

[0028] Intelligent consultation: For specific cases or difficult and complicated diseases, the multimodal large model of medical imaging is used to analyze and discuss the patient's imaging examination results with the doctor to obtain more accurate medical imaging analysis results and treatment recommendations.

[0029] (III) Beneficial effects

[0030] Compared with the prior art, the medical image analysis system based on multimodal large model provided by the present invention has the following beneficial effects:

[0031] 1) The multimodal feature extractor extracts the visual feature information of the medical image in the current case and the text feature information of the patient information in the current case, and fuses the visual feature information and the text feature information to obtain the multimodal features of the current case. The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base according to the multimodal features of the current case, and sends them to the medical image multimodal big model as retrieval enhancement information. The medical image multimodal big model assists doctors in performing medical image analysis based on the multimodal features of the current case, retrieval enhancement information, and feedback and correction suggestions, so as to accurately and efficiently analyze the medical images and obtain accurate medical image analysis results;

[0032] 2) Make full use of the advantages of specialized clinical databases and typical case knowledge bases, medical imaging multimodal large models, and professional doctors to conduct a comprehensive analysis of the multimodal characteristics of the current case, provide preventive health care, similar case recommendations, and imaging data interpretation services, and assist doctors in medical image analysis, treatment plan formulation, prognosis analysis, and intelligent consultation. At the same time, it can iteratively update professional knowledge based on feedback and correction suggestions provided by experts, which is helpful to promote the intelligent development of medical image analysis systems. The medical image analysis system provided by the present invention can be applied to the fields of medical image analysis and multimodal large models, which helps patients and doctors obtain more professional medical image analysis results and related services. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a system schematic diagram of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Medical image analysis systems based on multimodal large models, such as Figure 1 As shown, it includes a specialist clinical database and a typical case knowledge base, a multimodal feature extractor, a multimodal RAG (Retrieval Augmented Generation) module, a medical imaging multimodal large model, and an expert feedback and correction module;

[0037] The multimodal feature extractor extracts the visual feature information of the medical image in the current case and the text feature information of the patient information in the current case, fuses the visual feature information and the text feature information to obtain the multimodal features of the current case, and sends them to the multimodal RAG module and the medical image multimodal large model;

[0038] The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and typical case knowledge base according to the multimodal characteristics of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information;

[0039] The multimodal large model of medical imaging provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal characteristics of the current case, retrieval enhancement information, and feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation;

[0040] The expert feedback and correction module provides feedback and correction suggestions from experts on the information stored in the specialist clinical database and typical case knowledge base, as well as the output information of the medical imaging multimodal large model, based on the medical images and patient information in the current case, so as to dynamically update the specialist clinical database and typical case knowledge base and improve the accuracy of the medical imaging multimodal large model.

[0041] ① In the technical solution of this application, the specialist clinical database and typical case knowledge base are constructed based on multimodal medical imaging data;

[0042] Among them, multimodal medical imaging data include massive medical images formed by all imaging modalities including X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscopy, and corresponding text data.

[0043] ② The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and typical case knowledge base according to the multimodal characteristics of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information, including:

[0044] According to the multimodal characteristics of the current case, the multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base through knowledge retrieval based on keywords or vector similarity, image retrieval based on vector similarity, and retrieval based on a neural network model, and sends them to the medical imaging multimodal large model as retrieval enhancement information.

[0045] ③ The multimodal large model of medical imaging provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal characteristics of the current case, retrieval enhancement information, feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation, including:

[0046] Preventive health care: Provides information on the causes of various diseases, clinical prevention, and suggestions for improving health status;

[0047] Similar case recommendation: recommend similar or typical cases for the current case to better analyze the current case;

[0048] Interpretation of imaging data: providing interpretation and analysis of medical images, diagnostic reports, examination reports and test reports in various scenarios;

[0049] Medical image analysis: analysis of cell, tissue, organ, system conditions and related diseases, as well as lesion location and analysis report generation;

[0050] Treatment planning: providing examinations and tests, medication, physical therapy, psychotherapy, surgical treatment, nutritional guidance and rehabilitation plans for specific diseases or health problems;

[0051] Prognostic analysis: Provide personalized rehabilitation plans, disease risk assessment and lifestyle guidance based on disease severity, health status and treatment effects;

[0052] Intelligent consultation: For specific cases or difficult and complicated diseases, the multimodal large model of medical imaging is used to analyze and discuss the patient's imaging examination results with the doctor to obtain more accurate medical imaging analysis results and treatment recommendations.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image analysis system based on a multimodal large model, characterized by: It includes specialized clinical database and typical case knowledge base, multimodal feature extractor, multimodal RAG module, multimodal large model of medical imaging, and expert feedback and correction module; The multimodal feature extractor extracts the visual feature information of the medical image in the current case and the text feature information of the patient information in the current case, fuses the visual feature information and the text feature information to obtain the multimodal features of the current case, and sends them to the multimodal RAG module and the medical image multimodal large model; The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and typical case knowledge base according to the multimodal characteristics of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information; The multimodal large model of medical imaging provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal characteristics of the current case, retrieval enhancement information, and feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation; The expert feedback and correction module provides feedback and correction suggestions from experts on the information stored in the specialist clinical database and typical case knowledge base, as well as the output information of the medical imaging multimodal large model, based on the medical images and patient information in the current case, so as to dynamically update the specialist clinical database and typical case knowledge base and improve the accuracy of the medical imaging multimodal large model.

2. The medical image analysis system based on multimodal large model according to claim 1, characterized in that: The specialist clinical database and typical case knowledge base are constructed based on multimodal medical imaging data; Among them, multimodal medical imaging data include massive medical images formed by all imaging modalities including X-ray, CT, MRI, ultrasound, pathology, PET, endoscopy, digital subtraction angiography, dermatoscopy, and corresponding text data.

3. The medical image analysis system based on multimodal large model according to claim 1, characterized in that: The multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base according to the multimodal features of the current case, and sends them to the medical imaging multimodal large model as retrieval enhancement information, including: According to the multimodal characteristics of the current case, the multimodal RAG module retrieves the specialist clinical knowledge and typical case data related to the current case from the specialist clinical database and the typical case knowledge base through knowledge retrieval based on keywords or vector similarity, image retrieval based on vector similarity, and retrieval based on a neural network model, and sends them to the medical imaging multimodal large model as retrieval enhancement information.

4. The medical image analysis system based on multimodal large model according to claim 1, characterized in that: The multimodal large model of medical imaging provides preventive care, similar case recommendations and imaging data interpretation services based on the multimodal features of the current case, retrieval enhancement information, and feedback and correction suggestions, and assists doctors in medical imaging analysis, diagnosis and treatment plan formulation, prognosis analysis and intelligent consultation, including: Preventive health care: Provides information on the causes of various diseases, clinical prevention, and suggestions for improving health status; Similar case recommendation: recommend similar or typical cases for the current case to better analyze the current case; Interpretation of imaging data: providing interpretation and analysis of medical images, diagnostic reports, examination reports and test reports in various scenarios; Medical image analysis: analysis of cell, tissue, organ, system conditions and related diseases, as well as lesion location and analysis report generation; Treatment planning: providing examinations and tests, medication, physical therapy, psychotherapy, surgical treatment, nutritional guidance and rehabilitation plans for specific diseases or health problems; Prognostic analysis: Provide personalized rehabilitation plans, disease risk assessment and lifestyle guidance based on disease severity, health status and treatment effects; Intelligent consultation: For specific cases or difficult and complicated diseases, the multimodal large model of medical imaging is used to analyze and discuss the patient's imaging examination results with the doctor to obtain more accurate medical imaging analysis results and treatment recommendations.

Images