A large model system applied to disease diagnosis
By combining the collaborative scheduling of basic large models and expert models, the problem that multimodal large models cannot handle high-dimensional medical images is solved, and collaborative reasoning and high-accuracy diagnosis of complex diseases are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-26
AI Technical Summary
Existing multimodal large models cannot effectively process 3D or 4D medical image data, resulting in insufficient accuracy in disease diagnosis. Furthermore, independent single-point models cannot achieve collaborative reasoning and processing of complex diseases.
By combining a basic large model with multiple expert models, and through receiving, analyzing, scheduling, and output modules, the system achieves collaborative scheduling of multiple tasks and multiple expert models. It leverages the accuracy of expert models and the processing power of the basic large model to handle complex disease diagnosis tasks.
It enables collaborative reasoning and highly accurate diagnosis of complex diseases, and can process 3D and 4D medical image data, thereby improving the overall accuracy and processing capabilities of disease diagnosis.
Smart Images
Figure CN122290966A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of smart healthcare technology, and in particular to a large model system for disease diagnosis. Background Technology
[0002] Currently, there are disease diagnostic models for various diseases that are designed for specific tasks, and some of these models can achieve high accuracy under controlled conditions. However, these diagnostic models have fundamental limitations in clinical practice. This is because each diagnostic model is essentially an independently operating single-point model, capable of diagnosing only specific diseases and tasks, and unable to achieve collaborative reasoning and processing for complex diseases.
[0003] With the rapid development of large models, the emergence of multimodal large models has made it possible to jointly process visual and textual information. However, existing multimodal large models are mainly developed and validated based on two-dimensional images or single-layer volumetric data, and cannot handle the 3D or even 4D image information that is usually involved in disease diagnosis. Multidimensional image information is beyond the understanding scope of multimodal large models. Therefore, directly applying multimodal large models to disease diagnosis is still immature and cannot achieve a certain level of accuracy. Summary of the Invention
[0004] This invention provides a large-scale model system for disease diagnosis. By combining a large-scale model with task scheduling and analysis capabilities with an expert model adapted for disease diagnosis, it leverages the expert model to ensure accuracy in disease diagnosis while simultaneously enabling collaborative scheduling among multiple tasks and expert models using the basic large-scale model. Therefore, the large-scale model system for disease diagnosis provided by this invention can not only perform collaborative reasoning for complex disease tasks but also guarantee a certain level of accuracy.
[0005] This invention provides a large-scale model system for disease diagnosis, comprising: a basic large-scale model and multiple expert models; wherein, the basic large-scale model includes a receiving module, an analysis module, a scheduling module, and an output module; the receiving module is used to receive task instructions and target medical image data; the analysis module is used to analyze the task instructions and the target medical image data to generate analysis results; the analysis results indicate at least one target expert model corresponding to the task instructions among the multiple expert models and the scheduling order of each target expert model; the scheduling module is used to schedule each target expert model according to the target medical image data and the analysis results, and obtain a first output result output by each target expert model; the output module is used to generate a second output result conforming to the output format of the basic large-scale model based on each of the first output results.
[0006] Optionally, the basic large model is a multimodal large model; and / or, each of the expert models corresponds to a disease diagnosis category.
[0007] Optionally, the disease diagnosis category includes at least one of image segmentation, diagnostic classification, lesion detection, and cross-modal registration; the expert model includes at least one of image segmentation model, diagnostic classification model, lesion detection model, and cross-modal registration model.
[0008] Optionally, for each of the disease diagnosis categories: a general model corresponding to the disease diagnosis category is trained using training data to obtain an expert model corresponding to the disease diagnosis category; wherein, the training data is medical image data from various medical scenarios.
[0009] Optionally, when there are multiple target expert models and the multiple target expert models are scheduled sequentially, the scheduling module is further configured to: determine the first target expert model to be scheduled from the multiple target expert models; use the target medical image data as input to the first target expert model and output a first result; and for the second scheduling and the second target expert model after the second scheduling, repeatedly execute the following process until all target expert models are scheduled: use the target medical image and the first result output by the first target expert model of the previous scheduling as input and output a second result.
[0010] Optionally, the image segmentation model includes a localization module and a segmentation module; wherein, the localization module is used to crop the target medical image data to obtain first image data containing the target image; the segmentation module is used to segment the target image from the first image data to generate a first output result.
[0011] Optionally, the diagnostic classification model includes at least one modality feature extraction module, a cross-modal contrastive learning module, a decision fusion module, and a classification diagnosis module; wherein, the feature extraction module is used to extract features from the medical image data to obtain image features corresponding to each feature extraction module; the cross-modal contrastive learning module is used to calibrate and complement all the image features to obtain fused features; the decision fusion module is used to fuse the various fused features to obtain a total fused feature; and the classification diagnosis module is used to output a first output result based on the total fused feature.
[0012] Optionally, the target medical image data is 3D and / or 4D medical image data.
[0013] Optionally, the output module is further configured to integrate the various first output results to obtain an image result and a text result; and to use the text result and the image result as the second output result.
[0014] Optionally, the analysis module is further configured to determine whether an expert model needs to be invoked based on the task instruction; if so, to perform the step of analyzing the task instruction and the target medical image data until a second output result is generated; if not, to use the output module to generate a third output result corresponding to the task instruction.
[0015] This invention provides a large-scale model system for disease diagnosis. By combining a large-scale model with task scheduling and analysis capabilities with an expert model adapted for disease diagnosis, it can leverage the expert model to ensure the accuracy of disease diagnosis while simultaneously enabling collaborative scheduling among multiple tasks and expert models using the basic large-scale model. Therefore, the large-scale model system for disease diagnosis provided by this invention can not only perform collaborative reasoning for complex tasks but also guarantee a certain level of accuracy. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of the structure of the large model system for disease diagnosis provided by the present invention.
[0018] Explanation of reference numerals in the attached figures: 100 - Large model system; 110 - Basic large model; 111 - Receiving module; 112 - Analysis module; 113 - Scheduling module; 114 - Output module; 120 - Expert model. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0020] Figure 1 This is a schematic diagram of the structure of the large model system for disease diagnosis provided by the present invention, as shown below. Figure 1 As shown, the large model system 100 applied to disease diagnosis includes: The basic large model 110 and multiple expert models 120; among them, The basic large model 110 includes a receiving module 111, an analysis module 112, a scheduling module 113, and an output module 114; The receiving module 111 is used to receive task instructions and target medical image data; The analysis module 112 is used to analyze the task instruction and the target medical image data to generate analysis results; the analysis results indicate at least one target expert model corresponding to the task instruction among the multiple expert models and the scheduling order of each target expert model; The scheduling module 113 is used to schedule each of the target expert models according to the target medical image data and the analysis results, and to obtain the first output result of each target expert model. The output module 114 is used to generate a second output result that conforms to the output format of the basic large model based on each of the first output results.
[0021] In disease diagnosis, most pathological diagnoses require the presentation of scanned images. Therefore, in one optional embodiment, the basic large model is a multimodal large model. Specifically, existing large models can be categorized according to the types of information they can process, such as plain text large models (LLM), multimodal large models (LMM), audio large models (Audio LLM), and video large models (Video LMM). Among these, multimodal large models can simultaneously process text, images, audio, and video. They can speak like human beings, understand images, watch videos, and listen to audio, making them particularly suitable for disease diagnosis that requires simultaneous processing of images and text.
[0022] In this invention, expert model 120 refers to a model capable of performing specific data analysis for a particular type of disease. In an optional embodiment, each expert model corresponds to a specific disease diagnosis category. For example, for cardiovascular diseases, to effectively diagnose them, multi-parameter comprehensive assessment of cardiac anatomy, ventricular function, myocardial perfusion, and tissue composition is typically performed using cardiac magnetic resonance imaging (CMR). Separate expert models 120 can be set up for cardiac anatomy and ventricular function to accurately assess them respectively. Similarly, for other types of diseases such as lung diseases and tumors, separate models can be trained based on common disease types and assessment indicators that reflect pathological characteristics, resulting in multiple expert models 120 suitable for different diseases and data analyses. It is understandable that different types of diseases typically require different testing methods and corresponding indicators. Many diseases have a strong correlation with these testing indicators. Therefore, expert models 120 that are more specific to a particular disease type will generally yield more accurate evaluation results. Conversely, expert models 120 with greater general applicability will have relatively lower accuracy due to their increased applicability. Therefore, in an optional embodiment of this invention, each expert model corresponds to a specific disease diagnosis category, which maximizes the accuracy of each expert model 120.
[0023] In one optional embodiment, the disease diagnosis category includes at least one of image segmentation, diagnostic classification, lesion detection, and cross-modal registration. Image segmentation involves dividing specific regions within a complete medical image. For example, a short-axis movie image, which displays dynamic MRI images acquired in a square along the short axis of the heart, with each slice continuously capturing the entire heartbeat, typically covering from the apex to the base, can be segmented to divide various regions of the heart, such as the left ventricle, right ventricle, left atrium, right atrium, myocardium, valves, and great vessels, facilitating subsequent diagnostic assistance based on the segmentation results. Diagnostic classification refers to identifying potential disease types through analysis of medical images, such as diagnosing infectious diseases, neoplastic diseases, airway diseases, etc., using CT images of the lungs. Lesion detection focuses on detecting lesions, such as masses, hemangiomas, and vascular plaques. Cross-modal registration can be understood as the registration between detection results from different modalities, such as the registration between MR and CT, or the registration between MR and ultrasound results. It is understood that the above-mentioned settings for disease diagnosis categories can be adjusted according to the actual needs of disease diagnosis and are not limited to the four listed above; this invention does not impose specific limitations on them.
[0024] Furthermore, to correspond to the aforementioned disease diagnosis categories, the expert model 120 may also include at least one of an image segmentation model, a diagnostic classification model, a lesion detection model, and a cross-modal registration model. It should be noted that, for each disease diagnosis category: a general model corresponding to the disease diagnosis category is trained using training data to obtain an expert model corresponding to the disease diagnosis category; wherein, the training data consists of medical image data from various medical scenarios. For example, the image segmentation model is trained based on the general model `segment anything`, and the diagnostic classification model is trained based on the general model `detection anything`.
[0025] Specifically, the image segmentation model may further include a localization module and a segmentation module; wherein, the localization module is used to crop the target medical image data to obtain first image data containing the target image; the segmentation module is used to segment the target image from the first image data to generate a first output result.
[0026] For example, taking a cardiac segmentation expert model, a general cardiac segmentation expert model can first be used to coarsely crop the original 3D medical image. The cropping volume space is determined by a random sampling center, and the whole cardiac region of interest is extracted. Then, the cropped whole cardiac region of interest is resampled to a uniform voxel spacing. During the resampling process, the intensity image uses continuous interpolation to preserve signal continuity, and the segmentation labels use nearest neighbor interpolation to maintain boundary integrity. Finally, the cropped and resampled images are processed by min-max normalization and input into a unified U-shaped backbone network, outputting the precise cardiac localization result (i.e., the first image data containing the target image).
[0027] The cardiac segmentation expert model is a customized network architecture designed to address the dynamic characteristics and anatomical complexity of cardiac images. It employs a collaborative optimization paradigm of localization and segmentation. First, the localization module identifies the entire heart's Region of Interest (ROI), defining the effective range for subsequent segmentation tasks. Then, the entire ROI is input into the segmentation module, achieving pixel-level precision segmentation of key anatomical structures such as the left ventricle, right ventricle, and myocardium, resulting in multiple segmented cardiac regions. To further enhance the segmentation robustness in complex clinical scenarios, the cardiac segmentation expert model incorporates a dual attention mechanism: motion feature enhancement attention and anatomical structure adaptive attention. Motion feature enhancement attention uses optical flow field estimation and spatiotemporal feature fusion technology to accurately capture the motion trajectory and deformation patterns of the heart during the cardiac cycle, simultaneously achieving preliminary localization of abnormal motion regions. Anatomical structure adaptive attention introduces a normal cardiac anatomical template as prior knowledge and constructs an anatomical structure similarity measurement space through a comparative learning mechanism. This adaptively identifies abnormal anatomical regions that deviate from the normal cardiac anatomical template and applies additional supervisory constraints to these abnormal anatomical regions, forcing the cardiac segmentation expert model to focus on structural variation details. This significantly improves the reliability of cardiac structure segmentation under pathological conditions while ensuring overall segmentation accuracy. The cardiac segmentation expert model can accurately identify and delineate cardiac chambers, myocardium, and lesion areas. Lesion areas include areas of abnormal motion and abnormal anatomy. Finally, multiple cardiac segmentation areas are input into the calculation module to obtain the functional parameter values corresponding to each cardiac segmentation area output by the calculation module. For example, functional parameter values include ejection fraction (EF), end-diastolic volume (EDV), end-systolic volume (ESV), and left ventricular myocardial mass (LVM). In addition, it can also identify and quantify scar or fibrotic areas in LGE images and areas with insufficient blood perfusion in perfusion images.
[0028] In one optional embodiment, the diagnostic classification model includes at least one modality feature extraction module, a cross-modal contrastive learning module, a decision fusion module, and a classification diagnosis module; wherein, the feature extraction module is used to extract features from medical image data to obtain image features corresponding to each feature extraction module; the cross-modal contrastive learning module is used to calibrate and complement all image features to obtain fused features; the decision fusion module is used to fuse the various fused features to obtain a total fused feature; and the classification diagnosis module is used to output a first output result based on the total fused feature.
[0029] For example, taking a cardiac diagnostic classification model, it is built upon the segmentation results of a cardiac segmentation expert model and cardiac imaging features, employing a multi-level classification strategy. The first-level classification determines whether the heart is normal or abnormal, while the second-level classification determines whether the abnormal heart belongs to ischemic cardiomyopathy or non-ischemic cardiomyopathy, etc. These two classification stages can use a Convolutional Neural Network (CNN) to compress the multimodal features, and then the integrated multimodal features are input into a Transformer model for final diagnosis. The cardiac diagnostic classification model proposes a two-level progressive diagnostic framework of coarse screening and fine classification, constructing a full-process decision-making system from macroscopic disease identification to microscopic subtype grading. In the first stage of coarse screening, a multi-pathway feature contrast learning network is designed for multimodal cardiac images. This network includes at least one modality feature extraction module, a cross-modality contrast learning module, a decision fusion module, and a classification and diagnosis module. The modality feature extraction module extracts features from the cardiac images of each modality to obtain the image features for that modality. Then, the image features from each modality are input into the cross-modality contrast learning module to calibrate and complement the image features of all modalities, resulting in calibrated and complemented fused features. These calibrated and complemented fused features are then input into the decision fusion module to obtain the total fused features output by the decision fusion module. Finally, the total fused features and the analysis results of the cardiac segmentation expert model are input into the classification and diagnosis module to obtain the classification and diagnosis results output by the classification and diagnosis module. This ultimately completes the three-category differentiation of normal heart, ischemic heart disease, and non-ischemic heart disease. For non-ischemic heart disease, it can be further subdivided into subtypes such as dilated cardiomyopathy, hypertrophic cardiomyopathy, myocarditis, amyloidosis cardiomyopathy, and sarcoidosis cardiomyopathy, achieving accurate pathological feature identification and assisted diagnosis.
[0030] In this embodiment of the invention, multiple expert models are each provided with an independent API. Once the target expert model is determined, it is only necessary to call the API corresponding to the target expert model to invoke the target expert model.
[0031] Furthermore, as described in the background section of this invention, existing multimodal large models can process simple single-layer volumetric data or two-dimensional image data. Therefore, in this embodiment of the invention, the target medical image data is 3D and / or 4D medical image data. In other words, this embodiment of the invention is particularly capable of processing high-dimensional medical image data that cannot be processed by existing technologies, thereby achieving technical effects that cannot be achieved by existing technologies.
[0032] As can be seen, by setting up a basic large model and various expert models corresponding to different disease diagnosis categories, the embodiments of the present invention can schedule multiple expert models according to task instructions. This not only enables the use of the powerful processing capabilities of the basic large model to handle complex tasks, but also ensures the accuracy of disease diagnosis by utilizing the expert models.
[0033] In an optional embodiment, the analysis module 112 is further configured to determine whether an expert model needs to be invoked based on the task instruction; if so, to perform the step of analyzing the task instruction and the target medical image data until a second output result is generated; if not, to use the output module 114 to generate a third output result corresponding to the task instruction. In this invention, the expert model 120 is mainly set for disease diagnosis categories that require medical image data analysis or require image output. For simple text question-and-answer or simple text analysis that does not require medical image data analysis, the multimodal large model can handle the task independently. Therefore, in this embodiment of the invention, after receiving the task instruction and the target medical image data, the multimodal large model will first make a judgment based on the task instruction to determine the subsequent implementation method.
[0034] The following provides a detailed explanation of various scheduling scenarios: (a) The target expert model is one In this case, after receiving the task instruction, the basic large model 110 only needs to analyze whether the target expert model needs to be called. If so, the interaction process with the target expert model is realized through the receiving module 111, the analysis module 112 and the scheduling module 113 in sequence, and finally the second output result is output through the output module 114.
[0035] (ii) The target expert model consists of multiple To address this situation, upon receiving a task instruction, the basic large-scale model 110 needs to analyze not only whether to invoke the target expert model, but also which target expert models to invoke and in what order. Specifically, after receiving the user-input task instruction, the instruction can first undergo text cleaning and standardization. For text-based instructions, text cleaning includes removing redundant spaces, standardizing punctuation, and converting between simplified and traditional Chinese characters. For speech-based instructions, speech recognition errors need to be corrected, and spell checking and error correction can be performed using a medical terminology dictionary. Specifically, for different types of diseases, dictionaries containing professional terminology can be established, including disease names, anatomical structure names, imaging sequence names, functional parameter names, etc., with a dictionary size exceeding ten thousand entries.
[0036] After cleaning and standardization, a basic large model 110 is needed to perform semantic understanding of the task instructions. Specifically, a Transformer-based visual language large model can be used to parse the task execution instructions, helping the machine understand the user's goal, and further, based on the goal, obtain the required multiple target expert models and the order in which they are called.
[0037] In an optional embodiment, the scheduling module 113 is further configured to: determine a first target expert model for scheduling from multiple target expert models; use the target medical image data as input to the first target expert model and output a first result; and for the second scheduling and subsequent second target expert models, repeatedly execute the following process until all target expert models are scheduled: use the target medical image and the first result output by the previously scheduled first target expert model as input and output a second result. That is, for multiple target expert models executed sequentially, the output of the previously scheduled target expert model is typically used as input for the subsequently scheduled target expert model to achieve a step-by-step calculation, with the final second output result obtained from the last scheduled target expert model.
[0038] In another optional embodiment, a specific execution plan can be formulated based on the analysis results. This involves assigning a unique task identifier to each target expert model and adding the target tasks corresponding to each model to an execution queue. The advantage of setting up an execution queue is that it supports both batch inference of expert models (processing multiple inputs at once to improve throughput) and asynchronous execution strategies, using message queues to distribute tasks and collect results. When a target expert model completes its task, it sends the analysis results to the results queue and simultaneously triggers downstream tasks that depend on those results.
[0039] In summary, the embodiments of the present invention are particularly capable of sequentially or asynchronously calling multiple target expert models when dealing with complex task instructions, and using the called target expert models to achieve corresponding disease diagnosis functions.
[0040] In addition, the large model system for disease diagnosis provided in this embodiment of the invention may also include a model manager for maintaining the registration information and running status of all expert models, so as to facilitate the management and maintenance of expert models.
[0041] The large-scale model system for disease diagnosis provided in this invention combines a large-scale model with task scheduling and analysis capabilities with an expert model adapted for disease diagnosis. This approach leverages the expert model to ensure accuracy in disease diagnosis while simultaneously enabling collaborative scheduling among multiple tasks and expert models using the basic large-scale model. Therefore, the large-scale model system for disease diagnosis provided in this invention can perform collaborative reasoning for complex tasks while maintaining a certain level of accuracy.
[0042] The complete process of the large-scale model system for disease diagnosis provided by this invention will be described below: All the above modules are integrated into a unified end-to-end process. The process begins with the user inputting a task instruction and target medical image data through the interactive interface. After receiving the task instruction and target medical image data, the receiving module 111 packages them together and sends them to the analysis module 112. The analysis module 112 first determines whether an expert model needs to be invoked based on the task instruction. If so, it further identifies and analyzes the target medical image data to determine the actual target expert model to be invoked to complete the task instruction and the invocation order of the target expert models (i.e., the analysis result), and sends the analysis result to the scheduling module 113. After receiving the analysis result, the scheduling module 113 invokes the target expert models sequentially according to the invocation order. During the invocation process, it sequentially obtains the output result of each target expert model (i.e., the first output result) and stores the first output result. After obtaining the first output results corresponding to all the target expert models, the system considers the entire invocation process complete. At this point, the output module 114 generates a second output result based on each first output result. The second output result can be a diagnostic report that integrates all the first output results. Each first output result can be displayed on the interactive interface in a graphic and textual format for user viewing. If not, the task instruction is directly sent to the output module 114, which independently considers and analyzes the task instruction and the target medical image data to output a third output result.
[0043] The large-scale model system for disease diagnosis provided in this invention combines a large-scale model with task scheduling and analysis capabilities with an expert model adapted for disease diagnosis. This approach leverages the expert model to ensure accuracy in disease diagnosis while simultaneously enabling collaborative scheduling among multiple tasks and expert models using the basic large-scale model. Therefore, the large-scale model system for disease diagnosis provided in this invention can perform collaborative reasoning for complex tasks while maintaining a certain level of accuracy.
[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A large-scale model system for disease diagnosis, characterized in that, include: The basic large model and multiple expert models; among them, The basic large model includes a receiving module, an analysis module, a scheduling module, and an output module; The receiving module is used to receive task instructions and target medical image data; The analysis module is used to analyze the task instruction and the target medical image data to generate analysis results; the analysis results indicate at least one target expert model corresponding to the task instruction among the multiple expert models and the scheduling order of each target expert model; The scheduling module is used to schedule each of the target expert models according to the target medical image data and the analysis results, and to obtain the first output result of each target expert model. The output module is used to generate a second output result that conforms to the output format of the basic large model based on each of the first output results.
2. The large-scale model system for disease diagnosis according to claim 1, characterized in that, The basic large model is a multimodal large model; And / or, Each of the expert models corresponds to a disease diagnosis category.
3. The large-scale model system for disease diagnosis according to claim 2, characterized in that, The disease diagnosis categories include at least one of image segmentation, diagnostic classification, lesion detection, and cross-modal registration; The expert model includes at least one of the following: image segmentation model, diagnostic classification model, lesion detection model, and cross-modal registration model.
4. The large-scale model system for disease diagnosis according to claim 2, characterized in that, For each of the disease diagnosis categories: a general model corresponding to the disease diagnosis category is trained using training data to obtain an expert model corresponding to the disease diagnosis category; The training data consists of medical image data from various medical scenarios.
5. The large-scale model system for disease diagnosis according to claim 2, characterized in that, When there are multiple target expert models, and these multiple target expert models are scheduled sequentially, the scheduling module is further configured to: The first target expert model for scheduling is determined from the multiple target expert models; The target medical image data is used as input to the first target expert model, and the first result is output. For the second scheduler and the second target expert model after the second scheduler, repeat the following process until all target expert models have been scheduled: The target medical image and the first result output by the first target expert model of the previous scheduling are used as inputs to obtain the second result.
6. The large-scale model system for disease diagnosis according to claim 3, characterized in that, The image segmentation model includes a localization module and a segmentation module; wherein... The positioning module is used to crop the target medical image data to obtain first image data containing the target image; The segmentation module is used to segment the target image from the first image data and generate a first output result.
7. The large-scale model system for disease diagnosis according to claim 3, characterized in that, The diagnostic classification model includes at least one modality feature extraction module, a cross-modal contrastive learning module, a decision fusion module, and a classification diagnosis module; wherein... The feature extraction module is used to extract features from the medical image data to obtain image features corresponding to each feature extraction module; The cross-modal contrastive learning module is used to calibrate and complement all the image features to obtain fused features; The decision fusion module is used to fuse the various fusion features to obtain the total fusion feature; The classification and diagnosis module is used to output a first output result based on the total fusion features.
8. The large-scale model system for disease diagnosis according to claim 1, characterized in that, The target medical image data is 3D and / or 4D medical image data.
9. The large-scale model system for disease diagnosis according to claim 1, characterized in that, The output module is further configured to integrate the first output results to obtain image results and text results; The text result and the image result are used as the second output result.
10. The large-scale model system for disease diagnosis according to claim 1, characterized in that, The analysis module is also used to determine whether an expert model needs to be invoked based on the task instructions. If so, perform the step of analyzing the task instructions and the target medical image data until a second output result is generated; If not, the output module is used to generate a third output result corresponding to the task instruction.