An intelligent medical image assisted diagnosis system based on multi-modal fusion

Abstract

The application discloses an intelligent medical image auxiliary diagnosis system based on multi-modal fusion, relates to the technical field of medical image processing, and is suitable for image recognition and diagnosis evaluation of diseases such as lung nodules, brain tumors and breast lesions. The technical key points are that the system comprises an image acquisition module, an image preprocessing module, a multi-modal feature extraction and alignment module, a fusion decision module and a man-machine interaction module. The system can perform non-rigid registration and unified formatting processing on medical images of different modalities such as CT, MRI, PET and ultrasound, extract the spatial structure and functional characteristics of each modality, and realize automatic segmentation and classification of the lesion area through a deep fusion network. The system supports doctors to visually correct the diagnosis results, and the feedback information can be used for continuous training and optimization of the model. The system has the advantages of high fusion precision, fast diagnosis efficiency and strong adaptability, and is suitable for clinical auxiliary diagnosis applications in multiple diseases and multiple scenes.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to an intelligent medical image-assisted diagnostic system based on multimodal fusion. Background Technology

[0002] With the development of medical imaging technology, various imaging modalities are now widely used in clinical practice for disease diagnosis and assessment, including computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and ultrasound (US). Each of these imaging modalities has its own characteristics; for example, CT provides high clarity in depicting bony structures, MRI excels in soft tissue resolution, and PET can provide metabolic information and reflect tissue activity. These modalities are highly complementary; therefore, doctors often need to integrate information from multiple modalities to make informed decisions in actual diagnosis and treatment.

[0003] However, in current clinical practice, the analysis of multimodal imaging data still heavily relies on the experience and subjective judgment of physicians. Physicians need to manually compare information such as spatial alignment, lesion morphology, and signal intensity changes across different modalities. This process is not only time-consuming and labor-intensive but also limited by human visual perception, easily leading to missed diagnoses and misjudgments. Especially in typical scenarios such as small lung nodules, brain gliomas, and early breast lesions, due to the small size or blurred morphology of the lesions, a single modality often cannot provide sufficient diagnostic evidence.

[0004] In recent years, medical image-assisted diagnostic methods based on artificial intelligence (AI) and deep learning technologies have developed rapidly and achieved remarkable results. Convolutional neural networks (CNNs) have been widely used in tasks such as medical image classification, segmentation, and detection, and have achieved results comparable to or even exceeding those of junior doctors on some single-modal images. However, most current AI medical imaging models still mainly rely on single-modal images for training and prediction. This approach ignores the complementary information between different modalities, limiting the model's generalization ability and diagnostic accuracy.

[0005] Furthermore, existing multimodal fusion methods still face several technical challenges that require breakthroughs. First, different modal images differ in acquisition time, scanning posture, and resolution; direct stitching or overlaying can lead to information distortion, necessitating high-precision non-rigid registration algorithms for spatial alignment. Second, multimodal images exhibit semantic heterogeneity; effectively modeling the correlations between modalities at the feature layer is crucial for improving fusion performance. Third, clinical applications also require consideration of the interpretability of the diagnostic process, the visualization of results, and mechanisms for physician feedback and correction; however, most existing systems lack robust human-computer interaction design.

[0006] Therefore, we urgently need to design an intelligent medical image-assisted diagnosis system based on multimodal fusion to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing an intelligent medical image-assisted diagnostic system based on multimodal fusion, thereby solving the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: an intelligent medical image-assisted diagnostic system based on multimodal fusion, comprising: an image acquisition module for acquiring multimodal medical image data of a target patient, denoted as... ,in: Indicates the total number of modes; Indicates the first The original images of each modality (such as CT, MRI, PET, ultrasound); The image preprocessing module is used to process images of various modalities. Preprocessing is performed to obtain images in a uniform format. ,in: The image is processed by resolution matching, non-rigid registration, noise suppression, and normalization. The feature extraction and alignment module is used to align each preprocessed image Input convolutional neural network to extract feature maps And aligned to attention mechanism ,in: , , These are the height, width, and number of channels of the feature map, respectively. Representing modes The original feature map; Representing modes Aligned feature map; The fusion decision module aligns all feature maps. Input to the fusion network, output: Lesion segmentation mask , where 1 represents the lesion area; Binary classification prediction results , where 0 represents benign and 1 represents malignant; The human-computer interaction module is used to display the above results. , It also supports doctor feedback to correct labels. , ,in: , These are the segmentation and classification results manually corrected by the doctor, used for training data augmentation.

[0009] As a preferred technical solution of the present invention, in the image preprocessing module, any two modal images , Spatial alignment between them is based on maximizing mutual information, and the calculation formula is as follows: ; in: , Indicates the first registration , Modal images; The pixel intensity values ​​are respectively , Joint probability distribution at time; , The marginal probability distributions for each mode are given.

[0010] As a preferred technical solution of the present invention, modal feature extraction employs a multi-channel convolutional network, and the processing procedure is as follows: ; in: Indicates that it is dedicated to the first A convolutional neural network structure with multiple modalities; For the extracted first Feature maps of each modality; , For space dimensions, This represents the number of channels.

[0011] As a preferred technical solution of the present invention, feature alignment adopts cross-modal mapping based on an attention mechanism, and the calculation formula is as follows: ; in: For the first Input feature map of the modality; , , These are query, key, and value vectors, respectively. This is the weight matrix; The scaling factor for the feature dimension; Representing modes Aligned feature map.

[0012] As a preferred technical solution of the present invention, the fusion decision module adopts... A multi-layer Transformer structure outputs a globally fused feature vector. ,in: Indicates the number of stacked Transformer layers; The fused feature vector is fed into the classifier to produce the output. And decoded into a segmentation mask by the decoder. .

[0013] As a preferred technical solution of the present invention, the loss function during system training... The weighted sum of segmentation loss and classification loss: ; in: , The segmentation mask and classification results output by the system; , These are genuine labels manually annotated. Loss due to Dice or IoU; The cross-entropy loss function; These are the weighting coefficients.

[0014] As a preferred technical solution of the present invention, the human-computer interaction module includes: a visualization display submodule, used to display multimodal original images. Segmentation mask with model output Overlay displays generate a visual heatmap. ,in This indicates an image overlay operation; The diagnostic confidence output submodule is used to display the classification results. Corresponding probability value This indicates the system's confidence in the "malicious" label; The feedback acquisition submodule is used to receive the feedback mask edited by the doctor. With category correction tags And use it to update the training dataset; The model optimization interface is used to process feedback data. Incremental model training with the original feature input improves the system's prediction accuracy.

[0015] As a preferred embodiment of the present invention, the image acquisition module interfaces with the hospital's PACS system via the DICOM protocol, supporting the extraction of patient identifiers from DICOM files. Image sequence numbering Information such as image location index idx; Uniform formatted output as a standard image stack ,in: Indicates the first Number of slices in a modal image sequence; Indicates the first The first modality Image slices.

[0016] As a preferred technical solution of the present invention, the system is adapted to multimodal fusion configurations for different diseases, wherein: for lung lesion detection, the modal combination used is CT and PET, and the corresponding number of modalities is... The input image is ; For brain tumor identification, the modalities used are MRI and functional MRI (fMRI), denoted as { ; For breast lesion screening, the modalities used are MRI and ultrasound images, denoted as { The system automatically loads a predefined set of network structure parameters based on the disease type. This includes the number of network layers, fusion method, and training hyperparameters.

[0017] As a preferred technical solution of the present invention, the system adopts a containerized deployment method and has the following operational capabilities: the system is encapsulated in a Docker container and has cross-platform deployment capabilities; Supports GPU-accelerated inference, using a tensor computation framework for feature tensors. With fusion results Perform efficient processing; The system provides an open RESTful API for receiving image upload requests. Return diagnostic results Receive doctor feedback Supports remote model update functionality, dynamically loading the latest parameter and weight set via the model version identifier ModelID. .

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a complete multimodal medical image-assisted diagnostic system, achieving high-precision acquisition and non-rigid registration of various image data such as CT, MRI, PET, and ultrasound, effectively solving the problems of time-consuming and inaccurate traditional manual alignment processes. The system employs an image preprocessing algorithm based on mutual information, which can adaptively process images of different resolutions, scales, and orientations, ensuring spatial consistency among images of different modalities. This provides an accurate input foundation for subsequent feature fusion, improving the automation level and stability of pre-diagnostic image processing.

[0019] 2. The multimodal feature extraction and fusion mechanism proposed in this invention combines convolutional neural networks and Transformer structures to construct a deep fusion model with cross-modal semantic perception capabilities. The system aligns and fuses feature maps from different modalities through an attention mechanism, effectively utilizing the complementary information of structural and functional imaging to achieve accurate segmentation and benign / malignant determination of lesion regions. The output of the fusion network simultaneously includes mask segmentation and classification labels, supports multi-task learning, and improves the system's accuracy and diagnostic consistency in identifying complex lesions (such as lung nodules, brain tumors, and breast lesions).

[0020] 3. This invention integrates human-computer interaction and intelligent feedback modules. Doctors can visually view and make real-time corrections based on the results generated by the system. The system automatically records doctor feedback, forming a training closed loop. The system supports remote model hot updates and has Docker containerization deployment capabilities, making it suitable for rapid integration with local hospital servers or cloud platforms. Compared to traditional single-modal AI systems, this invention significantly improves model adaptability, diagnostic stability, and clinical usability, possessing broad application prospects and industrialization value. Attached Figure Description

[0021] Figure 1 This is a system block diagram of an intelligent medical image-assisted diagnostic system based on multimodal fusion proposed in this invention; Figure 2 This is a diagram showing the composition of the human-computer feedback module of an intelligent medical image-assisted diagnostic system based on multimodal fusion proposed in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See Figure 1 and Figure 2 As shown, the intelligent medical image-assisted diagnostic system based on multimodal fusion provided by this invention integrates multiple medical imaging modalities such as CT, MRI, PET, and ultrasound, solving the problems of insufficient information, registration difficulties, and unstable judgment in existing single-modal diagnostics. This system integrates modules such as image acquisition, preprocessing, multimodal feature extraction and alignment, deep fusion inference, human-computer interaction feedback, and remote deployment, constructing a scalable, iterative, and multi-disease-oriented AI-assisted diagnostic platform. To better illustrate the technical implementation of this system, several embodiments are described below.

[0024] Example 1: Assisted diagnosis of pulmonary nodules (CT+PET); This embodiment is used to detect and evaluate the benign or malignant nature of small pulmonary nodules, and the image modalities used are low-dose chest CT images and PET metabolic images.

[0025] The system first extracts the DICOM image sequence corresponding to the patient from the PACS system to obtain the raw image data. ,in: Chest CT images; Chest PET image; : Number of modes.

[0026] The image preprocessing module then performs the following operations: Image size normalized to 2. Align CT and PET images using a non-rigid registration algorithm based on mutual information: ; in For the processed image, Represents the joint distribution of pixel values. It is distributed at the edge.

[0027] This registration algorithm achieves an average spatial error of <2 pixels on standard lung field images. The aligned images are then fed into a modality feature extraction network, where a pre-trained ResNet variant is used for convolutional feature mapping to obtain feature maps. ; To unify the feature semantic space, a cross-modal alignment network based on an attention mechanism is adopted, and the calculation is as follows: ; in For the reason The query, key, and value vectors obtained through linear transformation. This is the channel dimension scaling factor. Alignment feature maps of the two modalities. They are spliced ​​together and input into a multimodal fusion Transformer network.

[0028] The network contains Layer encoders are stacked, and a multi-head cross-attention mechanism is used to process the fused information, ultimately producing a lesion segmentation mask at the output. ; Nodule classification prediction results , where 1 indicates high risk of malignancy.

[0029] During system training, the following joint loss function is used: ; in, , These are masks and classification labels manually marked by doctors. For Dice's loss, Cross-entropy loss function These are the weight parameters.

[0030] The diagnostic results are displayed via the human-computer interaction module, with a mask and image overlaid: Doctors can modify the mask. Rewrite tags The system records feedback data for secondary training.

[0031] Example 2: Intelligent assessment of brain tumors (MRI+fMRI); In the long axis of neurotumor, the system input images are structural MRI (T1 / T2) and functional MRI (fMRI) images: ; Image size normalized to The functional images use a standard BOLD alignment template.

[0032] The system integrates network parameter sets The number of channels in the feature extraction network is automatically adjusted according to the disease type. Increased to 128 to enhance the recognition of activation patterns in tumor margins and brain regions.

[0033] The system can ultimately output tumor volume, location mask, and malignancy score. Doctors can assess tumor progression trends online, and the system supports continuous medical record tracking and analysis.

[0034] Example 3: Assisted analysis for breast cancer screening (ultrasound + MRI); For breast mass identification, the system uses breast ultrasound images and enhanced MRI images as input: ; Images may contain high noise and occlusion at the acquisition end. The system introduces an image preprocessing algorithm based on adaptive mean filtering and a fusion network to jointly model high-contrast regions in enhanced MRI and low-echo structures in ultrasound, which can significantly improve the ability to identify occult nodules.

[0035] In a test involving 100 subjects, the Dice coefficient for the mass boundary of the fusion model improved from 0.82 for the MRI monomodal model to 0.89.

[0036] This system is deployed via Docker containers and supports GPU acceleration and multi-threaded inference. All modules are encapsulated as callable services, with a unified RESTful API interface, including image upload. Model prediction; Doctor correction; ModelID: Version identifier, used to load the model weight parameter set. .

[0037] The system supports switching between multiple disease-specific tasks; only configuration is required. The number of network layers, fusion method, and training hyperparameters can be adjusted for rapid adaptation. Deployment tests show that the inference time for the entire system on a single task is less than 1.3 seconds per case, and its accuracy is superior to conventional diagnostic aids.

[0038] The contents not described in detail in this description are existing technologies known to those skilled in the art. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal fusion-based intelligent medical image-assisted diagnostic system, characterized in that, include: Image acquisition module: Used to acquire multimodal medical image data of the target patient, represented as... ,in: Indicates the total number of modes; Indicates the first The original image of each modality; The image preprocessing module is used to process images of various modalities. Preprocessing is performed to obtain images in a uniform format. ,in: The image is processed by resolution matching, non-rigid registration, noise suppression, and normalization. Feature extraction and alignment module: used to align each preprocessed image Input convolutional neural network to extract feature maps And aligned to attention mechanism ,in: , , These represent the height, width, and number of channels of the feature map, respectively. Representing modes The original feature map; Representing modes Aligned feature map; Fusion Decision Module: Aligns all feature maps Input to the fusion network, output: Lesion segmentation mask , where 1 represents the lesion area; Binary classification prediction results , where 0 represents benign and 1 represents malignant; Human-computer interaction module: used to display the above results. , It also supports doctor feedback for label correction. , ,in: , These are the segmentation and classification results manually corrected by the doctor, used for training data augmentation.

2. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, In the image preprocessing module, any two modal images , Spatial alignment between them is based on maximizing mutual information, and the calculation formula is as follows: ； in: , Indicates the first registration , Modal images; The pixel intensity values ​​are respectively , Joint probability distribution at time; , The marginal probability distributions for each mode are given.

3. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The modality feature extraction employs a multi-channel convolutional network, and the processing procedure is as follows: ； in: Indicates that it is dedicated to the first A convolutional neural network structure with multiple modalities; For the extracted first Feature maps of each modality; , For space dimensions, This represents the number of channels.

4. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The feature alignment employs an attention-based cross-modal mapping, calculated using the following formula: ； in: For the first Input feature map of the modality; , , These are query, key, and value vectors, respectively. This is the weight matrix; The scaling factor for the feature dimension; Representing modes Aligned feature map.

5. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The fusion decision module adopts A multi-layer Transformer structure outputs a globally fused feature vector. ,in: Indicates the number of stacked Transformer layers; The fused feature vector is fed into the classifier to produce the output. And decoded into a segmentation mask by the decoder. .

6. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 5, characterized in that, The loss function during system training The weighted sum of segmentation loss and classification loss: ； in: , The segmentation mask and classification results output by the system; , These are genuine labels manually annotated. Loss due to Dice or IoU; The cross-entropy loss function; These are the weighting coefficients.

7. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The human-computer interaction module includes: a visualization display submodule, used to display multimodal raw images. Segmentation mask with model output Overlay displays generate a visual heatmap. ,in This indicates an image overlay operation; The diagnostic confidence output submodule is used to display the classification results. Corresponding probability value This indicates the system's confidence in the "malicious" label; The feedback acquisition submodule is used to receive the feedback mask edited by the doctor. With category correction tags And use it to update the training dataset; The model optimization interface is used to process feedback data. Incremental model training is performed using the original feature input.

8. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The image acquisition module interfaces with the hospital's PACS system via the DICOM protocol, supporting the extraction of patient identifiers from DICOM files. Image sequence numbering Information such as image location index idx; Uniform formatted output as a standard image stack ,in: Indicates the first Number of slices in a modal image sequence; Indicates the first The first modality Image slices.

9. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The system is adapted to multimodal fusion configurations for different diseases. For lung lesion detection, the modal combination used is CT and PET, with a corresponding number of modalities. The input image is ; For brain tumor identification, the modalities used are MRI and functional MRI (fMRI), denoted as { ; For breast lesion screening, the modalities used are MRI and ultrasound images, denoted as { The system automatically loads a predefined set of network structure parameters based on the disease type. This includes the number of network layers, fusion method, and training hyperparameters.

10. The intelligent medical image-assisted diagnostic system based on multimodal fusion according to claim 1, characterized in that, The system adopts a containerized deployment approach and has the following operational capabilities: The system is encapsulated in a Docker container and has cross-platform deployment capabilities; Supports GPU-accelerated inference, using a tensor computation framework for feature tensors. With fusion results Perform efficient processing; The system provides an open RESTful API for receiving image upload requests. Return diagnostic results Receive doctor feedback Supports remote model update functionality, dynamically loading the latest parameter and weight set via the model version identifier ModelID. .

Images