A medical image multi-modal calculation method and system based on cross-modal generation
By using a dual-path Siamese neural network and a cross-modal feature fusion module, the diagnostic error caused by differences in MRI data from different modalities is solved, achieving efficient multimodal medical image analysis and diagnosis, and improving accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
- Filing Date
- 2024-12-06
- Publication Date
- 2026-06-09
AI Technical Summary
Differences in contrast, resolution, and noise levels among MRI data from different modalities make accurate alignment and matching difficult, affecting the accuracy of multimodal medical image analysis and diagnosis.
A dual-path Siamese neural network and a cross-modal feature fusion module are employed. The cross-modal generative network learns the potential consistency features between different modalities and uses complementary information for feature fusion. Feature maps from the cross-modal generative network and the brain imaging diagnostic network are input into the cross-modal feature fusion module for diagnosis.
It significantly improves the consistency and fusion efficiency of multimodal data, reduces diagnostic errors caused by modal differences, enhances the accuracy and diagnostic capabilities of medical image analysis, reduces the workload of doctors, and improves medical efficiency.
Smart Images
Figure CN122175847A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical imaging technology, and in particular to a method and system for multimodal computation of medical images based on cross-modal generation. Background Technology
[0002] Multimodal fusion technology is an important tool for analyzing medical image data. It involves integrating and fusing information from different input modalities. MRI-based multimodal data contains rich visual information. However, different MRI modalities vary significantly in contrast, resolution, and noise levels. These differences make it difficult to accurately align and match features across modalities, thus affecting the accuracy of analysis and diagnosis. When multiple modalities coexist, it is necessary to extract target features from the different visual representations of the multimodal data.
[0003] To overcome these shortcomings, this application proposes a method and system for multimodal computation of medical images based on cross-modal generation. Summary of the Invention
[0004] The purpose of this application is to provide a method and system for multimodal computation of medical images based on cross-modal generation, aiming to solve the above-mentioned problems.
[0005] To achieve the above objectives, this application provides the following technical solution:
[0006] Firstly, this application provides a method for multimodal computation of medical images based on cross-modal generation, the specific steps of which include:
[0007] Brain imaging data is acquired and preprocessed; wherein, the brain imaging data includes brain MRI data and masking information in different modalities;
[0008] The preprocessed brain imaging data is input into a dual-path twin neural network for training; wherein, the dual-path twin neural network includes a cross-modal generative network and a brain imaging diagnostic network;
[0009] The first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network are input into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result.
[0010] The brain imaging data of the target user is acquired, and the brain imaging data of the target user is input into the trained dual-path Siamese neural network and the cross-modal feature fusion module to output the evaluation result.
[0011] Secondly, this application proposes a multimodal computing system for medical images based on cross-modal generation, comprising:
[0012] Preprocessing module: Acquires brain imaging data and preprocesses the brain imaging data; wherein, the brain imaging data includes brain MRI data and masking information under different modalities;
[0013] Training module: The dual-path twin neural network is trained based on preprocessed brain imaging data; wherein, the dual-path twin neural network includes a cross-modal generation network and a brain imaging diagnostic network;
[0014] Feature fusion module: The first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network are input into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result;
[0015] Output module: Acquires the target user's brain imaging data, inputs the target user's brain imaging data into the trained dual-path Siamese neural network and the cross-modal feature fusion module, and outputs the evaluation results.
[0016] Thirdly, this application provides an apparatus comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a medical image multimodal computing method based on cross-modal generation; the processor is configured to execute the program instructions stored in the memory to implement a medical image multimodal computing method based on cross-modal generation.
[0017] Fourthly, this application provides a storage medium storing processor-executable program instructions for executing a medical image multimodal computation method based on cross-modal generation.
[0018] This application provides a method and system for multimodal computation of medical images based on cross-modal generation, which has the following beneficial effects:
[0019] (1) The brain MRI data of multiple modalities are processed by a dual-path Siamese neural network, the data of different modalities are deeply analyzed, and these features are fused by a cross-modal feature fusion module; the network learns the potential consistency features between different modal data, which significantly improves the consistency and fusion efficiency between multimodal data; it helps to reduce diagnostic errors caused by modal differences and improves the accuracy of medical image analysis.
[0020] (2) Cross-modal generative networks are responsible for performing generative tasks under modality loss, that is, using partial modality data to generate images of other modalities, so that the network can explore and utilize complementary information under different modalities; by making full use of complementary information, key brain regions can be better located, thereby improving the ability to diagnose diseases.
[0021] (3) This application adopts a dual-path twin neural network and a cross-modal feature fusion module. By optimizing the network structure and training process, it can quickly provide accurate diagnostic results, thereby reducing the workload of doctors and improving medical efficiency. Attached Figure Description
[0022] Figure 1 This is a flowchart illustrating a medical image multimodal computation method based on cross-modal generation, according to Embodiment 1 of this application.
[0023] Figure 2 This is a schematic diagram of the dual-path twin neural network structure in Embodiment 1 of this application;
[0024] Figure 3 This is a schematic diagram of the structure of the integrated convolutional module in the dual-path Siamese neural network of Embodiment 1 of this application;
[0025] Figure 4 This is a schematic diagram of the cross-modal feature fusion module of Embodiment 1 of this application;
[0026] Figure 5 This is a schematic diagram of the structure of a medical image multimodal computing system based on cross-modal generation, according to Embodiment 2 of this application.
[0027] Figure 6 This is a schematic diagram of the device structure in Embodiment 3 of this application;
[0028] Figure 7 This is a schematic diagram of the storage medium structure of Embodiment 4 of this application. Detailed Implementation
[0029] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0030] The following analysis, based on relevant technologies, examines existing solutions.
[0031] Current multimodal fusion methods can be categorized into input-level fusion, hierarchical fusion, and decision-level fusion methods.
[0032] Input-level fusion divides multimodal data into multiple input channels and fuses features at the input layer to train the network. Current input and fusion methods mainly use four modalities of brain MRI data—T1, T1Gd, T2, and T2-flair—as multi-channel inputs for multi-task learning in order to obtain better diagnostic results. However, input-level fusion methods do not design additional networks to extract features for each modality; they only use different channels of the same network to randomly learn the visual attributes of different modalities, making it difficult for the model to fully extract the effective information in each modality, resulting in insufficient learning.
[0033] Hierarchical fusion methods typically construct multiple channel analysis networks, dividing different modalities of brain MRI data into multiple groups, training these groups on different channels, and then fusing the visual features obtained from all channels under different modalities to obtain more accurate disease prediction results. Hierarchical fusion methods require constructing multiple networks, all performing the same task objective. This obviously leads to extremely high similarity in the visual features learned by each network, which is not conducive to mining complementary information between different modalities.
[0034] Decision-level fusion methods require the simultaneous construction of multiple modality feature extraction networks, each trained independently through a completely closed and mutually exclusive training approach to obtain unique visual features for each modality. Furthermore, each network performs independent analysis and diagnosis, with each network's diagnostic results relying solely on the unique attributes of one modality. While decision-level fusion methods can amplify the information elements contained in each modality's data, excessive discrepancies can make it difficult for the model to accurately grasp detailed information during the final fusion process.
[0035] While the aforementioned methods offer different execution strategies and analytical perspectives for the multimodal fusion process, they all neglect the differences in data format and visual semantics between different modalities. This leads to the introduction of disruptive information into the fusion process, causing the model to be unable to judge and identify valid information. This application combines generation and discrimination strategies, which can not only fully find the common information in multiple modal data but also mine complementary information between different modal data. Furthermore, the model has low complexity, significantly reducing computational cost while maintaining accuracy.
[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0037] Example 1
[0038] Please see Figure 1 This is a flowchart illustrating a multimodal computation method for medical images based on cross-modal generation, according to Embodiment 1 of this application; the steps include:
[0039] S1: Acquire brain imaging data and preprocess the brain imaging data; wherein, the brain imaging data includes brain MRI data and masking information under different modalities.
[0040] In this embodiment, the brain imaging data is cropped; the cropped brain imaging data is then normalized using a normalization method, calculated as follows:
[0041]
[0042] Where i, j, and k represent the sagittal, coronal, and vertical axes of the brain MRI data, respectively; x ijk For brain imaging data, x i ′ jk The brain imaging data has been normalized.
[0043] S2: Training is performed on a dual-path twin neural network based on preprocessed brain imaging data; wherein the dual-path twin neural network includes a cross-modal generation network and a brain imaging diagnostic network.
[0044] In this embodiment, the cross-modal generation network and the brain imaging diagnosis network are composed of several sets of integrated convolutional modules. Each integrated convolutional module includes two batch normalization modules, two linear rectified function modules, and two convolutional neural network modules.
[0045] The batch normalization module is used to organize the data distribution in each processing batch, reducing the interference of abnormal data on the network training and learning process.
[0046] The linear rectified function module is used to increase the network's ability to fit nonlinear features in order to obtain an objective function that better reflects the data patterns.
[0047] The first convolutional layer in the convolutional neural network module is used to adaptively weight the region of interest and obtain a wider network field of interest; the second convolutional layer, while expanding the model's field of interest, will also reduce the resolution of the brain image data through pooling operations to obtain more effective sampling results.
[0048] The calculation formula for the training process of a dual-path Siamese neural network is as follows:
[0049] y1,y2=Conv(ReLU(BN(x1,x2)))
[0050] Where x1 and x2 are the input images of the cross-modal generative network and the brain imaging diagnostic network, respectively; BN is the batch normalization operation; ReLU is the linear rectified function module; Conv is the convolutional neural network module; and y1 and y2 are the two output results of the ensemble convolutional module of the cross-modal generative network and the ensemble convolutional module of the brain imaging diagnostic network, respectively.
[0051] Furthermore, the cross-modal generative network also includes: a first loss function for the cross-modal generative network, which guides the backpropagation of the network and trains the various parameters of the deep learning network. The formula for the first loss function is:
[0052]
[0053] in, The prediction results for the cross-modal generative network; y i The first training label is the actual diagnostic result; L recons L is the first loss function for cross-modal generative networks; mae L is the mean absolute error loss function; mse This is the mean squared error loss function.
[0054] The brain imaging diagnostic network also includes a second loss function designed to guide the network in learning and fitting the true distribution results. The formula for the second loss function is:
[0055]
[0056] Among them, L diag The second loss function of the brain imaging diagnostic network; d i The second training label is the brain imaging data of the preset modality; The prediction results are from the brain imaging diagnostic network.
[0057] S3: Input the first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network into the cross-modal feature fusion module for feature fusion to obtain the diagnosis result.
[0058] In this embodiment, the feature mappings of the two paths are fused through a cross-modal feature fusion module. Specifically, the cross-modal feature fusion module uses an attention gating mechanism to process and filter visual features from different tasks and distributions. The calculation formula is as follows:
[0059] N1,N2=Bottleneck(Dropout(SE(f1,f2)))
[0060] Among them, Bottleneck is the bottleneck projection module, which is used to integrate the fused cross-modal feature distribution; Dropout is a random selection of stopping layers, which is used to randomly shut down the information propagation of neurons, providing an implicit integration effect for the module; SE is the channel activation module, which is used to select the channel with the largest contribution and weight the channel; N1 and N2 are the output results of the two branches of the cross-modal feature fusion module; f1 and f2 are the inputs of the two branches of the cross-modal feature fusion module, namely the first intermediate layer feature map and the second intermediate layer feature map, respectively.
[0061] The final diagnostic result is obtained by decoding the cross-modal features fused by the decoder.
[0062] S4: Acquire the target user's brain imaging data, input the target user's brain imaging data into the trained dual-path Siamese neural network and the cross-modal feature fusion module, and output the evaluation result.
[0063] In this embodiment, a comparative experiment was conducted between the present application and the prior art, and the results are shown in the table below:
[0064] Method Dice coefficient Sensitivity Specificity Hausdorff distance ResU-Net 87.06 86.63 99.80 9.16 AttentionU-Net 87.90 87.36 99.85 7.37 ResU-Net++ 88.14 87.56 99.89 7.67 BiSeNet v2 86.63 85.73 99.84 8.71 PSPnet 87.60 85.90 99.91 5.95 DeepLabv3+ 88.37 87.69 99.90 6.95 Method of the present application 89.04 88.65 99.90 5.52
[0065] As can be seen, this application significantly surpasses existing methods in several technical indicators, including Dice coefficient, sensitivity, and Hausdorff distance. Therefore, this application can better uncover the correlation information between multiple modalities, providing a stronger guarantee for the analysis and diagnosis of brain images.
[0066] In summary, this application significantly improves the consistency and fusion efficiency of multimodal brain imaging data by constructing a dual-path Siamese neural network and a cross-modal feature fusion module. First, by collecting and preprocessing brain imaging data, the consistency in size and comparability in pixel intensity across different modalities is ensured, laying the foundation for subsequent analysis. Second, the feature extraction and alignment capabilities of the Siamese network are utilized to maximize the mining of potential consistent features between different modalities while reducing domain differences. Specifically, the cross-modal generative network can generate images of other modalities using data from some modalities, effectively utilizing complementary information from different modalities. Finally, the cross-modal feature fusion module achieves efficient fusion of feature maps from two paths through an attention gating mechanism, further improving the accuracy of diagnostic results. This application demonstrates significant advantages in the field of medical image analysis and provides strong support for the diagnosis of brain images.
[0067] Example 2
[0068] Please see Figure 5 This is a schematic diagram of the structure of a medical image multimodal computing system based on cross-modal generation, according to Embodiment 2 of this application; the specific content includes:
[0069] Preprocessing module: Acquires brain imaging data and preprocesses the brain imaging data; wherein, the brain imaging data includes brain MRI data and masking information under different modalities;
[0070] Training module: The dual-path twin neural network is trained based on preprocessed brain imaging data; wherein, the dual-path twin neural network includes a cross-modal generation network and a brain imaging diagnostic network;
[0071] Feature fusion module: The first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network are input into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result;
[0072] Output module: Acquires the target user's brain imaging data, inputs the target user's brain imaging data into the trained dual-path Siamese neural network and the cross-modal feature fusion module, and outputs the evaluation results.
[0073] Example 3
[0074] Please see Figure 6 This is a schematic diagram of the device structure in Embodiment 3 of this application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51.
[0075] The memory 52 stores program instructions for implementing the above-described method for multimodal computation of medical images based on cross-modal generation.
[0076] The processor 51 is used to execute program instructions stored in the memory 52 to implement a medical image multimodal computation based on cross-modal generation.
[0077] The processor 51 can also be referred to as a CPU (Central Processing Unit).
[0078] Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.
[0079] Example 4
[0080] Please see Figure 7 This is a schematic diagram of the storage medium in Embodiment 4 of this application. The storage medium in this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.
[0081] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0082] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
[0083] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.
[0084] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.
Claims
1. A multimodal computation method for medical images based on cross-modal generation, characterized in that, include: Brain imaging data is acquired and preprocessed; wherein, the brain imaging data includes brain MRI data and masking information in different modalities; The preprocessed brain imaging data is input into a dual-path twin neural network for training; wherein, the dual-path twin neural network includes a cross-modal generative network and a brain imaging diagnostic network; The first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network are input into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result. The brain imaging data of the target user is acquired, and the brain imaging data of the target user is input into the trained dual-path Siamese neural network and the cross-modal feature fusion module to output the evaluation result.
2. The medical image multimodal calculation method based on cross-modal generation according to claim 1, characterized in that, The steps of acquiring brain imaging data and preprocessing the brain imaging data specifically include the following steps: The brain imaging data is cropped; The cropped brain imaging data was normalized using a normalization method, calculated as follows: Where i, j, and k represent the sagittal, coronal, and vertical axes of the brain MRI data, respectively; x ijk For brain imaging data, x i ′ jk The brain imaging data has been normalized.
3. The method for multimodal computation of medical images based on cross-modal generation according to claim 1, characterized in that, The training of a dual-path Siamese neural network based on preprocessed brain imaging data, wherein the dual-path Siamese neural network includes a cross-modal generative network and a brain imaging diagnostic network, specifically includes the following steps: The cross-modal generation network and the brain imaging diagnosis network are composed of several sets of integrated convolutional modules, which include two batch normalization modules, two linear rectified function modules and two convolutional neural network modules. The calculation formula for the training process of the dual-path Siamese neural network is as follows: y1,y2=Conv(ReLU(BN(x1,x2))) Where x1 and x2 are the input images of the cross-modal generative network and the brain imaging diagnostic network, respectively; BN is the batch normalization operation; ReLU is the linear rectified function module; Conv is the convolutional neural network module; and y1 and y2 are the two output results of the ensemble convolutional module of the cross-modal generative network and the ensemble convolutional module of the brain imaging diagnostic network, respectively.
4. The medical image multimodal calculation method based on cross-modal generation according to claim 3, characterized in that, The cross-modal generation network further includes: training the parameters of the cross-modal generation network using a first loss function, the formula of which is: in, The prediction results for the cross-modal generative network; y i The first training label is the actual diagnostic result; L recons L is the first loss function for cross-modal generative networks; mae L is the mean absolute error loss function; mse Let be the mean squared error loss function.
5. The medical image multimodal calculation method based on cross-modal generation according to claim 3, characterized in that, The brain imaging diagnostic network further includes: training the parameters of the brain imaging diagnostic network using a second loss function, the formula of which is: Among them, L diag The second loss function of the brain imaging diagnostic network; d i The second training label is the brain imaging data of the preset modality; The prediction results are from the brain imaging diagnostic network.
6. The method for multimodal computation of medical images based on cross-modal generation according to claim 1, characterized in that, The step of inputting the first intermediate layer feature map of the acquired cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result specifically includes the following steps: The cross-modal feature fusion module uses an attention gating mechanism to process and filter visual features from different tasks and distributions. The calculation formula is as follows: N1,N2=Bottleneck(Dropout(SE(f1,f2))) Among them, Bottleneck is the bottleneck projection module, which is used to integrate the fused cross-modal feature distribution; Dropout is a random selection of stopping layers, which is used to randomly shut down the information propagation of neurons; SE is the channel activation module, which is used to select the channel with the largest contribution and weight the channel; N1 and N2 are the output results of the two branches of the cross-modal feature fusion module; f1 and f2 are the feature maps of the first intermediate layer and the second intermediate layer, respectively. The diagnostic results are obtained by decoding the cross-modal features fused by the cross-modal feature fusion module using a decoder.
7. A system for multimodal computation of medical images based on cross-modal generation according to claim 1, characterized in that, include: Preprocessing module: Acquires brain imaging data and preprocesses the brain imaging data; wherein, the brain imaging data includes brain MRI data and masking information under different modalities; Training module: The dual-path twin neural network is trained based on preprocessed brain imaging data; wherein, the dual-path twin neural network includes a cross-modal generation network and a brain imaging diagnostic network; Feature fusion module: The first intermediate layer feature map of the cross-modal generation network and the second intermediate layer feature map of the brain imaging diagnosis network are input into the cross-modal feature fusion module for feature fusion to obtain the diagnostic result; Output module: Acquires the target user's brain imaging data, inputs the target user's brain imaging data into the trained dual-path Siamese neural network and the cross-modal feature fusion module, and outputs the evaluation results.
8. A device, characterized in that, The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a medical image multimodal computing method based on cross-modal generation as described in any one of claims 1-6; the processor is used to execute the program instructions stored in the memory to implement a medical image multimodal computing method based on cross-modal generation.
9. A storage medium, characterized in that, The device stores processor-executable program instructions for performing a medical image multimodal computation method based on cross-modal generation as described in any one of claims 1-6.