Medical image data analysis system based on pixel-level multi-modal fusion

Abstract

The application discloses a medical image data analysis system based on pixel-level multi-modal fusion, comprising: a pixel-level feature extraction module for spatial alignment of preprocessed multi-modal medical image data at the pixel level and extraction of multi-modal multi-scale pixel-level feature maps; a dynamic weight generation module for generating multi-modal fusion weight maps corresponding to the multi-modal multi-scale pixel-level feature maps; a pixel-level fusion module for obtaining fused pixel-level feature maps according to the multi-modal multi-scale pixel-level feature maps and the corresponding multi-modal fusion weight maps; and a prediction module for micro-lesion prediction in medical image data through a pre-trained prediction network and pixel-level uncertainty quantification of the prediction result. The embodiment of the application solves the problem that heterogeneous medical image data cannot be accurately aligned and deeply fused, and achieves the technical effect of making full use of different characteristics of multi-modal medical image data for complementary advantages.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology in medical image analysis, and in particular to a medical image data analysis system based on pixel-level multimodal fusion. Background Technology

[0002] With the rapid development of modern medical imaging technologies, such as computed tomography (CT), high-resolution computed tomography (HRCT), whole slide imaging (WSI), and positron emission tomography (PET), doctors are able to obtain multi-dimensional and multi-scale medical imaging information of patients.

[0003] In recent years, artificial intelligence technology, especially deep learning, has demonstrated tremendous potential in the field of medical image analysis. By fusing multimodal medical image data, it can improve the accuracy and efficiency of medical image interpretation. However, existing technologies still face many challenges when processing minute lesions in medical images. For example, HRCT medical images typically focus on the macroscopic morphology and location information of lesions, while pathological WSI emphasizes revealing the microscopic structure and pathological features at the cellular level. How to accurately align and deeply fuse these heterogeneous data to fully utilize their complementary advantages is a problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a medical image data analysis system based on pixel-level multimodal fusion, which achieves the technical effect of fully utilizing the different characteristics of multimodal medical image data to complement each other's strengths.

[0005] This application provides a medical image data analysis system based on pixel-level multimodal fusion, the system comprising: A pixel-level feature extraction module is used to perform pixel-level spatial alignment of preprocessed multimodal medical image data through a deformable registration network to obtain aligned multimodal medical image data, and to extract multimodal multi-scale pixel-level feature maps from the aligned multimodal medical image data through a multi-scale feature extraction network. The pixel-level feature maps include pixel-level spatial information of the preprocessed multimodal medical image data. The dynamic weight generation module is used to receive the prediction task label, generate a task embedding vector based on the prediction task label, and generate a multimodal fusion weight map corresponding to the multimodal multi-scale pixel-level feature map through the task embedding vector and the multilayer perceptron network. The pixel-level fusion module is used to fuse the multi-scale pixel-level feature map of the multi-modality based on the multi-scale pixel-level feature map of the multi-modality and the corresponding multi-modality fusion weight map to obtain the fused pixel-level feature map. The prediction module is used to predict small lesions in medical image data based on the fused pixel-level feature map through a pre-trained prediction network, and to quantify the uncertainty of the prediction results at the pixel level, and output the prediction results of the small lesions and the corresponding pixel-level uncertainty map.

[0006] In some implementations, the prediction module is further configured to: The fused pixel-level feature map is propagated multiple times by a pre-trained prediction network. Each forward propagation yields a set of pixel-level prediction results. The prediction result of the micro lesion is obtained by averaging the multiple sets of pixel-level prediction results. The variance or entropy of the multiple sets of pixel-level prediction results is calculated to obtain the corresponding pixel-level uncertainty map.

[0007] In some embodiments, the pixel-level fusion module further includes: The weak signal enhancement submodule is used to construct a histogram equalization mapping function by adaptive histogram equalization (AHE) for small, low-contrast regions of the fused pixel-level feature map, and then use the mapping function for specific enhancement; or to use local contrast stretching (LCS) for specific enhancement.

[0008] In some embodiments, the pixel-level fusion module further includes: The context-aware attention submodule is used to calculate the correlation between each pixel and its neighborhood and the entire multi-scale pixel-level feature map of the multimodal multi-scale pixel-level feature map through the context-aware attention mechanism, so as to enhance the feature expression of small lesions and suppress background noise.

[0009] In some embodiments, the pixel-level feature extraction module further includes: The attention-guided submodule is used to combine the attention-guided mechanism to focus on small lesion areas when aligning multimodal medical image data through the deformable registration network.

[0010] In some embodiments, the preprocessed multimodal medical imaging data includes at least two of the following: computed tomography (CT) data, high-resolution computed tomography (HRCT) data, whole-slide pathology (WSI) images, and positron emission tomography (PET) data.

[0011] In some embodiments, the preprocessed multimodal medical image data is preprocessed high-resolution computed tomography (HRCT) data; Accordingly, the pixel-level feature extraction module is also used to extract multi-scale pixel-level features from the preprocessed HRCT data using 3D U-Net or 3D ResNet as the multi-scale feature extraction network. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

[0012] In some embodiments, the preprocessed multimodal medical image data is preprocessed whole-slide (WSI) image data. Accordingly, the pixel-level feature extraction module is also used to segment the preprocessed WSI data to obtain multiple WSI data blocks, extract multi-scale features from the multiple WSI data blocks through the multi-scale feature extraction network, and organize the multi-scale features of the multiple WSI data blocks into multi-scale pixel-level features. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

[0013] In some embodiments, the medical image data analysis system based on pixel-level multimodal fusion further includes: The multimodal image preprocessing module is used to standardize, denoise, and enhance the original multimodal medical image data to obtain structured multimodal medical image data, and to clean and encode the structured multimodal medical image data to obtain preprocessed multimodal medical image data. The original multimodal medical image data includes at least two different modalities of medical image data. The text and structured data processing module is used to clean and encode structured or text data to obtain preprocessed structured and text data; and to extract feature vectors from the preprocessed structured and text data to obtain structured and text data feature vectors. The results output module is used to interactively display the lesion area, uncertainty heatmap and multimodal fusion weights based on the prediction results of small lesions and the corresponding pixel-level uncertainty map, and to display non-medical image data based on the feature vectors of structured and textual data.

[0014] In some implementations, the prediction task label includes at least one of the following: lesion segmentation task, benign / malignant determination task, and treatment response prediction task.

[0015] The medical image data analysis system based on pixel-level multimodal fusion proposed in this application solves the problem of heterogeneous medical image data being unable to be accurately aligned and deeply fused through a pixel-level feature extraction module, a dynamic weight generation module, a pixel-level fusion module, and a prediction module. It achieves the technical effect of fully utilizing the different characteristics of multimodal medical image data to complement each other's strengths. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a medical image data analysis system based on pixel-level multimodal fusion, provided in Embodiment 1 of this application. Figure 2 This is a schematic diagram of the structure of a medical image data analysis system based on pixel-level multimodal fusion, provided in Embodiment 2 of this application. Detailed Implementation

[0018] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the field of Artificial Intelligence (AI) technology for medical image analysis, multimodal medical image data fusion has gradually become a research hotspot. However, most studies remain at the feature level or decision level. For example, some methods simply concatenate the feature vectors of medical image data from different modalities and input them into a classifier for analysis and prediction, or analyze and predict different modalities of medical image data separately and then vote on the results. This coarse-grained fusion approach makes it difficult to capture the correspondence and complementary information between different modalities of medical image data in the fine spatial dimension at the pixel level.

[0020] Minimal lesions in medical imaging data typically appear as small, low-contrast, and poorly defined areas with weak signal intensity, easily masked by background noise, normal tissue structures, or artifacts. This makes it difficult for existing medical imaging data analysis systems to effectively extract and utilize the key features of these small lesions when processing their weak signals, leading to missed diagnoses or misdiagnoses. This is especially true in scenarios with limited data, such as early lesions of rare diseases or specific subtypes, where high-quality labeled data is extremely scarce. This makes it difficult for medical imaging data analysis systems to learn robust features of small lesions, resulting in poor generalization ability and unsatisfactory performance in practical clinical applications. For example, for early small lesions of a rare lung disease, due to the limited number of cases, it is difficult for medical imaging data analysis systems to learn their unique patterns through large-scale data training.

[0021] Existing medical imaging data analysis systems typically only provide analytical and predictive results, such as the presence of target lesions and the probability of benign or malignant transformations in the target area, lacking a quantitative assessment of the reliability of these results. Due to the indistinct boundaries and atypical characteristics of small lesions, the analysis and prediction of small lesions often involve higher uncertainty, leading to a decrease in the accuracy of medical imaging data analysis systems. In clinical practice, doctors not only need the analytical and predictive results provided by medical imaging data analysis systems, but also need to understand the system's level of "confidence" in those results. Without pixel-level uncertainty quantification in medical imaging data analysis systems, doctors find it difficult to determine which analyses and predictions are high-risk and require further investigation or manual review, thus increasing the risk of clinical decision-making.

[0022] Different clinical tasks may focus on different aspects of small lesions. For example, in small lesion segmentation tasks, medical image data analysis systems need to accurately identify the boundaries of lesions; while in tasks involving determining the benign or malignant nature of a target area, more attention may be needed to focus on the texture, density, or metabolic characteristics within the lesion. Existing medical image data analysis systems often employ fixed or general medical image data fusion strategies, making it difficult to dynamically adjust the fusion strategy according to the specific needs of clinical tasks. This results in limited performance of medical image data analysis systems across multiple clinical task scenarios.

[0023] Example 1 Figure 1 This is a schematic diagram of a medical image data analysis system based on pixel-level multimodal fusion provided in Embodiment 1 of this application. The system described in this embodiment can be used to automatically analyze the patient's condition based on multimodal medical image data. The system includes a pixel-level feature extraction module 110, a dynamic weight generation module 120, a pixel-level fusion module 130, and a prediction module 140.

[0024] The pixel-level feature extraction module 110 is used to perform pixel-level spatial alignment of the preprocessed multimodal medical image data through a deformable registration network to obtain aligned multimodal medical image data, and to extract multimodal multi-scale pixel-level feature maps from the aligned multimodal medical image data through a multi-scale feature extraction network. The pixel-level feature maps include pixel-level spatial information of the preprocessed multimodal medical image data.

[0025] In this embodiment, the multimodal medical image data typically comes from different medical imaging devices and has different resolutions, contrasts, and information dimensions, including but not limited to computed tomography (CT) data, high-resolution computed tomography (HRCT) data, whole-slide imaging (WSI) data, positron emission tomography (PET) data, and nuclear magnetic resonance imaging (MRI) data. The medical image data analysis system based on pixel-level multimodal fusion described in this application can perform fusion and other processing on at least two types of multimodal medical image data.

[0026] Because of the significant resolution differences between different modalities of medical image data, such as HRCT data and WSI data, spatial alignment of the multimodal medical image data is first required, also known as registration. This application employs a deep learning-based deformable registration network, such as the VoxelMorph registration network. The VoxelMorph registration network learns a deformation field, enabling it to accurately register lesion regions on WSI data representation images to their corresponding three-dimensional spatial locations in HRCT data.

[0027] Optionally, the pixel-level feature extraction module further includes an attention-guided submodule, which is used to combine the attention-guided mechanism to focus on small lesion areas when aligning multimodal medical image data through the deformable registration network.

[0028] To improve registration accuracy, this application introduces an attention-guided mechanism based on micro-lesion regions. Specifically, when training the deformable registration network, in addition to the traditional image similarity loss function, an attention loss function based on micro-lesion regions is added. This micro-lesion region-based attention loss function guides the deformable registration network using pre-annotated micro-lesion regions or pixel-level probability maps of micro-lesions obtained through weakly supervised learning, minimizing the alignment error of the deformable registration network in micro-lesion regions. For example, the Dice Loss or IoU Loss function can be used to measure the alignment degree of micro-lesion regions. After registering HRCT data and WSI data, the pixels of the image represented by the WSI data are mapped to the coordinate system corresponding to the HRCT data, achieving pixel-level spatial alignment.

[0029] The advantage of setting up an attention-guided submodule is that, while achieving non-rigid registration of multimodal medical image data, it guides the medical image data analysis system to focus on small lesion areas, thereby improving the registration effectiveness of multimodal medical image data.

[0030] Multi-scale feature extraction networks, such as U-Net, ResNet, and Transformer, are used to extract multi-modal, multi-scale pixel-level feature maps from aligned multi-modal medical image data. These pixel-level feature maps retain the pixel-level spatial information of the preprocessed multi-modal medical image data during the extraction process and can map the features of different modalities to a unified pixel coordinate system.

[0031] The term "multi-scale" here refers to the extraction of feature maps through a series of convolutional and downsampling layers in deep learning networks, especially encoder-decoder structures like U-Net. In the shallow layers of the network, the spatial resolution of the feature maps is high, capturing local, fine-grained features of medical image data, such as pixel-level texture and edge information. In the deeper layers, the spatial resolution of the feature maps gradually decreases, but each pixel or voxel represents a larger receptive field, capturing more abstract and global semantic features, such as the overall shape of the lesion and its relative position to surrounding tissues. Two-dimensional (2D) medical image data consists of multiple pixels, while three-dimensional (3D) medical image data consists of multiple voxels. This application does not explicitly distinguish between these two dimensions; in some cases, "pixel" may refer to "voxel."

[0032] Optionally, when the preprocessed multimodal medical image data is preprocessed high-resolution computed tomography (HRCT) data, the pixel-level feature extraction module is further used to extract multi-scale pixel-level features from the preprocessed HRCT data using 3D U-Net or 3D ResNet as a multi-scale feature extraction network. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

[0033] HRCT data represents 3D medical imaging data. For pixel-level feature extraction of HRCT data, the encoder part of U-Net can extract 3D pixel-level feature maps from low-level texture to high-level semantics at different levels of abstraction and spatial resolution through multiple convolutional layers and downsampling layers. The 3D pixel-level feature map not only contains information about each pixel, but also the contextual information of that pixel in 3D space. In each downsampling layer, the corresponding pixel-level feature map is retained for subsequent pixel-level fusion.

[0034] Optionally, when the preprocessed multimodal medical image data is preprocessed whole-slide image (WSI) data, the pixel-level feature extraction module is further used to segment the preprocessed WSI data to obtain multiple WSI data blocks, extract multi-scale features from the multiple WSI data blocks through the multi-scale feature extraction network, and organize the multi-scale features of the multiple WSI data blocks into multi-scale pixel-level features. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

[0035] WSI data represents two-dimensional medical image data. Due to the extremely high resolution of medical images represented by WSI data, a patch-based feature extraction method based on Vision Transformer (ViT) or ResNet is employed. First, the WSI data is divided into multiple small patches. For example, the medical image represented by WSI data is divided into multiple 256x256 pixel patches. Multi-scale feature extraction networks are used to extract multi-scale features from each patch. These multi-scale feature extraction networks can be pre-trained pathological image analysis models, such as pathological ViT based on self-supervised learning. The multi-scale features extracted from the WSI data patches are two-dimensional features. To obtain pixel-level features, the features of these patches are reorganized into multi-scale pixel-level features corresponding to the pre-processed WSI data. During final fusion, if fusion with a three-dimensional pixel-level feature map is required, appropriate dimensionality transformation or processing is necessary.

[0036] HRCT data characterizes medical images that can provide information on the macroscopic morphology, location, and density of lesions, while WSI data characterizes medical images that focus on providing microscopic features such as cytology and histology of lesions. Assuming the resolution of HRCT data characterizes medical images is 512... 512 If Z (where Z is the number of layers) is used, then the resolution of medical images represented by WSI data can reach tens of thousands. Tens of thousands of pixels. This application provides a specific method and tools for extracting multi-scale pixel-level features using a pixel-level feature extraction module when the preprocessed multimodal medical image data is HRCT data or WSI data. This method can effectively extract multi-scale pixel-level features from medical image data of different modalities.

[0037] The dynamic weight generation module 120 is used to receive the prediction task label, generate a task embedding vector based on the prediction task label, and generate a multimodal fusion weight map corresponding to the multimodal multi-scale pixel-level feature map through the task embedding vector and the multilayer perceptron network.

[0038] Optionally, the prediction task label includes at least one of the following: lesion segmentation task, benign / malignant determination task, and treatment response prediction task.

[0039] In this embodiment, a task embedding vector is generated based on the label of the prediction task to be performed, such as the task of determining the benign or malignant nature of lung nodules, the task of segmenting small lesions of early liver cancer, or the task of detecting microbleeds in the brain in rare diseases. This task embedding vector can be a one-hot encoded vector of the task ID (Identity Document) selected by the user, or it can be a vector representation learned from the task description through a small neural network. Assuming that the label of the prediction task to be performed is the task of determining the benign or malignant nature of small lung nodules, the dynamic weight generation module 120 generates a task embedding vector for this task, which can be, for example, a one-hot encoded vector [0, 1, 0].

[0040] The dynamic weight generation module 120 generates a multimodal fusion weight map corresponding to the multi-scale pixel-level feature map of the multimodal data based on the task embedding vector and the multilayer perceptron network. For example, the task embedding vector [0, 1, 0] is input into a small multilayer perceptron (MLP) network. This MLP network takes the task embedding vector and preliminary fusion features from HRCT data and WSI data as input, and outputs two pixel-level fusion weight maps corresponding to the HRCT data and WSI data, respectively. Each pixel value in the fusion weight map represents the contribution of the corresponding modality to the current task at that pixel position. The preliminary fusion features of HRCT data and WSI data can be obtained through simple concatenation or early attention fusion. For example, for the task of determining the benignity or malignancy of small lung nodules, the MLP network may assign higher weights to regions with high cellular atypia in the WSI data, because the pathological features of these regions are crucial for determining benignity or malignancy. Alternatively, the task embedding vector and the preliminary fusion features of the preprocessed multimodal medical image data can be input into a lightweight convolutional neural network to obtain a multimodal fusion weight map corresponding to the multimodal multi-scale pixel-level feature map. For example, for the task of determining the benignity or malignancy of small liver cancer lesions, this lightweight convolutional neural network may assign higher weights to the metabolic features of PET or CT, since metabolic activity is an important indicator for determining the benignity or malignancy of tumors.

[0041] The term "dynamic" here refers to the fact that the fusion weight map is dynamically generated according to the needs of the current prediction task, and the fusion weight map is used to guide the subsequent fusion process of multimodal, multi-scale pixel-level feature maps, thereby adaptively focusing on the modal information that contributes most to the identification of small lesions.

[0042] The pixel-level fusion module 130 is used to fuse the multi-scale pixel-level feature map of the multi-modality according to the multi-scale pixel-level feature map of the multi-modality and the corresponding multi-modality fusion weight map to obtain the fused pixel-level feature map.

[0043] In this embodiment, the pixel-level fusion module 130 performs deep fusion of the multi-modal multi-scale pixel-level feature maps based on the alignment results of the multi-modal medical image data, the multi-scale pixel-level feature maps, and the task-specific fusion weight map. The pixel-level fusion module 130 receives the multi-scale pixel-level feature maps after pixel-level alignment and performs weighted fusion based on the dynamically generated multi-modal fusion weight map. Specifically, it can employ multi-layer convolution, residual connections, or a pixel-level fusion module 130 based on a Transformer deep learning network to capture complex interactions and long-distance dependencies between pixels.

[0044] Taking HRCT and WSI multimodal medical imaging data as examples, spatially aligned HRCT multi-scale pixel-level feature maps, WSI multi-scale pixel-level feature maps, and dynamically generated fusion weight maps are input into a pixel-level fusion network. This pixel-level fusion network can be based on a Transformer deep learning network; for example, it can be designed as a multi-head self-attention mechanism fusion network, where each "head" is responsible for processing the multi-scale pixel-level feature map of one modality, and interaction between modalities is achieved through a cross-attention mechanism. When calculating the attention weights, relevant information from the multimodal fusion weight map is incorporated, enabling adaptive emphasis on the modality information that contributes most to the identification of small lesions based on the type of prediction task. The fused pixel-level feature map will contain complementary small lesion information from the HRCT and WSI data at the pixel level.

[0045] The prediction module 140 is used to predict small lesions in medical image data based on the fused pixel-level feature map through a pre-trained prediction network, and to quantify the uncertainty of the prediction results at the pixel level, and output the prediction results of the small lesions and the corresponding pixel-level uncertainty map.

[0046] In this embodiment, based on the fused pixel-level feature map, the prediction of small lesions in medical image data is performed according to different prediction tasks. For example, under the condition that the task is lesion segmentation, pixel-level lesion region prediction results are obtained; under the condition that the task is lesion classification, benign or malignant probability prediction results are obtained; or under the condition that the task is lesion regression, lesion size, location, etc. are obtained. Specifically, the fused pixel-level feature map can be input into a pixel-level prediction network constructed by a convolutional neural network or a Transformer deep learning network, so as to output the nodule probability map of each pixel under the condition that the task is lesion segmentation, or output the benign or malignant probability of the entire nodule under the condition that the task is lesion classification.

[0047] This application embodiment not only obtains the prediction result, but also performs pixel-level uncertainty quantification on the prediction result. Optionally, the fused pixel-level feature map is propagated multiple times through a pre-trained prediction network. Each forward propagation yields a set of pixel-level prediction results. The prediction result of the micro lesion is obtained by averaging the multiple sets of pixel-level prediction results. The variance or entropy of the multiple sets of pixel-level prediction results is calculated to obtain the corresponding pixel-level uncertainty map.

[0048] This application integrates Bayesian deep learning methods, such as Monte Carlo Dropout, Bayesian Neural Networks, or deep ensemble learning algorithms, into a pixel-level prediction network. By performing multiple forward propagations during the inference phase, the probability distribution of the prediction result for each pixel is obtained, thereby quantifying the uncertainty of the prediction result and outputting a pixel-level uncertainty map, such as a heatmap, corresponding to the prediction result. For example, to quantify the uncertainty of the prediction result, this application embodiment can employ the Monte Carlo Dropout method. During the training phase, Dropout layers are introduced into multiple layers of the prediction network. During the inference phase, Dropout is not disabled; instead, multiple forward propagations are performed on the same medical image data, such as T=30 times. Each forward propagation yields a pixel-level prediction result. The mean of these T prediction results is calculated as the prediction result for the microlesion, and the variance or entropy of the T prediction results is calculated as the pixel-level uncertainty. The larger the variance, the more uncertain the prediction result at that pixel location, thus ultimately generating the prediction result for the microlesion and the corresponding pixel-level uncertainty map.

[0049] This embodiment of a medical image data analysis system based on pixel-level multimodal fusion solves the problem of heterogeneous medical image data being unable to be accurately aligned and deeply fused by using a pixel-level feature extraction module 110, a dynamic weight generation module 120, a pixel-level fusion module 130, and a prediction module 140. It achieves the technical effect of fully utilizing the different characteristics of multimodal medical image data to complement each other's advantages.

[0050] The technical solution described in this application uses a deformable registration network to perform pixel-level spatial alignment of preprocessed multimodal medical image data and extract multimodal multi-scale pixel-level feature maps, providing a pixel-level feature extraction and fusion scheme with higher fusion accuracy and stronger ability to identify small lesions. By generating a multimodal fusion weight map corresponding to the multimodal multi-scale pixel-level feature maps, different weights are matched according to different prediction task types of medical image data, thereby achieving complementary advantages of different types of medical image data. By performing pixel-level uncertainty quantification on the prediction results, the technical problem of missing uncertainty quantification is solved, thereby improving the reliability of clinical decision-making.

[0051] Example 2 Figure 2 The figure shows a schematic diagram of a medical image data analysis system based on pixel-level multimodal fusion provided in Embodiment 2 of this application. Compared with Embodiment 1, the medical image data analysis system based on pixel-level multimodal fusion in this embodiment adds a multimodal image preprocessing module 150, a text and structured data processing module 160, and a result output module 170.

[0052] The multimodal image preprocessing module 150 is used to standardize, denoise, and enhance the original multimodal medical image data to obtain structured multimodal medical image data, and to clean and encode the structured multimodal medical image data to obtain preprocessed multimodal medical image data. The original multimodal medical image data includes at least two different modalities of medical image data.

[0053] In this embodiment, the multimodal image preprocessing module 150 can receive raw multimodal medical image data such as HRCT, WSI, PET, CT, and MRI from a Picture Archiving and Communication System (PACS) or manually uploaded data, for example, acquiring the patient's HRCT and WSI data. It can also receive structured data and text data from an electronic medical record system. The module performs format conversion on the raw multimodal medical image data, such as converting DICOM format to NIFTI or PNG format, and performs operations such as denoising, artifact removal, intensity normalization, and image enhancement to obtain structured multimodal medical image data. Finally, it performs missing value imputation, outlier handling, and feature normalization on the structured multimodal medical image data to obtain preprocessed multimodal medical image data, which is then stored in a distributed file system or database.

[0054] The text and structured data processing module 160 is used to clean and encode structured data or text data to obtain preprocessed structured and text data. For example, it performs natural language processing such as word segmentation, word vector embedding, and entity recognition on text data to obtain preprocessed text data. It also extracts feature vectors from the preprocessed structured and text data to obtain structured and text data feature vectors. For example, it obtains feature vectors through MLP networks or pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) and Bio-ClinicalBERT.

[0055] The results output module 170 is used to interactively display the lesion area, uncertainty heatmap and multimodal fusion weights based on the prediction results of small lesions and the corresponding pixel-level uncertainty map, and to display non-medical image data based on the feature vectors of structured and textual data.

[0056] In this embodiment, the result output module 170 can output a structured diagnostic report and risk warning based on the prediction results and the corresponding pixel-level uncertainty map. For example, when a small lesion is unclear or difficult to diagnose, and the prediction uncertainty exceeds a preset threshold, an alarm is issued and the doctor is advised to perform a second interpretation or combine more clinical information. When the overall prediction uncertainty exceeds the threshold, the system issues an alarm and suggests multidisciplinary consultation or further examination. Specifically, taking the prediction task of determining the benign or malignant nature of small lung nodules as an example, the result output module 170 can output the segmentation results of small lung nodules by highlighting the nodule area, and can also output the benign or malignant classification results expressed in probability values ​​and a pixel-level uncertainty heatmap. It can also selectively output the original medical image data, registration results, modality fusion weight map, etc. Doctors can intuitively see the location, size, benign or malignant nature of the small lesions, as well as the system's confidence level in predicting each pixel through a visual interface, and can interactively view slices of different original medical image data and adjust the uncertainty threshold.

[0057] The structured and textual data feature vectors obtained through the text and structured data processing module 160 do not directly participate in the process of obtaining the fused pixel-level feature map. Instead, they are used as independent, high-level auxiliary information, combined with the prediction results of small lesions and the corresponding pixel-level uncertainty map, to provide doctors with comprehensive diagnostic basis and uncertainty assessment.

[0058] The technical solution described in this application improves the processing flow of original multimodal medical image data by adding a multimodal image preprocessing module 150, a text and structured data processing module 160, and a result output module 170. It also increases the consideration of structured data or text data, further improving the diagnostic accuracy of small lesions and the reliability of clinical decision-making.

[0059] Optionally, the pixel-level fusion module further includes: a weak signal enhancement submodule, used to construct a histogram equalization mapping function by applying adaptive histogram equalization (AHE) to the small, low-contrast regions of the fused pixel-level feature map, and to perform specific enhancement using the mapping function; or to perform specific enhancement by applying local contrast stretching (LCS).

[0060] For regions with low signal intensity or low contrast in the fused pixel-level feature map, such as micronodule regions, techniques such as adaptive histogram equalization or local contrast stretching are used to enhance the features of micro lesions, making them more prominent.

[0061] Optionally, the pixel-level fusion module further includes a context-aware attention submodule, which is used to calculate the correlation between each pixel and its neighborhood and the entire multi-scale pixel-level feature map of the multi-modal multi-scale pixel-level feature map through a context-aware attention mechanism, so as to enhance the feature expression of small lesions and suppress background noise.

[0062] The context-aware attention submodule not only focuses on the local features of the microlesion region, but also considers global contextual information such as the surrounding tissue structure and blood vessel distribution. For example, by learning a spatial attention map through a convolutional attention network, attention is focused on the region that may contain micronodules and background noise is suppressed. At the same time, "nodule prior" attention is introduced, which uses existing nodule knowledge, such as nodule size and shape range, to guide the attention mechanism and further amplify the signal of micronodules.

[0063] This embodiment further enhances the signal of small lesions in the fused pixel-level feature map by setting a weak signal enhancement submodule and a context-aware attention submodule in the pixel-level fusion module 130, thereby improving the accuracy and reliability of the prediction results of small lesions. It is especially suitable for scenarios with sparse data or small lesions, and provides more reliable auxiliary decision support for clinical diagnosis and treatment.

[0064] The pixel-level multimodal fusion medical image data analysis method of this application is implemented based on the software modules and hardware collaboration described in the above embodiments. The software modules can be designed based on a microservice architecture, and each module can be deployed and extended independently, communicating through an application programming interface (API). The main hardware may include: a processor (Central Processing Unit, CPU), such as a multi-core high-performance CPU, for data preprocessing, task scheduling, and some model inference; a graphics processing unit (GPU), such as an NVIDIA Tesla V100 or A100, for training and inference of deep learning models, especially computationally intensive tasks such as pixel-level feature extraction, fusion, and uncertainty quantization; random access memory (RAM) to meet the needs of processing high-resolution medical image data; storage devices, such as high-speed solid-state disk (SSD) arrays, for storing massive amounts of multimodal medical image data, parameters, and intermediate results, and providing fast read and write capabilities; and a network interface for data transmission and communication.

[0065] It is worth noting that the various units and modules included in the above embodiments are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0066] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A medical image data analysis system based on pixel-level multimodal fusion, characterized in that, The system includes: A pixel-level feature extraction module is used to perform pixel-level spatial alignment of preprocessed multimodal medical image data through a deformable registration network to obtain aligned multimodal medical image data, and to extract multimodal multi-scale pixel-level feature maps from the aligned multimodal medical image data through a multi-scale feature extraction network. The pixel-level feature maps include pixel-level spatial information of the preprocessed multimodal medical image data. The dynamic weight generation module is used to receive the predicted task label, generate a task embedding vector based on the predicted task label, and generate a multimodal fusion weight map corresponding to the multimodal multi-scale pixel-level feature map through the task embedding vector and the multilayer perceptron network. Each pixel value of the fusion weight map represents the contribution of the corresponding modality to the current task at that pixel position. The pixel-level fusion module is used to fuse the multi-scale pixel-level feature map of the multi-modality based on the multi-scale pixel-level feature map of the multi-modality and the corresponding multi-modality fusion weight map to obtain the fused pixel-level feature map. The prediction module is used to predict small lesions in medical image data based on the fused pixel-level feature map through a pre-trained prediction network, and to quantify the uncertainty of the prediction results at the pixel level, and output the prediction results of the small lesions and the corresponding pixel-level uncertainty map. The prediction module is further configured to: perform multiple forward propagations on the fused pixel-level feature map using a pre-trained prediction network, with each forward propagation yielding a set of pixel-level prediction results; calculate the mean of multiple sets of pixel-level prediction results to obtain the prediction result of the micro lesion; and calculate the variance or entropy of the multiple sets of pixel-level prediction results to obtain the corresponding pixel-level uncertainty map.

2. The medical image data analysis system based on pixel-level multimodal fusion according to claim 1, characterized in that, The pixel-level fusion module also includes: The weak signal enhancement submodule is used to construct a histogram equalization mapping function by adaptive histogram equalization (AHE) for small, low-contrast regions of the fused pixel-level feature map, and then use the mapping function for specific enhancement; or to use local contrast stretching (LCS) for specific enhancement.

3. The medical image data analysis system based on pixel-level multimodal fusion according to claim 1, characterized in that, The pixel-level fusion module also includes: The context-aware attention submodule is used to calculate the correlation between each pixel and its neighborhood and the entire multi-scale pixel-level feature map of the multimodal multi-scale pixel-level feature map through the context-aware attention mechanism, so as to enhance the feature expression of small lesions and suppress background noise.

4. The medical image data analysis system based on pixel-level multimodal fusion according to claim 1, characterized in that, The pixel-level feature extraction module also includes: The attention-guided submodule is used to combine the attention-guided mechanism to focus on small lesion areas when aligning multimodal medical image data through the deformable registration network.

5. The medical image data analysis system based on pixel-level multimodal fusion according to any one of claims 1-4, characterized in that, The preprocessed multimodal medical imaging data includes at least two of the following: computed tomography (CT) data, high-resolution computed tomography (HRCT) data, pathological whole slide image (WSI) data, and positron emission tomography (PET) data.

6. The medical image data analysis system based on pixel-level multimodal fusion according to claim 5, characterized in that, The preprocessed multimodal medical image data is preprocessed high-resolution computed tomography (HRCT) data; The pixel-level feature extraction module is used to extract multi-scale pixel-level features from preprocessed HRCT data using 3D U-Net or 3D ResNet as the multi-scale feature extraction network. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

7. The medical image data analysis system based on pixel-level multimodal fusion according to claim 5, characterized in that, The preprocessed multimodal medical image data is preprocessed pathological whole slide image WSI data; Accordingly, the pixel-level feature extraction module is also used to segment the preprocessed WSI data to obtain multiple WSI data blocks, extract multi-scale features from the multiple WSI data blocks through the multi-scale feature extraction network, and organize the multi-scale features of the multiple WSI data blocks into multi-scale pixel-level features. The multi-scale pixel-level features include pixel-level features ranging from low-level texture to high-level semantics.

8. The medical image data analysis system based on pixel-level multimodal fusion according to any one of claims 1-4, characterized in that, Also includes: The multimodal image preprocessing module is used to standardize, denoise, and enhance the original multimodal medical image data to obtain structured multimodal medical image data, and to clean and encode the structured multimodal medical image data to obtain preprocessed multimodal medical image data. The original multimodal medical image data includes at least two different modalities of medical image data. The text and structured data processing module is used to clean and encode structured or text data to obtain preprocessed structured and text data; and to extract feature vectors from the preprocessed structured and text data to obtain structured and text data feature vectors. The results output module is used to interactively display the lesion area, uncertainty heatmap and multimodal fusion weights based on the prediction results of small lesions and the corresponding pixel-level uncertainty map, and to display non-medical image data based on the feature vectors of structured and textual data.

9. The medical image data analysis system based on pixel-level multimodal fusion according to any one of claims 1-4, characterized in that, The prediction task labels include at least one of the following: lesion segmentation task, benign / malignant determination task, and treatment response prediction task.

Images