Multimodal medical image processing method based on importance perception alignment and selective fusion

Abstract

The application discloses a kind of based on importance perception alignment and selective fusion multimodal medical image processing method, including the automatic reading and pre-processing of paired multimodal medical image acquisition, obtain standardized tensor data;Multi-modal high-dimensional deep features are extracted by encoder, importance perception local alignment is performed by cross-modal grafting module, bidirectional cross-modal attention mechanism is used and combined with self-attention and cross-modal attention double-branch structure, and then channel level selective grafting fusion is obtained to obtain reconstruction features, finally, lesion segmentation prediction map is output by decoder.This application uses bidirectional cross-modal attention and modal self-attention double-branch structure to improve alignment accuracy, efficient feature fusion is completed by replacing preset proportion channel, combined with double loss function to complete model training, which can effectively improve the alignment accuracy and fusion effect of multimodal medical image features, enhance the accuracy and stability of model training of lesion segmentation.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a multimodal medical image processing method based on importance-aware alignment and selective fusion. Background Technology

[0002] In modern precision medicine and clinical diagnosis, doctors often need to comprehensively analyze multimodal medical images with complementary diagnostic information to accurately locate lesions within a patient's body. For example, computed tomography (CT) provides high-resolution anatomical boundaries, positron emission tomography (PET) reflects abnormal glucose metabolism in target tissues, and different sequences of magnetic resonance imaging (MRI), such as diffusion-weighted imaging (DWI) and fluid attenuated inversion recovery (FLAIR), can sensitively detect acute ischemia and chronic lesions, respectively. In recent years, deep learning algorithms have been widely applied to computer-aided diagnosis (CAD) of multimodal medical images, with the core being how to effectively extract and utilize heterogeneous complementary clinical information between different modalities.

[0003] However, existing multimodal medical image processing methods still face significant technical bottlenecks in clinical applications: 1. Over-alignment of modal features can easily lead to missed diagnoses of minute lesions. Current multimodal alignment methods often use global consistency loss to constrain the feature distribution between modalities. However, different medical modalities have fundamentally different imaging mechanisms, and high-value complementary pathological information (such as high metabolic signal points unique to PET) is concentrated in modal-specific differences. Forced global alignment assimilates specific pathological features into the normal tissue background, resulting in the loss of crucial diagnostic information. In clinical practice, this can easily mask the distinguishing signals between minute tumors and early infarcts, significantly increasing the risk of missed diagnoses.

[0004] 2. Dense feature fusion introduces noise, easily leading to false positives. Existing feature fusion methods often employ global dense fusion techniques such as channel splicing and element-wise addition. Inherent speckle noise, metallic artifacts, and irrelevant physiological high-metabolic backgrounds in medical images (such as physiological uptake from the brain or bladder in PET scans) can be directly mixed into the feature channels without screening. This not only fails to provide effective complementary information but also interferes with the expression of true lesion features, causing lesion boundary segmentation errors and false positive diagnoses.

[0005] 3. Scarcity of labeled data and the susceptibility of semi-supervised training to confirmation bias. Accurate labeling of lesions in medical imaging heavily relies on specialists; voxel-level labeling is costly and time-consuming, and high-quality multimodal labeled data is severely lacking. Existing semi-supervised learning methods are susceptible to fusion noise, leading to the generation of low-quality, structurally flawed pseudo-labels by the teacher network, thus creating confirmation bias and reducing model segmentation accuracy and clinical reliability.

[0006] In summary, achieving accurate anatomical structure alignment, preservation of high-value pathological features, and removal of interference noise in multimodal medical image processing has become crucial for improving the lesion segmentation accuracy and clinical applicability of medical AI models. To address this, this invention proposes a multimodal medical image processing method based on importance-aware alignment and selective fusion to overcome the aforementioned shortcomings of existing technologies. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the present invention at least solves the following technical problems: 1. In multimodal medical image processing, over-alignment of modal features and dense fusion noise lead to insufficient lesion segmentation accuracy and low recognition rate of small lesions; 2. High-quality labeled clinical data is scarce, and existing methods have unstable training performance under conditions of a small number of labeled samples, making it difficult for segmentation performance to meet clinical needs; 3. Multimodal image segmentation models have high computational complexity and slow inference speed, making it difficult to meet the actual requirements of real-time clinical processing and generalized deployment.

[0008] To achieve the above objectives, this invention discloses a multimodal medical image processing method based on importance-aware alignment and selective fusion, the method comprising the following steps: S1: Automatically read and preprocess the acquired paired multimodal medical images to obtain standardized multimodal tensor data; S2: Multi-scale feature extraction is performed on the multimodal tensor data by the encoder to obtain high-dimensional deep features of each modality; S3: Input the high-dimensional deep features into the cross-modal grafting module, perform importance-aware local alignment, and obtain aligned features; S4: Perform channel-level selective grafting and fusion of the alignment features to obtain reconstructed fused features; S5: Input the reconstructed fusion features into the decoder and output the lesion segmentation prediction map.

[0009] Furthermore, the importance-aware local alignment employs a bidirectional cross-modal attention mechanism, with the cross-modal attention weights calculated according to the following formula. ,in, For modality The query vector; For modality The key vector; For modality The value vector; For attention head dimension.

[0010] Furthermore, the alignment feature is composed of intramodal self-attention branches and cross-modal attention branches.

[0011] Furthermore, the importance-aware local alignment uses the dynamic allocation of cross-modal attention weights to strongly align semantically consistent regions while preserving difference information in modality-specific regions.

[0012] Furthermore, the selective grafting fusion includes: Constructing cross-modal cross-correlation matrix With the intramodal autocorrelation matrix ; According to the cross-modal cross-correlation scores, the channels with the highest scores are selected as high-value donor channels, and the remaining channels are selected as low-value recipient channels to be replaced. The low-value receptor channel is removed and replaced one-to-one with the high-value donor channel to complete the channel grafting process.

[0013] Furthermore, the preset ratio is 20%.

[0014] Furthermore, the method employs a teacher-student collaborative semi-supervised architecture, where teacher network parameters are updated using an exponential moving average strategy:

[0015] in: For teacher network parameters; For student network parameters; This is the smoothing coefficient.

[0016] Furthermore, the model is trained using the aforementioned teacher-student collaborative semi-supervised architecture. The model training process includes: For labeled data, the supervision loss is calculated by combining cross-entropy loss and Dice loss. ; For unlabeled data, the pseudo-labels generated by the teacher network are used as supervision signals to calculate the consistency loss. .

[0017] Furthermore, the preprocessing includes performing rigid registration, voxel resampling, image cropping, and normalization on the multimodal image, wherein the image cropping is to remove pure background regions.

[0018] Furthermore, the voxel resampling is isotropic voxel resampling.

[0019] This invention achieves at least the following beneficial technical effects: 1. Achieve adaptive local anatomical alignment, ensuring consistency in the macroscopic structure of organs while fully preserving the specific pathological signals required for segmenting micro-lesions, significantly reducing the rate of missed diagnosis of occult lesions; 2. Achieve selective feature grafting by "removing the weakest and introducing the best", effectively suppressing metal artifacts and interference from normal physiological hypermetabolism, and improving the model's ability to distinguish the real target area in complex anatomical backgrounds; 3. Even with an extremely low annotation ratio (such as only 10% expert annotation), it can still achieve lesion segmentation accuracy close to that of full supervision, significantly reducing the R&D cost of medical AI and the threshold for implementation in hospitals at all levels.

[0020] At the same time, this invention has strong practicality and industrial application prospects: In precision medicine, multimodal medical imaging (such as PET / CT and multi-sequence MRI) has become an important tool for delineating target areas of malignant tumors and assessing the ischemic penumbra in stroke. This invention can complete model learning using only a small amount of labeled data (requiring only 10% labeled data) and relying on a massive amount of historical unlabeled images from hospitals, significantly reducing labeling costs and shortening the development cycle. This invention demonstrates excellent generalization ability, outperforming existing methods in a wide range of medical imaging tasks, such as ischemic stroke (ISLES 2022) and systemic tumors (AutoPET-IV). Even under extreme conditions with only 10% labeled data, this method still achieves a Dice coefficient of 67.53% (stroke task), significantly surpassing existing semi-supervised comparison methods, and exhibits excellent boundary localization accuracy and false positive noise suppression capabilities. The Cross Modality Grafting (CMG) module used in this invention is a lightweight, plug-and-play structure that does not require changes to the existing encoder-decoder infrastructure. The overall model parameter size remains at a lightweight level of 18.43M, and the single-pass multimodal feature fusion and inference time is as low as 5.81 milliseconds. It does not rely on top-tier medical supercomputers and can be directly deployed in existing Picture Archiving and Communications Systems (PACS) and conventional image reading workstations in hospitals at all levels. It does not rely on high-performance computing equipment, can meet the needs of real-time clinical processing, and has excellent clinical adaptability and promotional value. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the multimodal medical image processing method based on importance-aware alignment and selective fusion of the present invention. Figure 2This is a schematic diagram of the overall architecture of the multimodal medical image processing method based on a teacher-student collaborative architecture of the present invention; Figure 3 This is a schematic diagram illustrating the workflow of the cross-modal grafting module of the present invention; Figure 4 This is a flowchart illustrating the multimodal medical image clinical reasoning processing method of the present invention. Detailed Implementation

[0022] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0023] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0024] This invention proposes a multimodal semi-supervised image processing method based on importance-aware alignment and selective fusion. Its core lies in utilizing the Cross Modality Grafting (CMG) principle to train a deep learning model capable of accurately extracting and fusing complementary multimodal features through a combination of self-supervised and semi-supervised methods. Figure 1 As shown, the method includes the following steps: S1: Automatically read and preprocess the acquired paired multimodal medical images to obtain standardized multimodal tensor data; S2: Multi-scale feature extraction is performed on the multimodal tensor data by the encoder to obtain high-dimensional deep features of each modality; S3: Input the high-dimensional deep features into the cross-modal grafting module, perform importance-aware local alignment, and obtain aligned features; S4: Perform channel-level selective grafting and fusion of the alignment features to obtain reconstructed fused features; S5: Input the reconstructed fusion features into the decoder and output the lesion segmentation prediction map.

[0025] The implementation process of the method of the present invention will be described in detail below through specific embodiments.

[0026] Example 1

[0027] like Figure 2-3 As shown, this embodiment implements multimodal image segmentation under a teacher-student collaborative semi-supervised architecture. The specific steps are as follows: Step 1: Data Acquisition and Preprocessing. Acquire paired multimodal medical images, such as positron emission tomography (PET) and computed tomography (CT) images, diffusion-weighted imaging (DWI) and fluid attenuation inversion recovery sequence (FLAIR) images. Perform rigid registration, isotropic voxel resampling, image cropping, and normalization on the images, removing pure background regions to obtain standardized multimodal tensor data. In a semi-supervised setting, the dataset is divided into a small portion (e.g., 10% or 20%) of "labeled dataset" and a large portion of "unlabeled dataset".

[0028] Step 2: Multi-scale Feature Extraction. Construct a teacher-student collaborative network, with the student network having the same structure as the teacher network. Input paired multimodal tensor data into the encoder, and extract the modalities through continuous convolution and downsampling operations at the network bottleneck layer. High-dimensional deep features With mode High-dimensional deep features ,like Figure 3 As shown.

[0029] Step 3: Importance-aware local alignment based on Bidirectional Query-guided Attention (BQA). This involves aligning features... , Input bidirectional query guides the attention module, such as Figure 3 As shown: 3.1 Generate query, key, and value vectors, and generate query vectors for each modality by sharing a linear projection matrix. , ), key vector ( , ), value vector ( , ); 3.2 Bidirectional cross-modal attention interaction: Alignment features are jointly constructed by intra-modal self-attention branches and cross-modal attention branches, using modal... Query To retrieve the mode The key Sum The calculation formula is: ,in, For modality query vectors For modality key vector, For modality value vector For attention head dimension; 3.3 Adaptive alignment: Alignment strength is dynamically allocated through attention weights, strong alignment is applied to semantically consistent regions, modality-specific pathological information is preserved, and alignment features are output. , .

[0030] Step 4: Selective grafting and fusion based on Co-occurrence Perception Channel Replacement (CCR) to align features. and Input co-occurrence sensing channel replacement module, such as Figure 3 As shown: 4.1 Constructing the cross-modal cross-correlation matrix With the intramodal autocorrelation matrix New feature vectors are generated through convolution operations, and a cross-modal cross-correlation matrix is ​​constructed using dot product operations. and intramodal autocorrelation matrix ; 4.2 According to The maximum value is taken along the column direction. The channels are sorted from high to low according to the cross-modal cross-correlation scores. The channel with the highest score and a preset proportion (preferably 20%) is selected as the high-value donor channel, and the remaining channels are selected as low-value recipient channels to be replaced. According to the intramodal autocorrelation matrix, the recipient channels with the lowest intramodal autocorrelation scores and the same proportion are selected as low-value isolated noise channels to be replaced. 4.3 Remove the channels marked as low value in the receptor features and replace them one-to-one with the selected high-value donor channels to generate the reconstructed fusion features. and .

[0031] Step 5: Decode the output and iteratively update the semi-supervised model. Input the reconstructed and fused features into the decoder, and output the lesion segmentation prediction map. Model training is as follows: Figure 2 As shown, a teacher-student collaborative semi-supervised architecture is adopted, where the student network simultaneously receives labeled and unlabeled data to calculate the loss; for labeled data, a combination of cross-entropy loss and Dice loss is used to calculate the supervised loss. For unlabeled data, pseudo-labels generated by the teacher network are used as supervision signals to calculate the consistency loss. The parameters of the teacher network do not participate in direct backpropagation. Instead, they are updated smoothly based on the exponential moving average (EMA) of the historical weights of the student network. The update formula is as follows: ,in, For teacher network parameters, For student network parameters, To smooth out the coefficients, high-confidence pseudo-labels are output based on the robust features of the cross-modal grafting module, thus mitigating confirmation bias in semi-supervised learning.

[0032] Example 2

[0033] Combination Figure 4 As shown, this embodiment provides a multimodal medical image clinical inference processing method based on a trained model, specifically including: 1. Automatically acquire and parse paired multimodal medical images of patients, read image data in Digital Imaging and Communications in Medicine (DICOM) format based on the Medical Image Archive and Communication System and Radiology Information System (RIS), and perform spatial rigid registration and voxel resampling preprocessing on the image data; 2. The preprocessed paired multimodal images are input into a lightweight deep learning inference model. The importance-aware local alignment and channel-level selective grafting fusion are performed through the cross-modal grafting module, which effectively removes normal physiological high metabolic noise and imaging artifacts, and completes the accurate extraction and fusion of lesion features. 3. Visualize the lesion segmentation results output by the model, that is, convert the high-precision lesion mask into a clinically visualized color region of interest (ROI), highlight the lesion outline on the three-dimensional orthogonal views of cross section, coronal plane and sagittal plane, provide early warning for small lesions, provide physicians with fine-tuning and confirmation tools, improve diagnostic efficiency and reduce the rate of missed diagnosis.

[0034] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0035] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal medical image processing method based on importance-aware alignment and selective fusion, characterized in that, The method includes the following steps: S1: Automatically read and preprocess the acquired paired multimodal medical images to obtain standardized multimodal tensor data; S2: Multi-scale feature extraction is performed on the multimodal tensor data by the encoder to obtain high-dimensional deep features of each modality; S3: Input the high-dimensional deep features into the cross-modal grafting module, perform importance-aware local alignment, and obtain aligned features; S4: Perform channel-level selective grafting and fusion of the alignment features to obtain reconstructed fused features; S5: Input the reconstructed fusion features into the decoder and output the lesion segmentation prediction map.

2. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 1, wherein the importance-aware local alignment adopts a bidirectional cross-modal attention mechanism, and the cross-modal attention weight is calculated according to the following formula. ,in, For modality The query vector; For modality The key vector; For modality The value vector; For attention head dimension.

3. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 2, wherein the alignment feature is composed of intramodal self-attention branch and cross-modal attention branch.

4. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 3, wherein the importance-aware local alignment is dynamically allocated through the cross-modal attention weights to strongly align semantically consistent regions and retain differential information in modality-specific regions.

5. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 1, wherein the selective grafting fusion includes: Constructing cross-modal cross-correlation matrix With the intramodal autocorrelation matrix ; According to the cross-modal cross-correlation scores, the channels with the highest scores are selected as high-value donor channels, and the remaining channels are selected as low-value recipient channels to be replaced. The low-value receptor channel is removed and replaced one-to-one with the high-value donor channel to complete the channel grafting process.

6. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 5, wherein the preset ratio is 20%.

7. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 1, wherein the method adopts a teacher-student collaborative semi-supervised architecture, and the teacher network parameters are updated through an exponential moving average strategy: in: For teacher network parameters; For student network parameters; This is the smoothing coefficient.

8. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 7, wherein the model training is performed using the teacher-student collaborative semi-supervised architecture, the model training process comprising: For labeled data, the supervision loss is calculated by combining cross-entropy loss and Dice loss. ; For unlabeled data, the pseudo-labels generated by the teacher network are used as supervision signals to calculate the consistency loss. .

9. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 1, wherein the preprocessing includes performing rigid registration, voxel resampling, image cropping and normalization on the multimodal image, wherein the image cropping is to remove pure background regions.

10. The multimodal medical image processing method based on importance-aware alignment and selective fusion according to claim 9, characterized in that, The voxel resampling is isotropic voxel resampling.

Images