A progressive multimodal medical image fusion method based on multiple scales

Abstract

A progressive multimodal medical image fusion method based on multiple scales is disclosed. This invention relates to the field of multimodal medical image fusion technology, specifically a progressive multimodal medical image fusion method based on multiple scales. Through an integrated design of "multi-scale extraction - hierarchical encoding - layer-by-layer fusion - adaptive modeling," this invention achieves efficient and accurate fusion of multimodal medical images, providing reliable technical support for clinical applications. The method includes the following steps: acquiring images of modalities A and B and preprocessing them; performing multimodal parallel encoding on the preprocessed images: multimodal parallel encoding includes four encoding stages; the input of each encoding stage sequentially passes through an adaptive visual RWKV module, an efficient cross-modal module, and a progressive fusion module to obtain the output of the multimodal parallel encoding; and then, through image conversion, obtains the RGB fused image.

Description

Technical Field

[0001] This invention relates to the field of multimodal medical image fusion technology, and specifically to a progressive multimodal medical image fusion method based on multiple scales. Background Technology

[0002] In modern clinical medicine, multimodal medical image fusion technology plays an indispensable role. Medical images from different modalities can provide complementary information about human tissues and organs. For example, MRI images excel in soft tissue contrast, clearly presenting the anatomical structures of soft tissues such as brain tissue and muscles, while PET images accurately reflect tissue metabolic activity, aiding in the early screening of lesions such as tumors. PET has high sensitivity and is suitable for metabolic and molecular imaging; while SPECT is relatively inexpensive and uses a wide variety of tracers, making it widely used for cardiac and cerebral blood flow perfusion imaging. Through multimodal image fusion, image information from different sources can be organically integrated, compensating for the information limitations of single-modal images and providing richer decision support for clinical diagnosis, treatment planning, and efficacy evaluation.

[0003] Despite some progress in multimodal medical image fusion technology, existing solutions still suffer from several core shortcomings, failing to meet clinical demands for high-precision and efficient fusion results. First, cross-modal feature misalignment is a significant issue. Most existing methods employ a "single-stage fusion" approach, fusing multimodal features only in the final stage of the network encoding process. The lack of effective semantic association and spatial alignment during the initial independent encoding of multimodal features leads to detail loss, artifact interference, and semantic misalignment in the fused image, severely impacting the accuracy of clinical diagnosis. Second, long-distance dependency modeling is prohibitively expensive. Traditional Transformer-type models rely on self-attention mechanisms to capture global relationships between features, but their computational complexity is O(n^2) (where n is the number of image pixels). When processing high-resolution medical images of 512×512 and above, the computational load increases exponentially, requiring substantial hardware resources and resulting in slow inference speeds, making it difficult to meet the needs of real-time clinical diagnosis.

[0004] Furthermore, the fusion module lacks adaptability. Existing fusion schemes mostly employ fixed-weight fusion strategies, failing to dynamically adjust fusion weights based on the importance of different modal features and the feature differences of image regions. This makes it difficult to adapt to complex and ever-changing medical image scenarios, resulting in limited fusion performance. Finally, the ability to balance local and global features is insufficient. Existing models often focus on single-scale feature extraction, either overemphasizing local fine details while neglecting global structural information, or emphasizing global features while losing key anatomical details. However, the diagnosis of medical images requires both accurate local lesion information and complete global structural information, and this single-scale feature extraction method cannot meet both needs. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to propose a progressive multimodal medical image fusion method based on multiple scales. Through an integrated design of "multi-scale extraction - hierarchical encoding - layer-by-layer fusion - adaptive modeling," efficient and accurate fusion of multimodal medical images is achieved, providing reliable technical support for clinical applications.

[0006] The method includes the following steps:

[0007] S1. Obtain the grayscale image of mode A and the RGB image of mode B, and perform preprocessing on each;

[0008] The RGB image of the preprocessed mode B is then converted into Y channel, Cr channel and Cb channel;

[0009] S2. Perform multimodal parallel encoding on the preprocessed grayscale image of mode A and the Y channel of mode B:

[0010] Multimodal parallel coding includes four coding stages;

[0011] The input at each encoding stage passes sequentially through the adaptive visual RWKV module, the cross-modal attention module, and the progressive fusion module to obtain the output of multimodal parallel encoding.

[0012] S3. The output of the multimodal parallel encoding is compressed into a single-channel luminance feature map after being decoded by the output layer. aisle;

[0013] S4, will The Cr and Cb channels are spliced ​​together, and the spliced ​​image is then transformed to obtain an RGB fused image.

[0014] Furthermore, the preprocessing includes normalization and size alignment.

[0015] Furthermore, in the first, second, and third encoding stages, the output of the progressive fusion module is downsampled through a convolutional module with a stride of 2;

[0016] The output of the first encoding stage is used as the input of the second encoding stage;

[0017] The output of the second encoding stage is used as the input of the third encoding stage;

[0018] The output of the third encoding stage is used as the input of the fourth encoding stage;

[0019] In the fourth encoding stage, the output of the progressive fusion module is subjected to the first upsampling process and then the first residual fusion is performed with the output of the third encoding stage.

[0020] The result of the first residual fusion is then subjected to a second upsampling process and then subjected to a second residual fusion with the output of the second encoding stage;

[0021] The result of the second residual fusion is then subjected to a third upsampling process and then subjected to a third residual fusion with the output of the first encoding stage;

[0022] The result of the third residual fusion is the output of multimodal parallel coding.

[0023] Furthermore, the workflow of the adaptive vision RWKV module is as follows:

[0024] The input of the adaptive vision RWKV module is processed by layer normalization, and then processed in parallel by channel branch and spatial branch to obtain channel blend features and spatial blend features. The channel blend features and spatial blend features are then concatenated to obtain modality A features and modality B features.

[0025] Furthermore, the channel branch, from input to output, includes: a channel mixing submodule, an RWKV state unit, an adaptive local shift module, and an adaptive feature fusion module.

[0026] The spatial branch, from input to output, includes: a spatial mixing submodule, an RWKV state unit, an adaptive local shift module, and an adaptive feature fusion module.

[0027] Furthermore, the calculation formula for the RWKV state element is as follows:

[0028] in, This represents the feature output at the current scale. This represents the feature output from the previous scale. This represents the feature input at the current scale. Represents a linear transformation function. express Activation function This represents the Sigmoid activation function. This indicates element-wise multiplication.

[0029] Furthermore, the workflow of the adaptive local shift module is as follows:

[0030] The input to the adaptive local shift module is first processed in parallel through 3×3 depthwise separable convolution, 5×5 depthwise separable convolution and 7×7 depthwise separable convolution to extract small, medium and large local features;

[0031] Small, medium, and large local features are then processed sequentially through global average pooling, two linear transformations, and the ReLU activation function to generate small, medium, and large attention weights.

[0032] The adaptive feature fusion module multiplies small, medium, and large local features with small, medium, and large attention weights element-wise to obtain fused features;

[0033] Fusion features include: channel blending features and spatial blending features.

[0034] Furthermore, the working process of the cross-modal attention module is as follows:

[0035] Modality A features and Modality B features are processed sequentially through layer normalization, linear mapping layer and bidirectional linear attention;

[0036] Linear mapping layer output: Query matrix of mode A Key matrix Sum matrix And the query matrix of mode B Key matrix Sum matrix ;

[0037] Bidirectional linear attention processing consists of two branches, in one branch, and The attention weights are calculated to obtain the attention guidance weights of modality B features on modality A features. Then, the attention guidance weights of modality B features on modality A features are residually connected with modality A features to output the modality A features after cross-modal interaction.

[0038] In another branch, and The attention weights are calculated to obtain the attention guidance weights of modality A features on modality B features. Then, the attention guidance weights of modality A features on modality B features are residually connected with modality B features to output the modality B features after cross-modal interaction.

[0039] Furthermore, the working process of the progressive fusion module is as follows:

[0040] After cross-modal interaction, the modal A features and the modal B features are concatenated in the channel dimension to obtain concatenated features. The concatenated features are then subjected to global average pooling, the first 1×1 convolution, the ReLU activation function, the second 1×1 convolution, and the Sigmoid activation function to obtain adaptive fusion weights.

[0041] The adaptive fusion weights and the concatenated features are sequentially multiplied element-wise and weighted summed to obtain preliminary fusion features.

[0042] The initial fusion features are processed sequentially through the first convolution, layer normalization, ReLU activation function, and the first convolution, and then residual fusion is performed with the initial fusion features to obtain the output of the progressive fusion module.

[0043] Furthermore, the decoding output layer includes a 1×1 convolution.

[0044] The beneficial effects of the method described in this invention are as follows:

[0045] (1) The method described in this invention inserts a cross-modal attention module (ECM Block) and a progressive feature fusion module after each layer of encoding, changing the traditional "single-stage fusion at the back end" mode. The above design enables multimodal features to begin semantic interaction (through the ECM Block) and adaptive fusion alignment in the early stages of encoding, achieving "early association and layer-by-layer alignment". It avoids semantic fragmentation and spatial deviation accumulation caused by independent encoding in the early stages from the source, so that the complementary information of different modalities (such as anatomical structure and functional metabolism) in the final fused image can accurately correspond in spatial position, effectively reducing detail loss and artifact interference.

[0046] (2) The adaptive vision module of this invention uses a state transfer mechanism with linear complexity (O(n)) to capture long-distance dependencies, replacing the traditional self-attention mechanism with a computational complexity of O(n²). Simultaneously, the cross-modal attention module employs bidirectional linear attention computation. These two designs significantly reduce computational load and memory consumption when processing medical images of 512×512 or even higher resolutions, thus enabling faster training and inference speeds on ordinary computing hardware and providing feasibility for real-time or near-real-time clinical diagnosis.

[0047] (3) The progressive feature fusion module of this invention generates channel-level adaptive fusion weights through global average pooling and lightweight convolution. These weights can be dynamically adjusted according to the actual importance of different regions and modal features in the input image. For example, in tumor regions, the network can automatically assign higher weights to PET features that highlight metabolic abnormalities; in normal anatomical regions, it assigns higher weights to MRI features that show clear boundaries. This data-driven adaptive fusion mechanism overcomes the limitations of fixed-weight fusion strategies, enabling the model to better adapt to diverse lesion morphologies and imaging conditions, and improving the robustness of the fusion effect.

[0048] (4) The method described in this invention achieves a balance between local and global aspects through a multi-scale collaborative architecture design. On the one hand, the adaptive local shift module uses 3×3, 5×5, and 7×7 multi-scale depthwise separable convolutions in parallel, specifically designed to accurately capture local features ranging from fine textures to medium-sized structures. On the other hand, the RWKV state unit is responsible for modeling the overall morphology of the organ and the global spatial relationship between the lesion and surrounding tissues. Furthermore, the downsampling-upsampling residual link throughout the process fuses the deeply fused global semantic features with the shallowly preserved detail features layer by layer during decoding, ensuring that the final output image has both the consistency of the overall structure and contains rich anatomical details.

[0049] (5) The method described in this invention emphasizes efficiency design. The parameter complexity of the progressive feature fusion module is only O(c) (c is the number of channels), which is much lower than that of complex multi-level fusion networks. The adaptive vision module uses linear attention, and the adaptive local shift module uses depthwise separable convolution and linear complexity modeling. This strategy of "lightweight modules, layer-by-layer use" enables the entire end-to-end network to maintain high performance while keeping the number of parameters and computational overhead low, which is conducive to the rapid deployment of the model and its clinical application. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method described in this invention;

[0051] Figure 2 This is a schematic diagram illustrating the working process of the adaptive vision RWKV module described in this invention.

[0052] Figure 3 This is a schematic diagram illustrating the working process of the cross-modal attention module described in this invention;

[0053] Figure 4 This is a schematic diagram illustrating the working process of the progressive fusion module described in this invention. Detailed Implementation

[0054] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1

[0056] This embodiment provides a progressive multimodal medical image fusion method based on multiple scales. The multimodal medical image fusion method of this embodiment adopts an end-to-end architecture of hierarchical encoding-residual fusion-decoding. The overall process follows the logical link of "input preprocessing-parallel encoding-cross-modal interaction-progressive fusion-upsampling residual recovery-decoding output". Each link is closely connected to ensure the integrity of feature transfer and the effectiveness of fusion.

[0057] The flowchart of the method is as follows: Figure 1 As shown, the method includes the following steps:

[0058] S1. Obtain the grayscale image of mode A and the RGB image of mode B, and perform preprocessing.

[0059] The RGB values ​​of the preprocessed mode B are then converted into Y, Cr, and Cb channels via image processing.

[0060] The input layer primarily receives two different modalities of medical images. One type is single-channel grayscale images, such as MRI images (modality A), with a size of [missing information]. The core advantage lies in its ability to clearly present tissue anatomical structures; another type is three-channel RGB images (modal B), such as PET images and SPECT images, with a size of... It can provide functional information such as tissue metabolism and blood flow. Indicates the height of the image. Indicates the width of the image.

[0061] The core objective of input preprocessing is to normalize, align, and convert the color space of multimodal images, providing standardized input data for subsequent network training and inference. First, for the input modality A (grayscale medical image, pixel value matrix denoted as...),... ), and mode B (RGB medical image, pixel value matrix denoted as ), and mode B (RGB medical image, pixel value matrix denoted as ). , (Representing the real number field), pixel value normalization is performed. A linear normalization formula is used to map all pixel values ​​to the [0,1] interval. The normalization formula is as follows:

[0062]

[0063] in, For the original image ( include: and )exist The pixel value of the location, , These are the minimum and maximum pixel values ​​of the image, respectively. This represents the normalized pixel value. This formula eliminates differences in pixel value ranges, preventing a single modality feature from dominating the entire network training process due to excessively large values, while also accelerating network convergence.

[0064] Subsequently, the two types of images are adjusted to the same resolution using a bilinear interpolation algorithm. In this embodiment, the resolution is uniformly adjusted to 512×512. The pixel calculation formula for bilinear interpolation is:

[0065]

[0066] in, For the target pixel coordinates, These are the coordinates of the four nearest pixels in the original image. and It is an index variable used to iterate through the four nearest neighbor pixels surrounding the target pixel. Indicates the index of adjacent pixels in the horizontal direction (x direction). Indicates the index of adjacent pixels in the vertical direction (y direction); for The corresponding original pixel values ​​are used to ensure the size consistency of multimodal features during subsequent feature extraction and fusion processes, and to avoid feature misalignment caused by size differences.

[0067] To avoid the impact of color distortion on the fusion effect, the preprocessed RGB image of mode B is first converted from the RGB color space to the YCrCb color space. This space can be separated into a luminance channel (Y channel) and chrominance channels (Cr channel and Cb channel). The luminance channel carries the core structure and detailed information of the image, while the chrominance channel is mainly responsible for color rendering. Based on this characteristic, this embodiment only performs subsequent feature encoding and fusion operations on the luminance channel. The Cr and Cb channels are stored in memory and concatenated with the fused luminance channel during the decoding stage to finally restore the RGB image. This ensures the fusion effect while avoiding color distortion.

[0068] S2. Perform multimodal parallel encoding on the preprocessed grayscale image of mode A and the Y channel of mode B:

[0069] Multimodal parallel encoding is the core of feature extraction in this embodiment. The brightness channels of modality A and modality B are respectively entered into two independent encoding branches. Each branch contains four encoding stages, each corresponding to a different feature resolution and number of channels, realizing progressive feature extraction from shallow to deep levels. The backbone unit of the encoding branch is the Adaptive Vision RWKV module (A-VR Block). Through a special structural design, this module can maintain linear computational complexity while taking into account the extraction of local details and global structural features, providing high-quality single-modal features for subsequent fusion operations.

[0070] After each encoding stage is completed, a cross-modal attention module (ECM Block) is set up. This module is responsible for cross-modal semantic interaction of the single-modal features output from the two encoding branches, breaking down information barriers between modalities and establishing preliminary cross-modal associations, laying the foundation for subsequent progressive fusion. Next, the progressive feature fusion module adaptively fuses and spatially aligns the multimodal features after cross-modal interaction. This module, as one of the core innovations of this invention, adopts a layer-by-layer insertion approach to achieve synchronous advancement of "encoding-fusion," fundamentally solving the feature misalignment problem caused by later fusion.

[0071] In the layered coding and fusion process (multimodal parallel coding), each stage progresses sequentially to achieve feature extraction, interaction, and fusion from shallow to deep levels. The following uses a complete 4-layer coding process as an example to explain the specific implementation process in detail.

[0072] The input at each encoding stage passes sequentially through the adaptive visual RWKV module, the cross-modal attention module, and the progressive fusion module to obtain the output of multimodal parallel encoding.

[0073] In the first, second and third encoding stages, the output of the progressive fusion module is also downsampled through a convolutional module with a stride of 2;

[0074] The output of the first encoding stage is used as the input of the second encoding stage;

[0075] The output of the second encoding stage is used as the input of the third encoding stage;

[0076] The output of the third encoding stage is used as the input of the fourth encoding stage;

[0077] In the fourth encoding stage, the output of the progressive fusion module is subjected to the first upsampling process and then the first residual fusion is performed with the output of the third encoding stage.

[0078] The result of the first residual fusion is then subjected to a second upsampling process and then subjected to a second residual fusion with the output of the second encoding stage;

[0079] The result of the second residual fusion is then subjected to a third upsampling process and then subjected to a third residual fusion with the output of the first encoding stage;

[0080] The result of the third residual fusion is the output of multimodal parallel coding.

[0081] The first layer encoding stage corresponds to an input image with an original resolution of 512×512 and 32 channels. First, the grayscale image of modality A and the Y channel of modality B are input to the Adaptive Vision RWKV Block (A-VR Block). The core calculation of this block is based on the RWKV state transfer formula:

[0082]

[0083] in, This represents the feature output at the current scale. This represents the feature output from the previous scale. This represents the feature input at the current scale. Represents a linear transformation function. express Activation function This represents the Sigmoid activation function. This indicates element-wise multiplication.

[0084] After channel mixing, spatial mixing, and adaptive local shifting operations within the module, the preliminary features of mode A are output. Preliminary characteristics of mode B The two types of features respectively carry local details and preliminary global structural information of two different modalities. Subsequently, and The input is fed into the cross-modal attention module, where it undergoes bidirectional cross-modal linear attention calculation. The linear attention weight formula is as follows:

[0085] in, Let A be the attention weights of mode A for mode B. , These are the query matrix for mode A and the key matrix for mode B, respectively. For the dimensions of the query key matrix, This represents the length of the feature sequence. Similarly, the attention weights of mode B on mode A are calculated. Thus, the features after interaction are obtained. ( , These are the value matrices for modalities A and B, respectively, enabling preliminary semantic association between the features of the two modalities. Next, the interacted features are input into the progressive fusion module. express The OK, express No. Transpose of a line.

[0086] The weight generation formula for the adaptive fusion stage is:

[0087]

[0088] in, This is a channel splicing feature. Represents aggregate functions, Indicates global average pooling. Represents a 1×1 convolution. For the Sigmoid function, For the generated channel-level adaptive weights, This represents the ReLU activation function. The initial fusion features are... After further convolution and residual connection in the feature alignment stage, the first layer of fused features is obtained. The dimensions remain 512×512×32. Finally, regarding... Perform a 3×3 convolution operation with a stride of 2, and the downsampling feature formula is: to obtain downsampling features This provides input for the second-level encoding stage.

[0089] The second layer encoding stage corresponds to a resolution of 256×256 and 64 channels. The features downsampled from the first layer... The adaptive visual RWKV module, with inputs to two separate encoding branches, is able to capture more abstract, deeper features due to the increased number of channels, outputting features of modality A. Features of Mode B All dimensions are 256×256×64. The "cross-modal attention interaction-progressive fusion" process is then repeated to obtain the second layer of fused features. Then, through convolution downsampling with a stride of 2, we obtain... (Size 128×128, number of channels 128), enter the third layer encoding stage.

[0090] The implementation process for the third and fourth layers of encoding is the same as the first two layers, only differing in resolution and number of channels. The third layer encoding stage has a resolution of 128×128 and 128 channels, ultimately downsampling features. The size is 64×64, and the number of channels is 256; the fourth layer encoding stage is the deepest encoding layer, with a resolution of 64×64 and 256 channels. After fusion, the deepest fused feature is obtained. This feature carries multimodal, deep semantic information and global structural information, providing a core basis for subsequent upsampling and recovery.

[0091] The adaptive vision RWKV module is designed to address the limitations of traditional RWKV models in directly adapting to 2D medical images and their weak local feature capture capabilities. It retains the core advantage of RWKV's linear computational complexity, achieving a feature extraction effect that combines linear complexity with superior local and global features. This module, as the core unit of the encoding branch, is integrated into every encoding stage. Its structure primarily comprises two sub-modules: Channel Mix and Spatial Mix. These two sub-modules work together, respectively responsible for feature modeling in the channel and spatial dimensions. Both integrate a normalization layer, RWKV state units, and an adaptive local shift module to ensure the stability and effectiveness of feature extraction.

[0092] The Spatial Mix module handles global attention computation. It obtains R by performing a four-way shift (Q-Shift) operation on the input tokens and then feeding them into three parallel linear layers. s K s V s The matrix is ​​then used to perform global information aggregation with linear complexity via a bidirectional WKV (Bi-WKV) mechanism, while sigmoid gating (σ(R)) is applied. s By controlling the output weights and stabilizing the features through linear projection and layer normalization, this module can effectively capture pixel-level spatial correlations and global dependencies in medical images of different modalities. It can efficiently process high-resolution images without relying on window operations, perfectly meeting the need for preservation of fine spatial structures in medical image fusion.

[0093] The Channel Mix module focuses on feature fusion along the channel dimension. It similarly processes the input through a Q-Shift operation, then passes it through two parallel linear layers to obtain the final output. and ,right Obtained by linear projection after applying ReLU activation. Ultimately, it is σ( Gated output regulation This module can enhance the feature complementarity of different modal medical images in the channel dimension and improve the feature expression ability of the fused image.

[0094] like Figure 2 As shown, the workflow of the adaptive vision RWKV module is as follows:

[0095] The input of the adaptive vision RWKV module is processed by layer normalization, and then processed in parallel by channel branch and spatial branch to obtain channel blend features and spatial blend features. The channel blend features and spatial blend features are then concatenated to obtain modality A features and modality B features.

[0096] The layer normalization process (normalization layer) uses the layer normalization (LayerNorm) operation to normalize the channel dimension of the feature map. By eliminating the feature distribution differences between channels, it stabilizes the network training process, accelerates the convergence speed, improves the network's generalization ability, and avoids overfitting.

[0097] The channel branch, from input to output, includes: Channel Mixing, RWKV state unit, adaptive local shift module, and adaptive feature fusion module.

[0098] The spatial branch, from input to output, includes: Spatial Mixing, RWKV state unit, adaptive local shift module, and adaptive feature fusion module.

[0099] The RWKV state unit, as the core component of the module, follows the core formula:

[0100]

[0101] This unit captures long-distance dependencies between features through state propagation, and its computational complexity is only [missing information]. Compared to the self-attention mechanism of traditional Transformers, it can effectively capture global structural information of medical images, such as the overall positional relationship between lesions and surrounding tissues, and the complete morphology of organs, while significantly reducing computational costs.

[0102] The adaptive local shift module is key to the accurate capture of local features in this module. Its core design lies in the dynamic selection of local features at different scales through multi-scale local feature extraction and adaptive weight allocation, adapting to anatomical structures and lesion regions of varying sizes in medical images. The workflow of this adaptive local shift module is as follows:

[0103] The input to the adaptive local shift module first passes through 3×3 depthwise separable convolution, 5×5 depthwise separable convolution, and 7×7 depthwise separable convolution in parallel to extract small, medium, and large local features. Depthwise separable convolution can further reduce the number of parameters and computational cost while maintaining feature extraction capability. The three convolution kernel sizes correspond to small-scale, medium-scale, and large-scale local features, respectively, which can comprehensively cover the fine anatomical details, medium-sized structures, and large-sized tissues in medical images.

[0104] Subsequently, the small, medium, and large local features are first compressed by global average pooling (AvgPool2d) to convert the two-dimensional feature map into a one-dimensional global feature map in the channel dimension. Then, after two linear transformations and ReLU activation function processing, attention weights of small, medium, and large sizes are generated. These weights can reflect the importance of local features of different scales in the current image region.

[0105] The adaptive feature fusion module combines small, medium, and large local features with small, medium, and large attention weights element-wise and then fused them to obtain adaptively selected local features (fused features), thus achieving accurate extraction and dynamic optimization of local features.

[0106] Fusion features include: channel blending features and spatial blending features.

[0107] like Figure 3 As shown, the working process of the cross-modal attention module is as follows:

[0108] Features of modality A and modality B are first normalized to eliminate differences in feature distribution within the modality, providing a stable feature foundation for subsequent attention calculation. Then, the normalized features of the two modalities are transformed into three matrices—query (Q), key (K), and value (V)—through a linear mapping layer, where the feature mapping of modality A is the query matrix of modality A. Key matrix Sum matrix The feature mapping of mode B is the query matrix of mode B. Key matrix Sum matrix .

[0109] Unlike traditional cross-modal attention mechanisms, this module employs a cross-modal linear attention (bidirectional linear attention) computation method. Bidirectional linear attention processing includes two branches; in one branch, and The attention weights are calculated to obtain the attention guidance weights of modality B features on modality A features. Then, the attention guidance weights of modality B features on modality A features are residually connected with modality A features to output the modality A features after cross-modal interaction.

[0110] In another branch, and The attention weights are calculated to obtain the attention guidance weights of modality A features on modality B features. Then, the attention guidance weights of modality A features on modality B features are residually connected with modality B features to output the modality B features after cross-modal interaction.

[0111] This bidirectional cross-modal attention computation method enables bidirectional semantic interaction between features from two modalities, enhancing the correlation between modalities, while the linear attention mechanism reduces computational complexity to... This avoids the redundant computation of traditional attention mechanisms, balancing interactive effects with computational efficiency.

[0112] Finally, by performing residual connection between the feature output obtained from attention calculation and the original features of the corresponding modality, the core information of the original features is preserved through the residual structure, avoiding the problem of feature information loss during attention calculation, further improving the expressive power of the features, and providing high-quality cross-modal interactive features for the subsequent progressive fusion module.

[0113] The progressive feature fusion module, one of the core innovations of this embodiment, is designed to achieve "early association, layer-by-layer alignment, and dynamic fusion" of multimodal features. This fundamentally solves the cross-modal semantic and spatial misalignment problem caused by single-step fusion in existing technologies, while also ensuring the module's lightweight and efficient operation, avoiding the introduction of excessive parameters that increase network training and inference costs. The module employs a two-stage structure: an adaptive feature fusion stage and a feature alignment stage. These two stages progress sequentially to achieve accurate fusion of multimodal features.

[0114] The core objective of the adaptive feature fusion stage is to dynamically allocate fusion weights based on the importance of multimodal features, thereby achieving adaptive feature fusion.

[0115] like Figure 4 As shown, the working process of the progressive fusion module is as follows:

[0116] After concatenating the modality A features and modality B features following cross-modal interaction along the channel dimension, the resulting dimension is... The spliced ​​feature map, in which , The number of channels for features in modality A and modality B are respectively used to achieve initial integration of multimodal features through concatenation. Subsequently, a global average pooling operation is performed on the concatenated feature map to compress the two-dimensional feature map into a one-dimensional channel-level global feature. This global feature can reflect the overall importance of each channel feature and provide a basis for subsequent weight generation.

[0117] Based on global features, adaptive fusion weights are generated through channel reduction and restoration operations. Specifically, firstly, a first 1×1 convolution reduces the number of channels in the global features to one-quarter of the original number, decreasing computational cost. Then, a ReLU activation function is applied to introduce non-linearity, enhancing the model's expressive power. Subsequently, a second 1×1 convolution restores the number of channels to the original number of stitched features. Finally, a Sigmoid activation function maps the feature values ​​to the [0,1] interval, yielding channel-level adaptive fusion weights. These weights dynamically reflect the importance of different channels and modalities in the current fusion stage. For example, when processing MRI and PET fusion tasks, if the lesion area is more prominent in PET features, the weights will tilt towards the PET feature channels; if the normal tissue anatomy is clearer in MRI features, the weights will tilt towards the MRI feature channels. Finally, the generated adaptive weights are multiplied element-wise with the original stitched features, and then a weighted sum is performed to obtain the preliminary fusion features, completing the adaptive fusion process.

[0118] The core objective of the feature alignment stage is to refine and align the initially fused features at both spatial and semantic levels, eliminating residual feature misalignments and enhancing the consistency of the fused features. First, two convolutional operations are performed on the initially fused features, interspersed with layer normalization and ReLU activation functions. The first convolution adjusts the number of feature channels to adapt the feature dimensions to subsequent processing requirements, layer normalization stabilizes the feature distribution, and the ReLU activation function introduces a non-linear relationship to enhance the expressive power of the features. The second convolution further refines the features, corrects spatial positional deviations, and achieves accurate alignment of multimodal features. Finally, the refined features are residually connected to the initially fused features. The residual structure preserves the core information of the initially fused features while integrating the refined and aligned features, enhancing the overall expressive power of the fused features and completing the entire progressive fusion process.

[0119] The parameter complexity of this module is only Where c is the number of feature channels, which is much lower than that of existing complex fusion modules. It is lightweight and efficient, and can be inserted after each layer of encoding to achieve synchronous progress of "encoding-fusion". This ensures that multimodal features are gradually associated and aligned during the extraction process, providing high-quality fusion features for subsequent deep encoding and final decoding.

[0120] The downsampling-upsampling residual link runs through the entire encoding and decoding process. After each layer of encoding and fusion is completed, downsampling is performed through a convolution operation with a stride of 2. This reduces the feature map resolution while increasing the number of feature channels, allowing the network to focus on more abstract, deeper features. The number of channels in each encoding stage is 32→64→128→256, and the resolution is correspondingly reduced to 256. → → After the features are encoded in the deepest layer, an upsampling operation is performed through transposed convolution to gradually restore the feature map resolution. At the same time, the upsampled features are residually fused with the fused features from the corresponding encoding stage, which effectively alleviates the gradient vanishing problem, preserves feature information at each level, and ensures that the final decoded fused image has both deep semantic consistency and rich shallow details.

[0121] The core objective of upsampling and residual fusion is to progressively restore the feature map resolution while integrating fusion features from each encoding stage, preserving feature information at different levels, and ensuring that the final fused image possesses both deep semantic consistency and rich shallow details. This process fuses features from the deepest layers. Initially, the process proceeds step by step in the order of "deepest layer → shallowest layer", involving a total of 3 upsampling and residual fusion operations.

[0122] First, the fusion features of layer 4 Perform a transpose convolution operation. The feature calculation formula for the transpose convolution is:

[0123]

[0124] in, The transposed convolution kernel size (3 in this embodiment) is used. , ( Let the step size be 2). For convolution kernel weights, As input features, These are upsampled features. After this upsampling, we obtain... Subsequently, Integration features with layer 3 Perform residual fusion, the fusion formula is as follows: By integrating the two types of features through element addition, the shallow details of the third layer and the deep semantic information of the fourth layer are effectively preserved.

[0125] Next, regarding Perform transposed convolution upsampling to restore the resolution to 256×256 and reduce the number of channels to 64, resulting in... Then, the fusion features with the second layer Perform residual fusion to obtain Finally, regarding Perform transposed convolution upsampling to restore the resolution to 512×512 and reduce the number of channels to 32, resulting in... fusion features with layer 1 Residual fusion is performed to obtain the final fused features. The feature has a size of 512×512×32, which contains detailed information at all levels and has good semantic consistency and spatial alignment.

[0126] S3. The output of the multimodal parallel encoding is compressed into a single-channel luminance feature map after being decoded by the output layer. aisle;

[0127] As the final step of the method described in this invention, the decoding output layer first compresses the final fused feature map into a single-channel brightness feature map (Y' channel) through a 1×1 convolution, making its size consistent with the brightness channel of the original image;

[0128] The core objective of the decoding output stage is to fuse the final features. The image is converted into an RGB fusion image that meets clinical needs, completing the entire fusion process. First, the image is... Perform a 1×1 convolution operation, the channel compression formula is as follows: This yields a blended luminance channel with dimensions of 512×512×1. This ensures that the Cr and Cb chromaticity channels separated in the input preprocessing stage are of the same size.

[0129] S4, will The Cr and Cb channels are spliced ​​together, and the spliced ​​image is then transformed to obtain an RGB fused image.

[0130] The brightness feature map is then stitched together with the Cr and Cb chromaticity channels temporarily stored in the input preprocessing stage to reconstruct an image in YCrCb format.

[0131] Then, The Cr, Cb, and YCr channels are stitched together in the channel order of the YCrCb color space to obtain a YCrCb format image with a size of 512×512×3. Finally, through inverse color space conversion, YCrCb is converted to RGB color space to obtain an RGB image, resulting in the final multimodal medical fusion image. The image retains complementary information from different modalities, and possesses clear details, accurate colors, and good semantic consistency, making it suitable for direct clinical diagnosis.

Claims

1. A progressive multimodal medical image fusion method based on multiple scales, characterized in that, The method includes the following steps: S1. Obtain the grayscale image of mode A and the RGB image of mode B, and perform preprocessing on each; The RGB image of the preprocessed mode B is then converted into Y channel, Cr channel and Cb channel; S2. Perform multimodal parallel encoding on the preprocessed grayscale image of mode A and the Y channel of mode B: Multimodal parallel coding includes four coding stages; The input at each encoding stage passes sequentially through the adaptive visual RWKV module, the cross-modal attention module, and the progressive fusion module to obtain the output of multimodal parallel encoding. The workflow of the adaptive vision RWKV module is as follows: The input of the adaptive vision RWKV module is processed by layer normalization, and then processed in parallel by channel branch and spatial branch to obtain channel blending features and spatial blending features. The channel blending features and spatial blending features are concatenated to obtain modality A features and modality B features. The channel branch from input to output includes: channel blending submodule, RWKV state unit, adaptive local shift module and adaptive feature fusion module. The spatial branch, from input to output, includes: a spatial mixing submodule, an RWKV state unit, an adaptive local shift module, and an adaptive feature fusion module. The workflow of the adaptive local shift module is as follows: The input to the adaptive local shift module is first processed in parallel through 3×3 depthwise separable convolution, 5×5 depthwise separable convolution and 7×7 depthwise separable convolution to extract small, medium and large local features; Small, medium, and large local features are then processed sequentially through global average pooling, two linear transformations, and the ReLU activation function to generate small, medium, and large attention weights. The adaptive feature fusion module multiplies small, medium, and large local features with small, medium, and large attention weights element-wise to obtain fused features; Fusion features include: channel blending features and spatial blending features; S3. The output of the multimodal parallel encoding is compressed into a single-channel brightness feature map after being decoded by the output layer. aisle; S4, will The Cr and Cb channels are stitched together, and the stitched image is then transformed to obtain an RGB fused image.

2. The method for progressive multimodal medical image fusion based on multiple scales according to claim 1, characterized in that, The preprocessing includes normalization and size alignment.

3. The method for progressive multimodal medical image fusion based on multiple scales according to claim 2, characterized in that, In the first, second and third encoding stages, the output of the progressive fusion module is also downsampled through a convolutional module with a stride of 2; The output of the first encoding stage is used as the input of the second encoding stage; The output of the second encoding stage is used as the input of the third encoding stage; The output of the third encoding stage is used as the input of the fourth encoding stage; In the fourth encoding stage, the output of the progressive fusion module is subjected to the first upsampling process and then the first residual fusion is performed with the output of the third encoding stage. The result of the first residual fusion is then subjected to a second upsampling process and then subjected to a second residual fusion with the output of the second encoding stage; The result of the second residual fusion is then subjected to a third upsampling process and then subjected to a third residual fusion with the output of the first encoding stage; The result of the third residual fusion is the output of multimodal parallel coding.

4. The method for progressive multimodal medical image fusion based on multiple scales according to claim 3, characterized in that, The calculation formula for RWKV state elements is: in, This represents the feature output at the current scale. This represents the feature output from the previous scale. This represents the feature input at the current scale. Represents a linear transformation function. express Activation function This represents the Sigmoid activation function. This indicates element-wise multiplication.

5. The method for progressive multimodal medical image fusion based on multiple scales according to claim 4, characterized in that, The cross-modal attention module works as follows: Modality A features and Modality B features are processed sequentially through layer normalization, linear mapping layer and bidirectional linear attention; Linear mapping layer output: Query matrix of mode A Key matrix Sum matrix And the query matrix of mode B Key matrix Sum matrix ; Bidirectional linear attention processing consists of two branches, in one branch, and The attention weights are calculated to obtain the attention guidance weights of modality B features on modality A features. Then, the attention guidance weights of modality B features on modality A features are residually connected with modality A features to output the modality A features after cross-modal interaction. In another branch, and The attention weights are calculated to obtain the attention guidance weights of modality A features on modality B features. Then, the attention guidance weights of modality A features on modality B features are residually connected with modality B features to output the modality B features after cross-modal interaction.

6. The method for progressive multimodal medical image fusion based on multiple scales according to claim 5, characterized in that, The progressive fusion module works as follows: After cross-modal interaction, the modal A features and the modal B features are concatenated in the channel dimension to obtain concatenated features. The concatenated features are then subjected to global average pooling, the first 1×1 convolution, the ReLU activation function, the second 1×1 convolution, and the Sigmoid activation function to obtain adaptive fusion weights. The adaptive fusion weights and the concatenated features are sequentially multiplied element-wise and weighted summed to obtain preliminary fusion features. The initial fusion features are processed sequentially through the first convolution, layer normalization, ReLU activation function, and the first convolution, and then residual fusion is performed with the initial fusion features to obtain the output of the progressive fusion module.

7. The method for progressive multimodal medical image fusion based on multiple scales according to claim 6, characterized in that, The decoding output layer consists of a 1×1 convolution.

Images