Image fusion system and method based on multi-branch feature fusion and attention mechanism

Abstract

This invention discloses an image fusion system and method based on multi-branch feature fusion and attention mechanisms, belonging to the field of multimodal medical image fusion. It solves problems such as insufficient single-scale feature extraction, inadequate attention mechanism modeling, inefficient multi-scale fusion strategies, incomplete feature capture due to complex network structures, unreasonable weight allocation, and information redundancy or loss. This invention constructs a multi-branch feature extraction module with parallel branches at shallow, mid, and deep levels, combined with BM, GCAConv, and FeatureEnhancement modules. Features are fused through a multi-scale fusion module, and then output by an image reconstruction module, achieving high-quality medical image fusion. This invention, through the collaboration of multi-branch extraction, multi-scale fusion, and reconstruction modules, comprehensively captures features at all levels, highlights key information, suppresses redundancy, and outputs a fused image that balances detail and structure, improving diagnostic efficiency and accuracy, and demonstrating strong practicality.

Description

Technical Field

[0001] This invention relates to the field of multimodal medical image fusion, and in particular to an image fusion system and method based on multi-branch feature fusion and attention mechanism. Background Technology

[0002] Multimodal medical image fusion technology integrates information from different imaging modalities to provide more comprehensive and accurate anatomical and functional details, thereby improving diagnostic accuracy, treatment planning, and disease tracking. By fusing these images, the limitations of a single imaging modality can be overcome, providing more comprehensive diagnostic information. With the development of deep learning technology, image fusion methods based on convolutional neural networks (CNNs) and Transformers have gradually emerged. However, existing methods suffer from problems such as simplistic branch design, insufficient feature enhancement, and imprecise multimodal information fusion, affecting the auxiliary value of fused images in medical diagnosis. Therefore, a technical solution is needed that can efficiently extract multimodal image features, perform multi-branch collaborative enhancement, and accurately fuse them.

[0003] With the development of artificial intelligence, convolutional neural networks have achieved remarkable results in the field of image processing. However, existing image enhancement methods generally face the following problems:

[0004] (1) Single-scale feature extraction is difficult to take into account both local details and global semantic information;

[0005] (2) Traditional attention mechanisms do not adequately model the channel or spatial dimensions and cannot effectively suppress noise interference;

[0006] (3) The multi-scale fusion strategy lacks an effective weight allocation mechanism, which affects the final enhancement effect;

[0007] (4) The network structure is highly complex, the inference speed is slow, and it is difficult to deploy to edge devices.

[0008] Therefore, a new image enhancement method is needed to improve model efficiency and generalization ability while ensuring enhancement quality. Summary of the Invention

[0009] This invention proposes an image fusion system and method based on multi-branch feature fusion and attention mechanism. By introducing the BM module (basic module), GCAConv module (graph convolutional attention module), and FeatureEnhancement module, three parallel branches of shallow, medium and deep layers are constructed. A multi-scale feature fusion strategy is adopted to achieve high-quality image enhancement, so as to solve the problems of insufficient single-scale feature extraction leading to incomplete capture of local details and global semantic information, insufficient attention mechanism modeling leading to unreasonable allocation of key feature weights, inefficient multi-scale fusion strategy causing feature information redundancy or loss, and complex network structure leading to low fusion efficiency and difficulty in adapting to edge device deployment.

[0010] An image fusion system based on multi-branch feature fusion and attention mechanisms includes an input processing module, a multi-branch feature extraction module, a feature fusion module, and an image reconstruction module. These modules are sequentially connected.

[0011] The input processing module is used to perform channel stitching of at least two modal medical images to be enhanced with a reference image, and to perform normalization processing on the stitched image;

[0012] The multi-branch feature extraction module is used to extract features from the stitched image;

[0013] The feature fusion module is used to concatenate the feature maps output by the multi-branch feature extraction module and then generate fused features through 1×1 convolution.

[0014] The image reconstruction module is used to process the fused features using the Sigmoid activation function to generate an output image while maintaining the same size as the input image.

[0015] Furthermore, the multi-branch feature extraction module includes shallow branches, mid-level branches, and deep branches, wherein,

[0016] Shallow branches include the BM module and the FeatureEnhancement module;

[0017] The middle-level branches include the BM module, the GCAConv module, and the FeatureEnhancement module;

[0018] The deep branch includes two GCAConv modules and a FeatureEnhancement module.

[0019] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0020] Furthermore, the GCAConv module includes convolutional layers, batch normalization layers, channel attention submodules, and spatial attention submodules. The convolutional layers and batch normalization layers are connected, and the batch normalization layers are simultaneously connected to both the channel attention submodules and the spatial attention submodules.

[0021] The channel attention submodule is used to generate channel attention weights.

[0022] The spatial attention submodule is used to generate spatial attention weights.

[0023] The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

[0024] Furthermore, the FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein,

[0025] The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales;

[0026] The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0027] The image fusion method based on multi-branch feature fusion and attention mechanism, and the image fusion system based on the above-mentioned multi-branch feature fusion and attention mechanism, includes the following steps:

[0028] S1. Input preprocessing: At least two modal medical images to be enhanced are stitched together with a reference image, and the stitched image is normalized.

[0029] S2. Feature Extraction: Feature extraction is performed on the stitched image through three parallel branches: shallow, medium, and deep layers.

[0030] The shallow branch passes through the BM module and the FeatureEnhancement module in sequence to extract local texture and edge information of the image and output a feature map;

[0031] The middle-level branch passes through the BM module, GCAConv module and FeatureEnhancement module in sequence to extract the structural and semantic information of the image and output the feature map;

[0032] The deep branch passes through two GCAConv modules and the FeatureEnhancement module in sequence to extract high-level semantic and contextual information of the image and output feature maps;

[0033] S3. Feature Fusion: The feature maps output by the shallow branch, the middle branch and the deep branch are concatenated by channels, and then fused by 1×1 convolution to generate fused features;

[0034] S4. Image Reconstruction: The fused features are processed using the Sigmoid activation function to generate an output image, while maintaining the same size as the input image.

[0035] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0036] Furthermore, the GCAConv module includes convolutional layers, batch normalization layers, channel attention submodules, and spatial attention submodules. The convolutional layers and batch normalization layers are connected, and the batch normalization layers are simultaneously connected to both the channel attention submodules and the spatial attention submodules.

[0037] The channel attention submodule is used to generate channel attention weights.

[0038] The spatial attention submodule is used to generate spatial attention weights.

[0039] The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

[0040] Furthermore, the FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein,

[0041] The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales;

[0042] The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0043] Furthermore, in S3, the feature maps output by the shallow branch, the middle branch, and the deep branch are stitched together in the channel dimension to obtain a stitched feature map, the number of channels of the stitched feature map being the sum of the number of channels of the output feature maps of each branch; in S4, the size of the fused feature is adjusted by bilinear interpolation so that the size of the output image is consistent with the input image.

[0044] The beneficial effects of the present invention: The image fusion system and method based on multi-branch feature fusion and attention mechanism of the present invention have the following advantages:

[0045] Multi-branch structure design: By using three parallel branches—shallow, middle, and deep—the local details and global semantic information of the image are captured respectively, enhancing the model's expressive power.

[0046] Attention mechanism fusion: The introduction of a dual attention mechanism of channel and space enhances feature selection ability and effectively suppresses interference from irrelevant features;

[0047] High-efficiency feature enhancement module: Combining multi-scale convolution and residual structure to achieve efficient feature enhancement and restoration;

[0048] Flexible and highly scalable: It supports a variety of image enhancement tasks, such as dehazing, noise reduction, low-light enhancement, super-resolution, etc., and can further improve performance by introducing the Transformer module;

[0049] Lightweight design: The parameters of each module are controllable, making it suitable for deployment on mobile and embedded devices. Attached Figure Description

[0050] Figure 1 This is a diagram illustrating the overall framework of the image fusion system based on multi-branch feature fusion and attention mechanism of the present invention.

[0051] Figure 2 This is a schematic diagram of the multi-branch feature extraction module of the present invention;

[0052] Figure 3 This is a flowchart of the attention mechanism of the present invention;

[0053] Figure 4 This is a schematic diagram of the FeatureEnhancement module of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Reference Figures 1-4 As shown, the image fusion system based on multi-branch feature fusion and attention mechanism includes an input processing module, a multi-branch feature extraction module, a feature fusion module, and an image reconstruction module. These modules are connected sequentially.

[0056] The input processing module is used to perform channel stitching of at least two modal medical images to be enhanced with a reference image, and to perform normalization processing on the stitched image;

[0057] The multi-branch feature extraction module is used to extract features from the stitched image;

[0058] The feature fusion module is used to concatenate the feature maps output by the multi-branch feature extraction module and then generate fused features through 1×1 convolution.

[0059] The image reconstruction module is used to process the fused features using the Sigmoid activation function to generate an output image while maintaining the same size as the input image.

[0060] Specifically, this invention introduces the BM module, GCAConv module, and FeatureEnhancement module to construct three parallel branches: shallow, mid-level, and deep. It also employs a multi-scale feature fusion strategy, significantly improving the quality of medical image fusion. Specifically, the parallel branch design allows for the simultaneous and thorough extraction of features at different levels. The shallow branch focuses on capturing local texture and edge information; the mid-level branch leverages the attention mechanism of the GCAConv module to enhance structural and mid-level semantic features; and the deep branch deeply mines high-level semantics and global contextual information. These different levels of features complement each other, providing a rich information source for subsequent fusion. The GCAConv module's convolutional layers, batch normalization layers, channel attention submodule, and spatial attention submodule work collaboratively to accurately weight features based on channel and spatial dimensions, highlighting key features, suppressing redundant and interfering information, and enhancing the relevance and effectiveness of feature representation. The multi-scale convolutional layers of the FeatureEnhancement module effectively capture features at different scales of the image, taking into account everything from fine textures at small scales to macroscopic global structural information. Combined with residual connections, it avoids information loss and accelerates model convergence, so that the final output fused image retains rich details while having a clearer and more accurate overall structure. This provides more comprehensive and reliable image evidence for clinical diagnosis, helping doctors to judge the condition more accurately and improve diagnostic accuracy.

[0061] Furthermore, the multi-branch feature extraction module includes shallow branches, mid-level branches, and deep branches, wherein,

[0062] Shallow branches include the BM module and the FeatureEnhancement module;

[0063] The middle-level branches include the BM module, the GCAConv module, and the FeatureEnhancement module;

[0064] The deep branch includes two GCAConv modules and a FeatureEnhancement module.

[0065] Specifically, this invention sets up three parallel branches—shallow, mid-level, and deep—in the multi-branch feature extraction module, enabling different branches to extract image features specifically. The shallow branch uses the BM module and the FeatureEnhancement module to extract local texture and edge information; the mid-level branch uses the BM module, the GCAConv module, and the FeatureEnhancement module to extract structural and semantic information; and the deep branch uses two GCAConv modules and the FeatureEnhancement module to extract high-level semantic and contextual information. The features extracted by each branch complement each other, enriching the hierarchy and comprehensiveness of the features. This multi-branch design avoids the limitations of a single branch in feature extraction, simultaneously considering both local details and global semantics of the image, improving the richness of feature expression, providing a better foundation for subsequent feature fusion, and thus improving the quality of medical image fusion. The fused image can more comprehensively present the key information of different modalities of medical images, better assisting medical diagnosis.

[0066] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0067] Specifically, this invention designs the convolutional layers of the shallow branch's BM module to contain three consecutive 3×3 convolutions, enabling progressive extraction and refinement of local image features. This continuous convolutional structure reduces the number of parameters while maintaining sufficient model expressive power, helping to capture fine texture and edge information in the image. In the mid-layer branch's GCAConv module, the fully connected layers of the channel attention submodule are designed to contain two consecutive 1×1 convolutions, which more effectively models the dependencies between channels and enhances attention to key feature channels. The spatial attention submodule uses 7×7 convolutions, which expands the receptive field and captures a wider range of spatial contextual information, allowing the model to better understand the structural relationships between different regions in the image. The deep branch's GCAConv module also adopts a similar design, further strengthening the extraction of high-level semantics and global contextual information. These specific convolutional structure designs enable each branch to extract features of different levels and types more specifically, improving the efficiency and accuracy of feature extraction and providing higher-quality, more representative feature representations for subsequent feature fusion, thereby improving the quality and effect of medical image fusion.

[0068] Furthermore, the GCAConv module includes convolutional layers, batch normalization layers, channel attention submodules, and spatial attention submodules. The convolutional layers and batch normalization layers are connected, and the batch normalization layers are simultaneously connected to both the channel attention submodules and the spatial attention submodules.

[0069] The channel attention submodule is used to generate channel attention weights.

[0070] The spatial attention submodule is used to generate spatial attention weights.

[0071] The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

[0072] Specifically, this invention significantly improves the quality and effectiveness of medical image fusion by designing a multi-scale fusion module comprising a feature weighting layer, a multi-scale feature extraction layer, and a feature aggregation layer. The feature weighting layer adaptively allocates the importance of different branch features during the fusion process by calculating the weights of features from each branch, ensuring that key features are fully preserved and enhanced while suppressing redundant information. The multi-scale feature extraction layer processes the weighted features using convolutional kernels of different sizes, simultaneously capturing feature information at different scales, effectively representing everything from local details to global structure, thus enriching the diversity of fused features. The feature aggregation layer integrates multi-scale features, further optimizing feature representation through residual connections and convolutional operations, enhancing feature complementarity and consistency. This design allows the fused image to simultaneously retain the advantageous features of each branch, including shallow fine texture and edge information, mid-level structural semantics, and deep global contextual information, thereby providing more comprehensive and accurate image evidence for medical diagnosis and helping doctors to more clearly observe lesion features and tissue structures.

[0073] Furthermore, the FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein,

[0074] The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales;

[0075] The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0076] Specifically, this invention employs an attention-based weighting mechanism in the feature weighting layer to achieve adaptive evaluation and allocation of the importance of features in each branch, significantly improving the targeting and accuracy of medical image fusion. Specifically, the channel attention unit, through global average pooling and a fully connected layer structure, can automatically learn the importance of features in each channel, highlighting features crucial for medical diagnosis, such as the feature expression of lesion regions. The spatial attention unit, on the other hand, analyzes the spatial dimension of the feature map through convolution operations, accurately locating key regions in the image, such as tumor boundaries or the edges of tissue structures. This synergistic effect of the dual attention mechanism allows for the full integration of complementary information from different modalities of medical images, while suppressing interference from irrelevant information. For example, when fusing PET and CT images, channel attention can enhance the expression of metabolically active regions, while spatial attention focuses on the precise location of anatomical structures, thereby generating a more diagnostically valuable fused image, providing doctors with clearer and more comprehensive lesion information, and assisting clinical decision-making.

[0077] The image fusion method based on multi-branch feature fusion and attention mechanism, and the image fusion system based on the above-mentioned multi-branch feature fusion and attention mechanism, includes the following steps:

[0078] S1. Input preprocessing: At least two modal medical images to be enhanced are stitched together with a reference image, and the stitched image is normalized.

[0079] S2. Feature Extraction: Feature extraction is performed on the stitched image through three parallel branches: shallow, medium, and deep layers.

[0080] The shallow branch passes through the BM module and the FeatureEnhancement module in sequence to extract local texture and edge information of the image and output a feature map;

[0081] The middle-level branch passes through the BM module, GCAConv module and FeatureEnhancement module in sequence to extract the structural and semantic information of the image and output the feature map;

[0082] The deep branch passes through two GCAConv modules and the FeatureEnhancement module in sequence to extract high-level semantic and contextual information of the image and output feature maps;

[0083] S3. Feature Fusion: The feature maps output by the shallow branch, the middle branch and the deep branch are concatenated by channels, and then fused by 1×1 convolution to generate fused features;

[0084] S4. Image Reconstruction: The fused features are processed using the Sigmoid activation function to generate an output image, while maintaining the same size as the input image.

[0085] Specifically, this invention designs the multi-scale feature extraction layer as a parallel convolutional layer containing three different kernel sizes: 3×3, 5×5, and 7×7. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, effectively improving the comprehensiveness and robustness of feature extraction. Different kernel sizes can capture local details (3×3 convolution), medium-scale structures (5×5 convolution), and global context (7×7 convolution) of the image, ensuring that various features from fine textures to macroscopic tissues in multimodal medical images are fully extracted. The addition of the batch normalization layer stabilizes the distribution of outputs from each layer, avoiding gradient anomalies during training, while the ReLU activation function enhances the model's ability to express complex features by introducing nonlinear transformations. This design enables the multi-scale feature extraction layer to generate richer and more discriminative feature representations, providing high-quality input for subsequent feature aggregation, ultimately improving the quality of the fused image. The fused result retains both accurate local information and a complete global structure, better meeting the clinical diagnostic needs for image detail and overall correlation.

[0086] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

[0087] Furthermore, the GCAConv module includes convolutional layers, batch normalization layers, channel attention submodules, and spatial attention submodules. The convolutional layers and batch normalization layers are connected, and the batch normalization layers are simultaneously connected to both the channel attention submodules and the spatial attention submodules.

[0088] The channel attention submodule is used to generate channel attention weights.

[0089] The spatial attention submodule is used to generate spatial attention weights.

[0090] The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

[0091] Furthermore, the FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein,

[0092] The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales;

[0093] The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0094] Furthermore, in S3, the feature maps output by the shallow branch, the middle branch, and the deep branch are stitched together in the channel dimension to obtain a stitched feature map, the number of channels of the stitched feature map being the sum of the number of channels of the output feature maps of each branch; in S4, the size of the fused feature is adjusted by bilinear interpolation so that the size of the output image is consistent with the input image.

[0095] Specifically, this invention constructs a medical image fusion system comprising a multi-branch feature extraction module, a multi-scale fusion module, and an image reconstruction module, achieving efficient fusion and accurate reconstruction of multimodal medical images. The multi-branch feature extraction module's parallel structure of shallow, mid-level, and deep layers captures local texture, mid-level semantics, and global contextual information respectively, ensuring comprehensive extraction of features at different levels. The multi-scale fusion module adaptively integrates features from each branch through feature weighting, multi-scale extraction, and aggregation, highlighting key information and suppressing redundancy. The image reconstruction module efficiently converts the fused features into the target image, ensuring output quality and resolution. The collaborative work of these modules allows the system to fully exploit the complementary information of different modalities and enhance the specificity and richness of feature representation through attention mechanisms and multi-scale strategies. The resulting fused image balances detail integrity and structural accuracy, providing more comprehensive and reliable image support for clinical diagnosis and contributing to improved diagnostic efficiency and accuracy.

[0096] The following is a specific example of the present invention:

[0097] The multi-branch attention fusion medical image enhancement system of this invention adopts a modular design, and its overall architecture is as follows: Figure 1 As shown, it mainly includes four core modules: input processing module (for receiving and preprocessing multimodal medical images), multi-branch feature extraction module (constructing three parallel branches—shallow, middle, and deep—to extract local details, structural information, and high-level semantics of the image, respectively), feature fusion module (performing channel splicing and weighted fusion of the features output by each branch to generate fused features), and output processing module (mapping the fusion result back to the image space to output high-quality medical images).

[0098] S1: The core function of the input processing module is to preprocess multimodal medical images to prepare for subsequent feature extraction. The specific implementation is as follows:

[0099] Image input and channel stitching:

[0100] It receives medical images of at least two modalities, such as MRI images (X) and SPECT images (Y), both of which are the same size, such as 256×256;

[0101] Then, a stitching operation is performed in the channel dimension to stitch the two modal medical images (such as MRI and SPECT) in the channel dimension to form a multi-channel input tensor. The shape of the stitched tensor is B, C, H, W; where B is the batch number B, C=2 is the number of channels, and H and W are the height and width of the image, respectively.

[0102] Initial feature extraction:

[0103] We use a 3×3 convolutional layer to perform initial feature extraction on the concatenated input, expanding the number of channels from 2 to 64:

[0104] In this step, a two-dimensional convolutional layer is defined with 2 input channels, 64 output channels, and a 3×3 kernel size. A 1-pixel padding is added to the edges of the input tensor. By applying this convolutional layer, the shape of the input tensor x changes from B, 2, H, W to B, 64, H, W. Since a stride of 1 is set, the spatial dimensions (height and width) of the output tensor after the convolution operation are the same as the input tensor. This step is the initial stage of feature extraction, providing the foundation for subsequent feature processing and enhancement.

[0105] S2: The multi-branch feature extraction module is the core component, consisting of three parallel branches: shallow, medium, and deep, each extracting image features at different levels. The structure of each branch is as follows:

[0106] Specifically, the Shallow Branch structure consists of BMModule + FeatureEnhancement; its function is to extract local texture and edge information from the image.

[0107] The BMModule effectively extracts and enhances the fundamental features of an image by defining two consecutive convolutional blocks (ConvBlocks). Each convolutional block consists of a 3×3 convolution, batch normalization (BN), and a ReLU activation function. The first convolutional block maps the number of channels of the input features from 64 to 64, extracting the basic features; the second convolutional block further enhances these features, progressively increasing their expressive power through successive convolutional operations. This design not only extracts the basic structural information of the image but also stabilizes the training process and introduces non-linearity through batch normalization and ReLU activation functions, providing a solid foundation for subsequent feature processing and fusion.

[0108] Then, the FeatureEnhancement module effectively enhances the expressive power of features by defining multi-scale convolutional layers (1×1 convolution for dimensionality reduction, 3×3 convolution for local feature extraction, and 1×1 convolution for dimensionality increase) and residual connections. Specifically, firstly, a 1×1 convolution reduces the number of channels in the input features from 64 to 32 to reduce computation while maintaining the expressive power of the features. Next, a 3×3 convolutional layer extracts local features, capturing texture and structural information in the image. Finally, a 1×1 convolution increases the number of channels back from 32 to 64, integrating the feature information. Furthermore, if the number of input channels is not equal to the number of output channels, a 1×1 convolutional layer is used to adjust the channels of the input features, ensuring the feasibility of residual connections. Residual connections directly add a portion of the input features to the output features, avoiding the loss of important information during multi-layer convolution operations while enhancing the expressive power of the features.

[0109] Specifically, the medium branch structure is: BMModule + GCAConv + FeatureEnhancement;

[0110] Its function is to introduce an attention mechanism to enhance the image's structural and semantic information.

[0111] Among them, the generalized convolutional module with attention mechanism (GCAConv) effectively enhances the expressive power of features by combining 3×3 convolution operation with channel attention mechanism (CAAttention) and spatial attention mechanism (SpatialAttention).

[0112] Specifically, local features are first extracted using a 3×3 convolutional layer. Then, channel attention weights and spatial attention weights are generated using a channel attention module and a spatial attention module, respectively. Finally, the intermediate features are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

[0113] Specifically, the Deep Branch structure is: GCAConv ×2 + FeatureEnhancement; its function is to extract high-level semantics and global contextual information of the image.

[0114] In the feature enhancement module, the expressive power of features is effectively enhanced by cascading two generalized convolutional modules with attention mechanisms (GCAConv).

[0115] Specifically, we first define a generalized convolutional module with an attention mechanism, which has 64 input channels and 64 output channels. This module extracts and enhances features by combining convolutional operations and an attention mechanism.

[0116] Next, a second generalized convolutional module with an attention mechanism is defined, with the same number of input and output channels (64). This module further enhances the features and improves their expressive power.

[0117] During the forward propagation process, the input feature x first passes through the first generalized convolutional module with an attention mechanism to obtain intermediate features.

[0118] Then, the intermediate features are used as input to a second generalized convolutional module with an attention mechanism, and the final output is the enhanced features.

[0119] S3: The feature fusion module is responsible for integrating the features output from shallow, mid, and deep branches to generate a unified fused feature tensor. The specific steps are as follows:

[0120] In particular, the feature fusion module achieves multi-scale feature fusion by splicing together the features of shallow, medium and deep branches.

[0121] Specifically, the input feature x is first extracted through shallow branches, and its shape is B, 64, H, W, where B is the batch size, 64 is the number of channels, and H and W are the height and width of the feature map, respectively.

[0122] Next, the input feature x is processed through the intermediate branch to extract intermediate features, which also have shapes of B, 64, H, and W. The intermediate branch introduces an attention mechanism on top of the basic features to enhance the semantic expression of the intermediate-level features.

[0123] Then, the input feature x is processed through deep branches to extract deep features, which also have shapes of B, 64, H, W. The deep branches use multi-layer convolution and attention mechanisms to deeply mine the semantic features and multimodal complementary information of the image.

[0124] Finally, the features extracted from the shallow, middle, and deep branches are spliced ​​together along the channel dimension to form a fused feature with the shape B, 192, H, W.

[0125] In this process, the convolution and activation are combined using a 1×1 convolution to compress the number of channels to 1, and the final output is generated through a Sigmoid activation function. In the feature fusion module, the fused multi-scale features are further processed using a 1×1 convolutional layer and a Sigmoid activation function to generate the final output image. Specifically, the fused features are first compressed from 192 channels to 1 through a 1×1 convolutional layer, while integrating feature information from different channels. Then, the Sigmoid activation function maps the values ​​of the intermediate features to the range [0, 1] to generate the final output image. This design not only reduces the dimensionality of the features but also retains important feature information, ensuring that the pixel values ​​of the output image are within a reasonable range.

[0126] S4: The output processing module ensures that the fused image and the input image maintain consistency in size and numerical range, specifically including:

[0127] Specifically, size alignment: In the output processing module, a size adjustment mechanism is specially designed to ensure the consistency between the fused output image and the input feature image in the spatial dimension.

[0128] First, the system compares the shape of the fused output image with the shape of the input feature image. If there is a difference in spatial dimensions (i.e., height and width), a resizing procedure is automatically initiated. Using a bilinear interpolation function, the output image is upsampled or downsampled with the spatial dimensions of the input feature image as the target size. During this process, the parameters of the bilinear interpolation function are carefully set to mode='bilinear' and align_corners=True to ensure that the interpolated image not only accurately aligns with the target size but also maintains visual smoothness and continuity, avoiding distortion or blurring. This effectively solves the size mismatch problem that may occur during feature extraction and fusion, ensuring perfect spatial alignment between the output image and the input feature image. This provides a high-quality data foundation for subsequent image processing and analysis, further improving the performance and reliability of the entire system.

[0129] Then, the numerical range is standardized: since the Sigmoid activation function restricts the output to the [0,1] range, it can be further linearly transformed to a suitable range (such as [0,255]) to adapt to the DICOM or NIfTI format storage standard according to the needs of medical images.

[0130] This invention relates to an image fusion system and method based on multi-branch feature fusion and attention mechanisms. By constructing a multi-branch feature extraction module comprising three parallel branches (shallow, mid-level, and deep), combined with BM, GCAConv, and FeatureEnhancement modules, it achieves comprehensive extraction of features at different levels, including local texture, mid-level semantics, and global context. The multi-scale fusion module, through feature weighting, multi-scale extraction, and aggregation, adaptively integrates features from each branch using an attention mechanism, highlighting key information and suppressing redundancy. The image reconstruction module efficiently converts the fused features into the target image, ensuring output quality and resolution. The synergistic effect of these modules ensures that the fused medical image maintains both detail integrity and structural accuracy, providing more comprehensive and reliable image support for clinical diagnosis, effectively improving diagnostic efficiency and accuracy. Furthermore, its lightweight design makes it suitable for deployment on edge devices, exhibiting strong practicality and scalability.

[0131] The specific embodiments of the invention have been described in detail above, but they are only examples, and the invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions to the invention are also within the scope of this invention. Therefore, all equivalent changes and modifications made without departing from the spirit and scope of this invention should be covered within the scope of this invention.

Claims

1. A medical image fusion system based on multi-branch feature fusion and attention mechanism, characterized in that, It includes an input processing module, a multi-branch feature extraction module, a feature fusion module, and an image reconstruction module, which are connected sequentially. The input processing module is used to perform channel stitching of at least two modal medical images to be enhanced with a reference image, and to perform normalization processing on the stitched image; A multi-branch feature extraction module is used to extract features from the stitched image. It includes shallow branches, mid-level branches, and deep branches. The shallow branches include a basic module and a FeatureEnhancement module. The mid-level branches include a basic module, a GCAConv module, and a FeatureEnhancement module. The deep branches include two GCAConv modules and a FeatureEnhancement module. Each GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule, and a spatial attention submodule. The convolutional layer and the batch normalization layer are connected, and the batch normalization layer is simultaneously connected to both the channel attention submodule and the spatial attention submodule. The channel attention submodule is used to generate channel attention weights. The spatial attention submodule is used to generate spatial attention weights. The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights respectively, and then added together to obtain the enhanced features; The feature fusion module is used to concatenate the feature maps output by the multi-branch feature extraction module and then generate fused features through 1×1 convolution. The image reconstruction module is used to process the fused features using the Sigmoid activation function to generate an output image while maintaining the same size as the input image.

2. The medical image fusion system based on multi-branch feature fusion and attention mechanism according to claim 1, characterized in that, The basic module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

3. The medical image fusion system based on multi-branch feature fusion and attention mechanism according to claim 1, characterized in that, The FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein... The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

4. A medical image fusion method based on multi-branch feature fusion and attention mechanism, based on the medical image fusion system based on multi-branch feature fusion and attention mechanism as described in any one of claims 1-3, characterized in that, Includes the following steps: S1. Input preprocessing: At least two modal medical images to be enhanced are stitched together with a reference image, and the stitched image is normalized. S2. Feature Extraction: Feature extraction is performed on the stitched image through three parallel branches: shallow, medium, and deep layers. The shallow branch passes through the basic module and the FeatureEnhancement module in sequence, extracting local texture and edge information of the image and outputting a feature map; The intermediate branch sequentially passes through the basic module, the GCAConv module, and the FeatureEnhancement module to extract the structural and semantic information of the image and output the feature map. The deep branch passes through two GCAConv modules and the FeatureEnhancement module in sequence to extract high-level semantic and contextual information of the image and output feature maps; S3. Feature Fusion: The feature maps output by the shallow branch, the middle branch and the deep branch are concatenated by channels, and then fused by 1×1 convolution to generate fused features; S4. Image Reconstruction: The fused features are processed using the Sigmoid activation function to generate an output image, while maintaining the same size as the input image.

5. The medical image fusion method based on multi-branch feature fusion and attention mechanism according to claim 4, characterized in that, The basic module includes two consecutive ConvBlocks, each ConvBlock comprising a convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence.

6. The medical image fusion method based on multi-branch feature fusion and attention mechanism according to claim 4, characterized in that, The GCAConv module includes convolutional layers, batch normalization layers, channel attention submodules, and spatial attention submodules. The convolutional layers and batch normalization layers are connected, and the batch normalization layers are simultaneously connected to both the channel attention submodules and the spatial attention submodules. The channel attention submodule is used to generate channel attention weights. The spatial attention submodule is used to generate spatial attention weights. The output features of the convolutional layer are multiplied by the channel attention weights and spatial attention weights to obtain the enhanced features.

7. The medical image fusion method based on multi-branch feature fusion and attention mechanism according to claim 4, characterized in that, The FeatureEnhancement module includes multi-scale convolutional layers and residual connections, wherein... The multi-scale convolutional layer includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence, used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

8. The medical image fusion method based on multi-branch feature fusion and attention mechanism according to claim 4, characterized in that, In S3, the feature maps output by the shallow branch, the middle branch, and the deep branch are stitched together along the channel dimension to obtain a stitched feature map. The number of channels in the stitched feature map is the sum of the number of channels in the output feature maps of each branch. In S4, the size of the fused feature is adjusted by bilinear interpolation so that the size of the output image is consistent with the size of the input image.

Images