Image fusion system and method based on multi-branch feature fusion and attention mechanism

Abstract

The invention discloses an image fusion system and method based on multi-branch feature fusion and an attention mechanism, and belongs to the field of multi-modal medical image fusion. The problems of incomplete feature capture, unreasonable weight distribution, information redundancy or loss and the like caused by insufficient single-scale feature extraction, insufficient attention mechanism modeling, low efficiency of a multi-scale fusion strategy and complex network structure are solved. According to the invention, a multi-branch feature extraction module containing shallow-layer, middle-layer and deep-layer parallel branches is constructed, BM, GCACConv and FeatureEnhancement modules are combined, features are fused through a multi-scale fusion module, and then output is carried out through an image reconstruction module, so that high-quality medical image fusion is realized. Through cooperation of the multi-branch extraction module, the multi-scale fusion module and the reconstruction module, features of all levels are comprehensively captured, key information is highlighted, redundancy is suppressed, a fusion image considering details and structures is output, the diagnosis efficiency and accuracy are improved, and practicability is high.

Description

Technical Field

[0001] The present invention relates to the field of multimodal medical image fusion, and in particular to an image fusion system and method based on multi-branch feature fusion and attention mechanism. Background Art

[0002] Multimodal medical image fusion technology integrates information from different imaging modalities to provide more comprehensive and accurate anatomical and functional details, thereby improving diagnostic accuracy, treatment planning, and disease tracking. By fusing these images, the shortcomings of a single imaging modality can be overcome, providing more comprehensive diagnostic information. With the development of deep learning technology, image fusion methods based on convolutional neural networks (CNNs) and Transformers have gradually emerged. However, existing methods suffer from problems such as single branch design, insufficient feature enhancement, and inaccurate fusion of multimodal information, which affect the auxiliary value of fused images for medical diagnosis. Therefore, a technical solution is needed that can efficiently extract multimodal image features, collaboratively enhance multiple branches, and accurately fuse them.

[0003] With the development of artificial intelligence, convolutional neural networks have achieved remarkable results in the field of image processing. However, existing image enhancement methods generally face the following problems: (1) It is difficult to extract features at a single scale while taking into account both local details and global semantic information; (2) The traditional attention mechanism does not adequately model the channel or spatial dimensions and cannot effectively suppress noise interference; (3) The multi-scale fusion strategy lacks an effective weight distribution mechanism, which affects the final enhancement effect; (4) The network structure is highly complex, the inference speed is slow, and it is difficult to deploy to edge devices.

[0004] Therefore, it is necessary to propose a new image enhancement method to improve model efficiency and generalization ability while ensuring enhancement quality. Summary of the Invention

[0005] The present invention proposes an image fusion system and method based on multi-branch feature fusion and attention mechanism. By introducing the BM module (basic module), the GCAConv module (graph convolution attention module) and the FeatureEnhancement module (feature enhancement module), three parallel branches of shallow, middle and deep layers are constructed, and a multi-scale feature fusion strategy is adopted to achieve high-quality image enhancement. This solves the problems in the existing technology, such as insufficient single-scale feature extraction leading to incomplete capture of local details and global semantic information, insufficient attention mechanism modeling leading to unreasonable distribution of key feature weights, inefficient multi-scale fusion strategy leading to redundant or lost feature information, and complex network structure leading to low fusion efficiency and difficulty in adapting to edge device deployment.

[0006] The image fusion system based on multi-branch feature fusion and attention mechanism includes an input processing module, a multi-branch feature extraction module, a feature fusion module and an image reconstruction module. The input processing module, the multi-branch feature extraction module, the feature fusion module and the image reconstruction module are connected in sequence, wherein, An input processing module, configured to perform channel splicing on at least two modal medical images to be enhanced and a reference image, and to perform normalization processing on the spliced ​​images; Multi-branch feature extraction module, used to extract features from the spliced ​​images; The feature fusion module is used to perform channel splicing on the feature maps output by the multi-branch feature extraction module, and then generate fused features through 1×1 convolution fusion; The image reconstruction module is used to process the fused features using a Sigmoid activation function to generate an output image, and keep the size of the output image consistent with the input image.

[0007] Furthermore, the multi-branch feature extraction module includes shallow branches, middle branches and deep branches, wherein: The shallow branch includes the BM module and the FeatureEnhancement module; The middle-level branch includes the BM module, GCAConv module, and FeatureEnhancement module; The deep branch includes two GCAConv modules and a FeatureEnhancement module.

[0008] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0009] Furthermore, the GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule, the convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time, wherein, The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

[0010] Furthermore, the FeatureEnhancement module includes a multi-scale convolutional layer and a residual connection, wherein: The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0011] The image fusion method based on multi-branch feature fusion and attention mechanism, based on the above-mentioned image fusion system based on multi-branch feature fusion and attention mechanism, includes the following steps: S1. Input preprocessing: performing channel splicing on at least two modal medical images to be enhanced and the reference image, and normalizing the spliced ​​images; S2. Feature extraction: Feature extraction is performed on the spliced ​​image through three parallel branches: shallow layer, middle layer and deep layer, where: The shallow branch passes through the BM module and the FeatureEnhancement module in sequence to extract the local texture and edge information of the image and output the feature map; The middle branch passes through the BM module, GCAConv module and FeatureEnhancement module in sequence to extract the structure and semantic information of the image and output the feature map; The deep branch passes through two GCAConv modules and FeatureEnhancement modules in sequence to extract high-level semantic and contextual information of the image and output feature maps; S3, feature fusion: The feature maps output by the shallow branch, the middle branch, and the deep branch are spliced ​​together, and then fused by 1×1 convolution to generate fused features; S4. Image reconstruction: Use the Sigmoid activation function to process the fused features to generate an output image, and keep the size of the output image consistent with the input image.

[0012] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0013] Furthermore, the GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule, the convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time, wherein, The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

[0014] Furthermore, the FeatureEnhancement module includes a multi-scale convolutional layer and a residual connection, wherein: The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0015] Furthermore, in S3, the feature maps output by the shallow branch, the middle branch and the deep branch are spliced ​​in the channel dimension to obtain a spliced ​​feature map, and the number of channels of the spliced ​​feature map is the sum of the number of channels of the feature maps output by each branch; in S4, the fusion feature is resized by bilinear interpolation so that the size of the output image is consistent with the input image.

[0016] Beneficial effects of the present invention: The image fusion system and method based on multi-branch feature fusion and attention mechanism of the present invention have the following advantages: Multi-branch structure design: Through three parallel branches at the shallow, middle, and deep levels, it captures local details and global semantic information of the image, enhancing the model's expressive power. Attention mechanism fusion: The introduction of dual channel and spatial attention mechanisms improves feature selection capabilities and effectively suppresses interference from irrelevant features; Efficient feature enhancement module: combines multi-scale convolution and residual structure to achieve efficient feature enhancement and restoration; Flexible and scalable: Supports a variety of image enhancement tasks, such as dehazing, denoising, low-light enhancement, and super-resolution, and can further improve performance by introducing the Transformer module. Lightweight design: The parameters of each module are controllable, suitable for deployment on mobile and embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the overall framework diagram of the image fusion system based on multi-branch feature fusion and attention mechanism of the present invention; Figure 2 Schematic diagram of the structure of the multi-branch feature extraction module of the present invention; Figure 3 This is a flowchart of the attention mechanism of the present invention; Figure 4 It is a structural diagram of the FeatureEnhancement module of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Reference Figures 1-4 As shown, the image fusion system based on multi-branch feature fusion and attention mechanism includes an input processing module, a multi-branch feature extraction module, a feature fusion module and an image reconstruction module. The input processing module, the multi-branch feature extraction module, the feature fusion module and the image reconstruction module are connected in sequence, wherein, An input processing module, configured to perform channel splicing on at least two modal medical images to be enhanced and a reference image, and to perform normalization processing on the spliced ​​images; Multi-branch feature extraction module, used to extract features from the spliced ​​images; The feature fusion module is used to perform channel splicing on the feature maps output by the multi-branch feature extraction module, and then generate fused features through 1×1 convolution fusion; The image reconstruction module is used to process the fused features using a Sigmoid activation function to generate an output image, and keep the size of the output image consistent with the input image.

[0020] Specifically, the present invention introduces the BM module, the GCAConv module and the FeatureEnhancement module to construct three parallel branches of shallow, middle and deep layers, and adopts a multi-scale feature fusion strategy, which significantly improves the fusion quality of medical images. Specifically, the parallel branch design enables features at different levels to be extracted synchronously and fully. The shallow branches focus on capturing local texture and edge information, the middle branches use the attention mechanism of the GCAConv module to strengthen the structure and middle-level semantic features, and the deep branches deeply mine high-level semantics and global context information. Features at different levels complement each other and provide a rich source of information for subsequent fusion. The convolutional layer, batch normalization layer, channel attention submodule and spatial attention submodule of the GCAConv module work together to accurately weight the attention of features in the channel and spatial dimensions, highlight key features, suppress redundant and interfering information, and enhance the pertinence and effectiveness of feature expression. The multi-scale convolutional layer of the Feature Enhancement module effectively captures features at different scales of the image, taking into account information from small-scale fine textures to macro-global structural information. Combined with residual connections, it not only avoids information loss but also accelerates model convergence. The final output fused image retains rich details while having a clearer and more accurate overall structure, thereby providing a more comprehensive and reliable image basis for clinical diagnosis, helping doctors to more accurately judge the condition and improve diagnostic accuracy.

[0021] Furthermore, the multi-branch feature extraction module includes shallow branches, middle branches and deep branches, wherein: The shallow branch includes the BM module and the FeatureEnhancement module; The middle-level branch includes the BM module, GCAConv module, and FeatureEnhancement module; The deep branch includes two GCAConv modules and a FeatureEnhancement module.

[0022] Specifically, the present invention sets three parallel branches, shallow, middle and deep, in a multi-branch feature extraction module, so that different branches can extract image features in a targeted manner. The shallow branch extracts local texture and edge information with the help of the BM module and the Feature Enhancement module, the middle branch extracts structure and semantic information through the BM module, the GCAConv module and the Feature Enhancement module, and the deep branch extracts high-level semantics and context information using two GCAConv modules and the Feature Enhancement module. The features extracted by each branch complement each other, enriching the hierarchy and comprehensiveness of the features. This multi-branch design avoids the limitations of a single branch in feature extraction, can take into account both the local details and the global semantics of the image, improves the richness of feature expression, and provides a better foundation for subsequent feature fusion, thereby improving the quality of medical image fusion, so that the fused image can more comprehensively present the key information of medical images of different modalities and better assist medical diagnosis.

[0023] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0024] Specifically, the present invention designs the convolution layer of the BM module of the shallow branch to include three consecutive 3×3 convolutions, which can gradually extract and refine the local features of the image. This continuous convolution structure reduces the number of parameters while maintaining sufficient model expression capabilities, which helps to capture the fine texture and edge information of the image. In the GCAConv module of the middle branch, the fully connected layer of the channel attention submodule is designed to include two consecutive 1×1 convolutions, which can more effectively model the dependencies between channels and enhance the focus on key feature channels; the spatial attention submodule adopts 7×7 convolution, which can expand the receptive field and capture a wider range of spatial context information, so that the model can better understand the structural relationship between different regions in the image. The GCAConv module of the deep branch also adopts a similar design to further enhance the extraction of high-level semantics and global context information. These specific convolutional structure designs enable each branch to more specifically extract features of different levels and types, improve the efficiency and accuracy of feature extraction, and provide higher-quality and more representative feature expressions for subsequent feature fusion, thereby improving the quality and effect of medical image fusion.

[0025] Furthermore, the GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule, the convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time, wherein, The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

[0026] Specifically, the present invention significantly improves the quality and effect of medical image fusion by designing the multi-scale fusion module to include a feature weighting layer, a multi-scale feature extraction layer and a feature aggregation layer. The feature weighting layer can adaptively allocate the importance of different branch features in the fusion process by calculating the weights of each branch feature, ensuring that key features are fully retained and enhanced, and suppressing redundant information. The multi-scale feature extraction layer uses convolution kernels of different sizes to process the weighted features, which can simultaneously capture the feature information of the image at different scales, and can effectively express everything from local details to global structures, enriching the diversity of fusion features. The feature aggregation layer integrates multi-scale features, and through residual connections and convolution operations, further optimizes feature expression, and enhances the complementarity and consistency of features. This design enables the fused image to retain the advantageous features of each branch at the same time, including both shallow fine texture and edge information, as well as mid-level structural semantics and deep global contextual information, thereby providing a more comprehensive and accurate image basis for medical diagnosis, helping doctors to observe lesion characteristics and tissue structures more clearly.

[0027] Furthermore, the FeatureEnhancement module includes a multi-scale convolutional layer and a residual connection, wherein: The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0028] Specifically, the present invention realizes the adaptive evaluation and allocation of the importance of each branch feature by adopting a weight calculation method based on the attention mechanism in the feature weighting layer, which significantly improves the pertinence and accuracy of medical image fusion. Specifically, the channel attention unit can automatically learn the importance of each channel feature through global average pooling and a fully connected layer structure, highlighting the feature channels that are crucial for medical diagnosis, such as the feature expression of the lesion area; the spatial attention unit analyzes the spatial dimension of the feature map through convolution operations to accurately locate the key areas in the image, such as the tumor boundary or the edge of the tissue structure. The synergistic effect of this dual attention mechanism enables the complementary information of medical images of different modalities to be fully integrated while suppressing the interference of irrelevant information. For example, when fusing PET and CT images, channel attention can enhance the expression of metabolically active areas, while spatial attention focuses on the precise location of anatomical structures, thereby generating a fused image with more diagnostic value, providing doctors with clearer and more comprehensive lesion information, and assisting clinical decision-making.

[0029] The image fusion method based on multi-branch feature fusion and attention mechanism, based on the above-mentioned image fusion system based on multi-branch feature fusion and attention mechanism, includes the following steps: S1. Input preprocessing: performing channel splicing on at least two modal medical images to be enhanced and the reference image, and normalizing the spliced ​​images; S2. Feature extraction: Feature extraction is performed on the spliced ​​image through three parallel branches: shallow layer, middle layer and deep layer, where: The shallow branch passes through the BM module and the FeatureEnhancement module in sequence to extract the local texture and edge information of the image and output the feature map; The middle branch passes through the BM module, GCAConv module and FeatureEnhancement module in sequence to extract the structure and semantic information of the image and output the feature map; The deep branch passes through two GCAConv modules and FeatureEnhancement modules in sequence to extract high-level semantic and contextual information of the image and output feature maps; S3, feature fusion: The feature maps output by the shallow branch, the middle branch, and the deep branch are concatenated and then fused by 1×1 convolution to generate fused features; S4. Image reconstruction: Use the Sigmoid activation function to process the fused features to generate an output image, and keep the size of the output image consistent with the input image.

[0030] Specifically, this invention effectively improves the comprehensiveness and robustness of feature extraction by designing the multi-scale feature extraction layer as a parallel convolutional layer consisting of three different kernel sizes: 3×3, 5×5, and 7×7. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, effectively improving the comprehensiveness and robustness of feature extraction. The different kernel sizes capture local image details (3×3 convolution), medium-scale structure (5×5 convolution), and global context (7×7 convolution), respectively, ensuring that all features in multimodal medical images, from subtle textures to macroscopic tissues, are fully exploited. The addition of the batch normalization layer stabilizes the distribution of the outputs of each layer, avoiding gradient anomalies during training, while the ReLU activation function enhances the model's ability to represent complex features by introducing nonlinear transformations. This design enables the multi-scale feature extraction layer to generate richer and more discriminative feature representations, providing high-quality input for subsequent feature aggregation. Ultimately, this improves the quality of the fused image, ensuring that the fused result retains both precise local information and a complete global structure, better meeting the clinical diagnostic requirements for both image detail and overall relevance.

[0031] Furthermore, the BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0032] Furthermore, the GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule, the convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time, wherein, The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

[0033] Furthermore, the FeatureEnhancement module includes a multi-scale convolutional layer and a residual connection, wherein: The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

[0034] Furthermore, in S3, the feature maps output by the shallow branch, the middle branch and the deep branch are spliced ​​in the channel dimension to obtain a spliced ​​feature map, and the number of channels of the spliced ​​feature map is the sum of the number of channels of the feature maps output by each branch; in S4, the fusion feature is resized by bilinear interpolation so that the size of the output image is consistent with the input image.

[0035] Specifically, the present invention realizes efficient fusion and accurate reconstruction of multimodal medical images by constructing a medical image fusion system including a multi-branch feature extraction module, a multi-scale fusion module and an image reconstruction module. The shallow, middle and deep parallel structures of the multi-branch feature extraction module can respectively capture local texture, middle-level semantics and global context information to ensure the comprehensive extraction of features at different levels; the multi-scale fusion module adaptively integrates the features of each branch through feature weighting, multi-scale extraction and aggregation, highlights key information and suppresses redundancy; the image reconstruction module efficiently converts the fused features into the target image to ensure output quality and resolution. The modules work together to enable the system to fully tap the complementary information of images of different modalities, and to enhance the pertinence and richness of feature expression through attention mechanisms and multi-scale strategies. The fused image finally generated takes into account both detail integrity and structural accuracy, providing more comprehensive and reliable image support for clinical diagnosis, and helping to improve diagnostic efficiency and accuracy.

[0036] The following is a specific example of the present invention: The multi-branch attention fusion medical image enhancement system of the present invention adopts a modular design, and the overall architecture is as follows: Figure 1 As shown in the figure, it mainly includes four core modules: input processing module (used to receive and preprocess multimodal medical images), multi-branch feature extraction module (constructs three parallel branches of shallow, middle and deep layers to extract local details, structural information and high-level semantics of the image respectively), feature fusion module (performs channel splicing and weighted fusion of the features output by each branch to generate fusion features) and output processing module (maps the fusion results back to the image space to output high-quality medical images).

[0037] S1: The core function of the input processing module is to preprocess multimodal medical images to prepare for subsequent feature extraction. The specific implementation is as follows: Image input and channel stitching: At least two modal medical images are received, such as an MRI image (X) and a SPECT image (Y), both of which have the same size, such as 256×256; Then, a splicing operation is performed on the channel dimension to splice the two modal medical images (such as MRI and SPECT) on the channel dimension to form a multi-channel input tensor. The shape of the spliced ​​tensor is B, C, H, W; where B is the batch number B, C=2 is the number of channels, and H and W are the height and width of the image, respectively.

[0038] Initial feature extraction: Use a 3×3 convolutional layer to perform initial feature extraction on the concatenated input, expanding the number of channels from 2 to 64: In this step, a 2D convolutional layer is defined with 2 input channels, 64 output channels, a kernel size of 3×3, and 1-pixel padding added to the edges of the input tensor. Applying this convolutional layer changes the shape of the input tensor x from B, 2, H, W to B, 64, H, W. Because the stride is set to 1, the spatial dimensions (height and width) of the output tensor after the convolution operation are the same as the input tensor. This step is the initial stage of feature extraction and provides the foundation for subsequent feature processing and enhancement.

[0039] S2: The multi-branch feature extraction module is the core part, consisting of three parallel branches: shallow, middle, and deep, which extract image features at different levels. The structure of each branch is as follows: Specifically, the shallow branch structure: BMModule + FeatureEnhancement; its function is to extract local texture and edge information of the image.

[0040] The basic module (BMModule) effectively extracts and enhances the basic features of an image by defining two consecutive convolutional blocks (ConvBlocks). Each convolutional block consists of a 3×3 convolution, batch normalization (BN), and a ReLU activation function. The first convolutional block maps the number of input features from 64 to 64 channels, extracting basic features. The second convolutional block further enhances these features, gradually increasing their expressive power through successive convolution operations. This design not only extracts the basic structural information of the image, but also stabilizes the training process and introduces nonlinearity through batch normalization and ReLU activation, providing a solid foundation for subsequent feature processing and fusion.

[0041] The feature enhancement module then effectively enhances the expressiveness of features by defining multi-scale convolutional layers (1×1 convolution for dimensionality reduction, 3×3 convolution for local feature extraction, and 1×1 convolution for dimensionality increase) and residual connections. Specifically, 1×1 convolutions are first used to reduce the number of input feature channels from 64 to 32, reducing computational effort while maintaining feature expressiveness. Next, 3×3 convolutions are used to extract local features, capturing texture and structural information in the image. Finally, 1×1 convolutions are used to increase the number of channels from 32 back to 64, integrating feature information. Furthermore, if the number of input and output channels is not equal, a 1×1 convolution layer is used to adjust the input channels to ensure the feasibility of residual connections. Residual connections directly add a portion of the input features to the output features, avoiding the loss of important information during multi-layer convolution operations while enhancing feature expressiveness.

[0042] Specifically, the middle branch structure is: BMModule + GCAConv + FeatureEnhancement; Its function is to introduce the attention mechanism and enhance the image structure and semantic information.

[0043] Among them, the generalized convolution module with attention mechanism (GCAConv) effectively enhances the feature expression ability by combining 3×3 convolution operation with channel attention mechanism (CAAttention) and spatial attention mechanism (SpatialAttention).

[0044] Specifically, the local features are first extracted through a 3×3 convolutional layer, and then the channel attention weight and spatial attention weight are generated by the channel attention module and the spatial attention module respectively. Finally, the intermediate features are multiplied by the channel attention weight and the spatial attention weight to obtain the enhanced features.

[0045] Specifically, the deep branch structure is: GCAConv ×2 + FeatureEnhancement; its function is to extract high-level semantics and global context information of the image.

[0046] Among them, in the feature enhancement module, the expressive ability of the features is effectively enhanced by cascading two generalized convolutional modules (GCAConv) with attention mechanism.

[0047] Specifically, we first define the first generalized convolution module with attention mechanism, whose input and output channels are both 64. This module extracts and enhances features by combining convolution operation and attention mechanism.

[0048] Next, we define a second generalized convolutional module with an attention mechanism, whose number of input and output channels is also 64. This module further strengthens the features and improves their expressiveness.

[0049] During the forward propagation process, the input feature x first passes through the first generalized convolution module with attention mechanism to obtain the intermediate features.

[0050] Then, the intermediate features are used as input to the second generalized convolutional module with attention mechanism, and the enhanced features are finally output.

[0051] S3: The feature fusion module is responsible for integrating the features output by the shallow, middle, and deep branches to generate a unified fused feature tensor. The specific steps are as follows: Among them, the feature fusion module realizes the fusion of multi-scale features by splicing the features of shallow, middle and deep branches.

[0052] Specifically, the input feature x first passes through the shallow branch to extract shallow features, whose shape is B, 64, H, W, where B is the batch size, 64 is the number of channels, and H and W are the height and width of the feature map, respectively.

[0053] Next, the input feature x passes through the middle-level branch to extract the middle-level features, whose shape is also B, 64, H, W. The middle-level branch introduces an attention mechanism on the basic features to strengthen the semantic expression of the middle-level features.

[0054] Then, the input feature x passes through the deep branch to extract deep features, whose shape is also B, 64, H, W. The deep branch deeply mines the semantic features and multimodal complementary information of the image through multi-layer convolution and attention mechanism.

[0055] Finally, the features extracted by the shallow, middle and deep branches are concatenated in the channel dimension to form the fused features, whose shape is B, 192, H, W.

[0056] The fused convolution and activation uses a 1×1 convolution to compress the number of channels to 1, and a sigmoid activation function is used to generate the final output. In the feature fusion module, the fused multi-scale features are further processed through a 1×1 convolution layer and a sigmoid activation function to generate the final output image. Specifically, the fused features are first passed through a 1×1 convolution layer to compress the number of channels from 192 to 1, integrating the feature information from different channels. Then, a sigmoid activation function is used to map the intermediate feature values ​​to the range [0, 1] to generate the final output image. This design not only reduces the feature dimensionality but also preserves important feature information, ensuring that the pixel values ​​of the output image are within a reasonable range.

[0057] S4: The output processing module ensures that the fused image is consistent with the input image in terms of size and value range. Specifically, it includes: Specifically, size alignment: In the output processing module, a size adjustment mechanism is specially designed to ensure the consistency of the fused output image and the input feature image in the spatial dimension.

[0058] First, the system compares the shape of the fused output image with the shape of the input feature image. If there is a difference in the spatial dimensions (i.e., height and width) between the two, a resizing procedure is automatically initiated. Using a bilinear interpolation function, the output image is upsampled or downsampled, with the spatial dimensions of the input feature image as the target size. During this process, the parameters of the bilinear interpolation function are carefully set to mode='bilinear' and align_corners=True to ensure that the interpolated image is not only precisely aligned with the target size, but also visually smooth and continuous, avoiding distortion or blur. This effectively solves the size mismatch problem that may arise during feature extraction and fusion, ensuring that the output image is perfectly aligned with the input feature image in terms of spatial dimensions, thereby providing a high-quality data foundation for subsequent image processing and analysis, further improving the performance and reliability of the entire system.

[0059] Then, the numerical range is standardized: Since the Sigmoid activation function limits the output to the [0, 1] interval, it can be further linearly transformed to a suitable range (such as [0, 255]) according to the requirements of medical images to adapt to the DICOM or NIfTI format storage standard.

[0060] The image fusion system and method based on multi-branch feature fusion and attention mechanism of the present invention, by constructing a multi-branch feature extraction module containing three parallel branches of shallow, middle and deep layers, combined with the BM module, GCAConv module and FeatureEnhancement module, realizes the full extraction of different levels of features such as local texture, middle semantics and global context; the multi-scale fusion module adaptively integrates the features of each branch through feature weighting, multi-scale extraction and aggregation, with the help of the attention mechanism, highlights key information and suppresses redundancy; the image reconstruction module efficiently converts the fused features into the target image, ensuring the output quality and resolution. The synergistic effect of each module enables the fused medical image to take into account both detail integrity and structural accuracy, providing more comprehensive and reliable image support for clinical diagnosis, effectively improving diagnostic efficiency and accuracy, and at the same time, the lightweight design makes it suitable for edge device deployment, with strong practicality and scalability.

[0061] While the specific embodiments of the present invention have been described in detail above, these are intended to be exemplary only, and the present invention is not limited thereto. Any equivalent modifications or substitutions to the present invention that would be apparent to those skilled in the art are also within the scope of the present invention. Therefore, any equivalent modifications or substitutions made without departing from the spirit and scope of the present invention are intended to be encompassed within the scope of the present invention.

Claims

1. Image fusion system based on multi-branch feature fusion and attention mechanism, characterized by: It includes an input processing module, a multi-branch feature extraction module, a feature fusion module and an image reconstruction module, which are connected in sequence. An input processing module, configured to perform channel splicing on at least two modal medical images to be enhanced and a reference image, and to perform normalization processing on the spliced ​​images; Multi-branch feature extraction module, used to extract features from the spliced ​​images; The feature fusion module is used to perform channel splicing on the feature maps output by the multi-branch feature extraction module, and then generate fused features through 1×1 convolution fusion; The image reconstruction module is used to process the fused features using a Sigmoid activation function to generate an output image, and keep the size of the output image consistent with the input image.

2. The image fusion system based on multi-branch feature fusion and attention mechanism according to claim 1, characterized in that: The multi-branch feature extraction module includes shallow branches, middle branches and deep branches, wherein, The shallow branch includes the BM module and the FeatureEnhancement module; The middle-level branch includes the BM module, GCAConv module, and FeatureEnhancement module; The deep branch includes two GCAConv modules and a FeatureEnhancement module.

3. The image fusion system based on multi-branch feature fusion and attention mechanism according to claim 2, characterized in that: The BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

4. The image fusion system based on multi-branch feature fusion and attention mechanism according to claim 2, characterized in that: The GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule. The convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time. The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

5. The image fusion system based on multi-branch feature fusion and attention mechanism according to claim 2, characterized in that: The FeatureEnhancement module includes multi-scale convolutional layers and residual connections, where The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

6. An image fusion method based on multi-branch feature fusion and attention mechanism, based on the image fusion system based on multi-branch feature fusion and attention mechanism according to any one of claims 1 to 5, characterized in that: The following steps are involved: S1. Input preprocessing: performing channel splicing on at least two modal medical images to be enhanced and the reference image, and normalizing the spliced ​​images; S2. Feature extraction: Feature extraction is performed on the spliced ​​image through three parallel branches: shallow layer, middle layer and deep layer, where: The shallow branch passes through the BM module and the FeatureEnhancement module in sequence to extract the local texture and edge information of the image and output the feature map; The middle branch passes through the BM module, GCAConv module and FeatureEnhancement module in sequence to extract the structure and semantic information of the image and output the feature map; The deep branch passes through two GCAConv modules and FeatureEnhancement modules in sequence to extract high-level semantic and contextual information of the image and output feature maps; S3, feature fusion: The feature maps output by the shallow branch, the middle branch, and the deep branch are spliced ​​together, and then fused by 1×1 convolution to generate fused features; S4. Image reconstruction: Use the Sigmoid activation function to process the fused features to generate an output image, and keep the size of the output image consistent with the input image.

7. The image fusion method based on multi-branch feature fusion and attention mechanism according to claim 6, characterized in that: The BM module includes two consecutive ConvBlocks, each ConvBlock includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

8. The image fusion method based on multi-branch feature fusion and attention mechanism according to claim 6, characterized in that: The GCAConv module includes a convolutional layer, a batch normalization layer, a channel attention submodule and a spatial attention submodule. The convolutional layer is connected to the batch normalization layer, and the batch normalization layer is connected to the channel attention submodule and the spatial attention submodule at the same time. The channel attention submodule is used to generate channel attention weights, The spatial attention submodule is used to generate spatial attention weights, The output features of the convolutional layer are multiplied by the channel attention weight and the spatial attention weight to obtain enhanced features.

9. The image fusion method based on multi-branch feature fusion and attention mechanism according to claim 6, characterized in that: The FeatureEnhancement module includes multi-scale convolutional layers and residual connections, where The multi-scale convolution layer includes a 1×1 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer connected in sequence, which is used to capture feature information of the image at different scales; The residual connection is used to fuse the input features with the output features of the multi-scale convolutional layer.

10. The image fusion method based on multi-branch feature fusion and attention mechanism according to claim 6, characterized in that: In S3, the feature maps output by the shallow branch, the middle branch and the deep branch are spliced ​​in the channel dimension to obtain a spliced ​​feature map, and the number of channels of the spliced ​​feature map is the sum of the number of channels of the feature maps output by each branch; in S4, the size of the fused feature is resized by bilinear interpolation so that the size of the output image is consistent with the size of the input image.

Images