A multimodal image fusion system and method based on multi-scale residuals and feature aggregation

Abstract

This invention relates to the field of image processing technology, and in particular to a multimodal image fusion system and method based on multi-scale residuals and feature aggregation. The system utilizes a dual-stream residual network backbone to acquire image features at different levels, enhances the feature information of different modalities through a differential feature enhancer, and then aggregates global and local feature information using a local-global feature extraction group to further enhance the clarity of the fused image and the semantic consistency between different modalities. A topological feature interaction pyramid is used to perform dense interactions with rich, multi-stage features, synthesizing a fused image with prominent edges and texture details and significant expression of infrared target regions. The method combines basic feature alignment loss, gradient loss, infrared salient region enhancement loss, structural similarity loss, and inter-layer supervision loss. By constraining the result loss of the fused image and the process loss of features between multiple layers, the utilization rate of features is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multimodal image fusion system and method based on multi-scale residuals and feature aggregation. Background Technology

[0002] Multimodal image fusion is a process of complementary enhancement of information. It aims to integrate multi-source images into a single multi-information image. The task of multimodal image fusion is to combine the advantages of different modalities to maximize the function and utility, and to help observers to better analyze and judge.

[0003] Current convolutional neural network (CNN) methods cannot effectively utilize global features, which is an inherent limitation. Although researchers use large kernel convolutions or dilated convolutions to expand the receptive field of CNNs, this increases the computational cost of the model, and the improvement in local features is extremely limited, sometimes even leading to a significant loss of detail. The Transformer method, however, uses a patch mechanism to divide the image into multiple sub-images, then projects the features of each different patch into vectors, and further enhances the association weights of different patches on the global patch using window self-attention (W-MHSA) and moving window self-attention (SW-MHSA) mechanisms. This mechanism redistributes the features of the entire image, thereby greatly improving the utilization rate of global features.

[0004] Chinese patent application CN120410883A, published on August 1, 2025, entitled "A Unified Multimodal Infrared and Visible Light Image Fusion and Conversion Method," discloses the use of multi-level CNN and Transformer mechanisms to obtain global and local feature information of different modalities. However, its loss function design is relatively rudimentary, only including three types of loss: intensity loss, texture loss, and structural similarity loss, lacking constraints on inter-layer features. This directly leads to a deterioration in the fusion effect. Chinese patent application CN111161201A, published on May 15, 2020, entitled "Infrared and Visible Light Image Fusion Method Based on Detail Enhancement Channel Attention," discloses the use of a detail enhancement module. However, judging from the effect diagram, excessive detail enhancement significantly affects the infrared radiation area, resulting in a dull color in the salient target area of ​​the fused image.

[0005] Furthermore, the current fusion methods have very rudimentary loss function designs, which cannot effectively control different fusion effects. Summary of the Invention

[0006] In view of this, the present invention aims to provide a multimodal image fusion system and method based on multi-scale residuals and feature aggregation. It fully extracts feature information from different modalities using a dual-stream residual feature extraction backbone, and combines it with a CNN network containing dense connections and channel spatial attention to fully utilize the global and local features of the image. Furthermore, the present invention incorporates five different losses: basic feature alignment loss, gradient loss, infrared salient region enhancement loss, structural similarity loss, and inter-layer supervision loss. The inter-layer supervision loss is a process loss, while the other four are outcome losses. By constraining the outcome loss of the fused image and the process loss of the inter-layer features, the utilization rate of features is improved.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0008] A multimodal image fusion system based on multi-scale residuals and feature aggregation includes: a dual-stream residual feature extraction backbone, which performs multi-scale residual processing on two different modalities of input images to obtain first-modal features and second-modal features at multiple scales; a differential feature enhancement group, including differential feature enhancers with the same number of scales as the first-modal features; in each differential feature enhancer, the first-modal features and second-modal features at the corresponding scales are subjected to channel and spatial attention operations, and the processed first-modal features and second-modal features are subjected to residual and fusion operations to obtain the enhanced features at the current scale; a local-global feature extraction group, including local-global feature extractors with the same number of scales as the differential feature enhancers; in each local-global feature extractor, the enhanced features from the corresponding differential feature enhancers are subjected to global and local feature extraction, and the global and local features are then fused to obtain local-global features; and a topological feature interaction pyramid, which performs feature interaction at different levels and stages on the local-global features at multiple different scales output by the local-global feature extraction group to obtain the final fused image.

[0009] Furthermore, the dual-stream residual feature extraction backbone includes a first-mode residual feature extraction branch and a second-mode residual feature extraction branch, wherein: in the first-mode residual feature extraction branch, the first-mode image is subjected to multiple consecutive residual processing, and each residual processing yields a first-mode feature at a certain scale; in the second-mode residual feature extraction branch, the second-mode image is subjected to multiple consecutive residual processing, and each residual processing yields a second-mode feature at a certain scale.

[0010] Furthermore, in each differential feature enhancer: spatial attention operations are performed on the first modal feature and the second modal feature at the corresponding scale, and then the features after spatial attention operations are added to the first modal feature and the second modal feature respectively to obtain the first modal spatial feature and the second modal spatial feature; a difference operation is performed on the first modal spatial feature and the second modal spatial feature, and a channel attention operation is performed on the obtained differential features; the features obtained by the channel attention operation are multiplied with the first modal spatial feature and the second modal spatial feature, and then added to the first modal feature and the second modal feature respectively, and the two types of features are fused to obtain the enhanced feature at the current scale.

[0011] Furthermore, in each local-global feature extractor: the enhanced features from the corresponding differential feature enhancer are encoded; global and local feature extraction are performed simultaneously on the encoded features; the extracted global and local features are fused, and then an FFN operation is performed on the fused features to obtain local-global features; during global feature extraction, the input features are simultaneously extracted using dual-path query features, key features, and value features; the two extracted query features are fused and then subjected to feature fusion attention operation; the features processed by feature fusion attention operation are combined with the two extracted key features and two value features respectively, resulting in two preliminary features; the two preliminary features are then fused, and then the fused features are sequentially subjected to feature fusion attention operation and multilayer perceptron operation to obtain global features; during local feature extraction, the input features are subjected to at least two consecutive channel-spatial aggregation operations to obtain local features; during each channel-spatial aggregation operation, the input features are simultaneously subjected to spatial attention operation and channel attention operation, and the features obtained from the two attention operations are fused to obtain output features.

[0012] Furthermore, during the feature fusion attention operation: the input features are grouped to obtain multiple sub-features; each sub-feature is divided into channels, and the divided features are subjected to dual-path operation; in each path operation, the features are scaled and shifted by channel attention, and then the operated features are activated by sigmoid, and the obtained features are fused with the input features of the current path operation to obtain the output of the current path operation; the inputs of the dual-path operation are fused and grouped and shuffled to obtain the output features.

[0013] Furthermore, during the spatial attention operation on the input features: the input features are matrix-sized to obtain reconstructed features, and then a two-branch operation is performed on the reconstructed features: in the first branch operation, the reconstructed features are simultaneously subjected to average pooling in the X and Y directions, and then the two pooled features are fused. The fused features are then split into channels to obtain a first width feature and a first height feature; the first width feature and the first height feature are then subjected to sigmoid activation operations, and then fused with the reconstructed features; the fused features are subjected to parallel matrix-sized adjustment and global average pooling. The features after matrix-sized adjustment are input into the second branch. The features after global average pooling are subjected to softmax operation and then multiplied by a matrix tensor with the features from the second branch to obtain the output features of the first branch. In the second branch operation, global average pooling, as well as average pooling in the X and Y directions, are performed on the reconstructed features simultaneously. After a softmax operation on the features after global average pooling, a matrix tensor multiplication operation is performed with the features after matrix size adjustment from the first branch. The features after matrix tensor multiplication are then combined with the output features of the first branch and the reconstructed features to obtain the output features of the spatial attention operation. The two pooled features after average pooling in the X and Y directions are fused, and the fused features are channel-split to obtain the second width feature and the second height feature. The second width feature and the second height feature are then activated by sigmoid operation and fused with the reconstructed features. After a matrix size adjustment operation is performed on the fused features, they are input into the first branch for matrix tensor multiplication.

[0014] Furthermore, during the channel attention operation on the input features, a dual-branch channel feature extraction is performed: in the first channel feature extraction branch, the input features are max-pooled, and the pooled features are then subjected to two fully connected operations to obtain the first channel features; in the second channel feature extraction branch, the input features are average-pooled, and the pooled features are then subjected to two fully connected operations to obtain the second channel features; the first and second channel features are then integrated and fused with the input features to obtain the output features of the channel attention operation.

[0015] Furthermore, the topological feature interaction pyramid comprises at least two stages, performed sequentially: In the first stage, cross-scale feature interactions are performed on multiple local-global features of different scales to obtain the output features of the first stage, and the number of output features in the first stage is the same as the number of input local-global features; cross-scale feature interactions are performed on the output features of the first stage to obtain the input features of the second stage; in the second to penultimate stages, cross-scale feature interactions are performed on the input features of each stage to obtain the output features of each stage, and the number of output features in each stage is the same as the number of input features in each stage; cross-scale feature interactions are performed on the output features of each stage to obtain the input features of the next stage; in the final stage, a Laplacian transform residual operation is performed on the input features to obtain the fused image.

[0016] A multimodal image fusion method based on multi-scale residuals and feature aggregation includes:

[0017] S1: Obtain image datasets of two different modalities and the corresponding fused image datasets; preprocess the three image datasets to obtain the training set;

[0018] S2: Construct a multimodal image fusion system as provided in this invention;

[0019] S3: Train the system constructed in step S2 using the training set obtained in step S1 to obtain a multimodal image fusion model, wherein: images of two different modalities in the training set are used as training input, and the fused image in the training set is used as training output; during the training process, the multimodal image fusion system is trained using result loss and inter-layer loss; result loss calculates the loss of the fused image obtained by the multimodal image fusion model, and inter-layer loss calculates the loss of the features output by the local-global feature extraction group in the multimodal image fusion model; after weighted summation of result loss and inter-layer loss, the total loss is obtained;

[0020] S4: Input the two different modal images to be fused into the multimodal image fusion model obtained in step S3 for fusion to obtain the corresponding fused image.

[0021] Furthermore, in step S3, the total loss is:

[0022] L total =L res +μL layers ;

[0023] Among them, L total L represents the total loss. res L represents the loss as a result. layers The interlayer loss is represented by μ, where μ represents the weight of the interlayer loss; the final loss is obtained by the following formula:

[0024] L res =αL align +βL gradient +γL foces +ρL ssim ;

[0025] Among them, L align L represents the basic feature alignment loss. gradient L represents the gradient loss. foces L represents the significant enhancement loss. ssim The loss represents the overall structural similarity loss, where α, β, γ, and ρ represent the corresponding weights of the four losses. In the basic feature alignment loss, the larger pixel value from the two different modalities is taken pixel by pixel, and the fused image is constrained by L1 loss based on the larger pixel value. The gradient loss includes forward gradient loss and backward gradient loss. Gradients are calculated for the two different modalities and the fused image, and the maximum gradient value of the two different modalities is calculated. The L1 loss is calculated using the maximum gradient value as the first term and the gradient of the fused image as the second term to obtain the forward gradient loss. The L1 loss is calculated using the gradient of the fused image as the first term and the maximum gradient value as the second term to obtain the backward gradient loss. The forward gradient loss and backward gradient loss are combined with the L1 loss between the gradient of the fused image and the gradient of one of the different modalities to obtain the gradient loss. The salient region enhancement loss is obtained by the following formula:

[0026] ;

[0027] Among them, I f Let I represent a fused image, E represent one of two different modalities, and M represent the expected value. ir The brightness region mask is obtained from the following formula:

[0028] M ir =Sig((max(I1,I2)-0.5)×10);

[0029] Where Sig represents the sigmoid activation function; the interlayer loss is obtained by the following formula:

[0030] ;

[0031] Where S represents the Sobel edge gradient operator, and This represents the corresponding features of two different modalities of images in the i-th layer of the system; Let L1 represent the feature map of the i-th layer in the system, and L2 represent the L1 loss and L2 represent the L2 loss. This indicates that the i-th layer in the system takes the maximum value.

[0032] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0033] (1) In the multimodal image fusion system based on multi-scale residual and feature aggregation created by the present invention, image features at different levels are obtained by using a dual-stream residual network backbone, and each stage is connected through residuals, which greatly reduces the loss of feature information at different stages and scales; the feature information of different modalities is enhanced by a differential feature enhancer, and then the global and local feature information are aggregated by a local-global feature extraction group, which further enhances the clarity of the fused image and the semantic consistency between different modalities; the topological feature interaction pyramid is used to perform dense interaction with rich multi-stage features at different stages, synthesizing a fused image with prominent edges and texture details and significant expression of infrared target areas;

[0034] (2) In the multimodal image fusion system based on multi-scale residuals and feature aggregation described in this invention, local feature extraction and global feature extraction are proposed. In local feature extraction, the dense connection structure of CNN, the hybrid channel attention mechanism and spatial attention are used to comprehensively improve the expression of high-frequency detail features of the fused image and reduce the loss of detail texture information. In global feature extraction, a global feature extractor is proposed. The feature fusion attention operation is used to mix the query features of different branches to achieve parameter sharing of the query features. Then, parameter operations are performed with key features and value features to comprehensively enhance the long-range feature dependence and context interaction capability of the image.

[0035] (3) In the multimodal image fusion method based on multi-scale residuals and feature aggregation described in this invention, the result loss and process loss are combined, and a novel and effective multiple loss function is proposed. The loss function includes: basic feature alignment loss, gradient preservation loss, infrared salient region enhancement loss, comprehensive structural similarity loss and inter-layer supervision loss. The features of different modalities can be arranged in a clear hierarchy, and the redundant information of different modalities can be fused to enhance visual information and improve the overall quality of the fused image. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0037] Figure 1 A schematic diagram of the overall multimodal image fusion system based on multi-scale residuals and feature aggregation as described in the embodiments of the present invention;

[0038] Figure 2A schematic diagram of the dual-stream residual feature extraction backbone described in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the differential feature enhancer described in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the local-global feature extractor described in an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of the global feature extraction process described in the embodiments of the present invention;

[0042] Figure 6 A schematic diagram illustrating the operation of the multilayer sensor as described in the embodiments of the present invention;

[0043] Figure 7 A schematic diagram of the feature fusion attention operation described in the embodiments of the present invention;

[0044] Figure 8 A schematic diagram of the local feature extraction process described in the embodiments of the present invention;

[0045] Figure 9 A schematic diagram of the spatial attention operation described in the embodiments of the present invention;

[0046] Figure 10 A schematic diagram of the channel attention operation described in an embodiment of the present invention;

[0047] Figure 11 A flowchart illustrating the topological feature interaction pyramid as described in an embodiment of the present invention;

[0048] Figure 12 A flowchart illustrating the multimodal image fusion method based on multi-scale residuals and feature aggregation as described in the embodiments of the present invention;

[0049] Figure 13 The experimental results are shown in the embodiments described in this invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0052] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0053] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] like Figure 1As shown in the embodiment of the present invention, the multimodal image fusion system based on multi-scale residual and feature aggregation includes a dual-stream residual feature extraction backbone, a differential feature enhancement group, a local-global feature extraction group, and a topological feature interaction pyramid. The dual-stream residual feature extraction backbone performs multi-scale residual processing on the two different modalities of the input images, obtaining first-modal features and second-modal features at multiple scales. The differential feature enhancement group includes the same number of differential feature enhancers as the number of scales of the first-modal features. In each differential feature enhancer, the first-modal features and second-modal features at the corresponding scale are subjected to channel and spatial attention operations, and then the processed first-modal features and second-modal features are subjected to residual and fusion operations to obtain the enhanced features at the current scale. The local-global feature extraction group includes the same number of local-global feature extractors as the number of differential feature enhancers. In each local-global feature extractor, the enhanced features from the corresponding differential feature enhancer are used to extract global and local features, and then the global and local features are fused to obtain local-global features. The topological feature interaction pyramid performs feature interactions at different levels and stages on multiple local-global features output from the local-global feature extraction group at different scales to obtain the final fused image. In this invention, the two input images of different modalities are visible light images and infrared images, respectively; that is, this embodiment of the invention completes the fusion of visible light images and infrared images.

[0056] In some embodiments, the dual-stream residual feature extraction backbone is as follows: Figure 2 As shown, it includes a first modality residual feature extraction branch and a second modality residual feature extraction branch, wherein: in the first modality residual feature extraction branch, the first modality image is subjected to multiple consecutive residual processing, and each residual processing yields a first modality feature at a certain scale; in the second modality residual feature extraction branch, the second modality image is subjected to multiple consecutive residual processing, and each residual processing yields a second modality feature at a certain scale.

[0057] In this embodiment of the invention, the first modality residual feature extraction branch performs one convolutional basic feature extraction process and three consecutive residual processes on the visible light image I (corresponding to the first modality image mentioned above), as follows: Figure 2 As shown in (a), a convolution operation is performed on the visible light image using a convolution kernel of size 3×3×16. Then, batch normalization (BN) and ReLU activation are performed on the convolutional features to obtain the first visible light mode residual features with a scale of 640×480×16. As shown in the following formula:

[0058] ;

[0059] in, This indicates a convolution operation with a kernel size of 3×3 and the number of channels changed from 1 to 16.

[0060] Residual characteristics of the first visible light mode After performing three consecutive convolution and batch normalization (BN) operations, a PReLU activation operation is performed; simultaneously, the residual features of the first visible light mode are processed. Perform convolution and ReLU activation, then add the corresponding elements of the features obtained after PReLU activation to obtain the residual features of the second visible light mode. The above process can be represented by the following formula:

[0061] ;

[0062] in, This indicates a convolution operation with a kernel size of 3×3 and the number of channels changing from 16 to 32. This indicates a convolution operation with a kernel size of 3×3, where the number of channels changes from 32 to 64. This indicates a convolution operation with a kernel size of 3×3 and the number of channels changed from 64 to 32.

[0063] Residual characteristics of the second visible light mode After performing three consecutive convolution and batch normalization (BN) operations, a PReLU activation operation is performed; simultaneously, the residual features of the second visible light mode are processed. Perform convolution and ReLU activation, then add the corresponding elements of the features obtained after PReLU activation to obtain the residual features of the third visible light mode. The above process can be represented by the following formula:

[0064] ;

[0065] in, This indicates a convolution operation with a kernel size of 3×3 and the number of channels changed from 64 to 128. This indicates a convolution operation with a kernel size of 3×3 and the number of channels changed from 128 to 64.

[0066] Residual characteristics of the third visible light mode After performing three consecutive convolution and batch normalization (BN) operations, a PReLU activation operation is performed; simultaneously, the residual features of the third visible light mode are processed. Perform convolution and ReLU activation, then add the corresponding elements of the features obtained after PReLU activation to obtain the residual features of the fourth visible light mode. The above process can be represented by the following formula:

[0067] ;

[0068] in, This indicates a convolution operation with a kernel size of 3×3 and the number of channels changing from 128 to 256. This indicates a convolution operation with a kernel size of 3×3 and the number of channels changed from 256 to 128.

[0069] It should be noted that the first visible light mode residual feature in the embodiments of the present invention Second visible light mode residual characteristics Third visible light mode residual characteristics and the residual characteristics of the fourth visible light mode These all belong to the first-mode features at different scales mentioned above. The second-mode residual feature extraction branch performs one convolution process and three consecutive residual processes on the infrared image V (corresponding to the second-mode image mentioned above), as follows: Figure 2 As shown in (b), the processing procedure is consistent with the processing procedure of the first mode residual feature extraction branch for the visible light image I, and the first infrared mode residual features are obtained sequentially. Second infrared mode residual characteristics Third infrared mode residual characteristics and the residual characteristics of the fourth infrared mode. This will not be elaborated further here. Similarly, the residual characteristics of the first infrared mode... Second infrared mode residual characteristics Third infrared mode residual characteristics and the residual characteristics of the fourth infrared mode. All of them belong to the second modal features at different scales mentioned above.

[0070] In some embodiments, each differential feature enhancer is as follows: Figure 3 As shown. In each differential feature enhancer, spatial attention operations are performed on the first modal feature and the second modal feature at the corresponding scale. The spatial attention-enhanced features are then added to the corresponding first and second modal features to obtain the first and second modal spatial features. A difference operation is performed on the first and second modal spatial features, and channel attention operations are performed on the resulting differential features. The channel attention-enhanced features are multiplied by the first and second modal spatial features, and then added to the corresponding first and second modal features. The two types of features are then fused to obtain the enhanced features at the current scale. The differential feature enhancer is one of the core modules of the multimodal image fusion system provided by this invention. Its main function is to enhance the difference features between different modalities by differentiating between different images, thereby improving the detailed texture information of the fused image and enhancing the saliency of highlighted areas.

[0071] In this embodiment of the invention, the visible light modal feature Xir (Corresponding to the first modality feature mentioned above) Perform spatial attention operation A s The features after spatial attention manipulation are compared with the visible light modal features X. ir Adding them together yields the corresponding visible light modal space features. (Corresponding to the first modal space feature mentioned above), this process is represented by the following formula:

[0072] .

[0073] In this embodiment of the invention, spatial attention operation A s The processing includes: processing the input visible light modal features X ir A convolution operation with a stride of 1 and a kernel size of 3 is performed, with zero padding during the convolution process. The features after the convolution operation are then subjected to 2D batch normalization and ReLU activation. This processed feature is then subjected to another convolution operation with a stride of 1, a kernel size of 1, and zero padding. The convolutional features are then activated using a sigmoid function to obtain spatial weights. These spatial weights are then compared with the visible light modal feature X. ir Element-wise multiplication enhances spatial information, thus completing spatial attention operation A. s The processing procedure. Spatial attention operation A s The processing procedure can be represented by the following formula:

[0074] ;

[0075] Where σ represents the sigmoid activation operation, Conv 3×3 This represents a convolution operation with stride s=1, kernel size k=3, and zero-padding p=1. BN2d represents two-dimensional batch normalization. For infrared modal features X... vi The spatial attention operation performed (corresponding to the second modality feature mentioned above), and the corresponding infrared modality spatial feature obtained. The processing procedure (corresponding to the second modal space features mentioned above) and the acquisition of visible light modal space features. The processing procedure is the same as the previous one, so it will not be repeated here.

[0076] In obtaining visible light modal space features and infrared modal spatial features Next, a difference operation is performed on the two modal spatial features, and the resulting difference features are then subjected to channel attention operations. Specifically, in this embodiment of the invention, the visible light modal spatial features are... The elements in the subtraction correspond to the infrared modal space features. The elements in the first difference feature are then subtracted, and the resulting feature is activated using ReLU to obtain the first difference feature. ,Right now:

[0077] ;

[0078] Similarly, the infrared modal spatial features The elements in the subtraction correspond to the visible light mode space features. The elements in the first feature are then subtracted, and a ReLU activation operation is performed on the resulting feature to obtain the second difference feature. ,Right now:

[0079] ;

[0080] The first difference feature obtained Second difference features Perform channel attention operations separately to obtain the corresponding first channel weights. Second channel weight After multiplying the two channel weights by the corresponding elements of the two difference features respectively, and then applying the visible light modal features X... ir and infrared modal features X vi Corresponding elements are added together, and the two resulting features are then concatenated using channel concatenation to obtain the enhanced features. The above process can be represented by the following formula:

[0081] ;

[0082] Among them, F out Indicates enhanced features, This indicates channel splicing.

[0083] In this embodiment of the invention, the first difference feature Second difference features Perform channel attention operations in the same manner to apply them to the first difference features. Perform channel attention operations to obtain the first channel weights. For example:

[0084] For the first difference feature Perform global average pooling, then apply sigmoid activation to the pooled features to obtain feature enhancement weights. As shown in the following formula:

[0085] ;

[0086] Among them, P GAP This indicates a global average pooling operation;

[0087] Then enhance the weights of the features. Perform a convolution operation with shuffled weights to obtain the weights of the first channel. As shown in the following formula:

[0088] ;

[0089] Here, WS represents the weight shuffle convolution operation.

[0090] In some embodiments, each local-global feature extractor such as Figure 4 As shown. In each local-global feature extractor, the enhanced features from the corresponding differential feature enhancer are encoded. Global and local features are extracted simultaneously on the encoded features. The extracted global and local features are then fused, and the fused features are subjected to FFN (Feedback Forward) operation to obtain local-global features.

[0091] The process of extracting global features and obtaining global features is as follows: Figure 5 As shown, the input features are simultaneously extracted using a dual-path approach, extracting query features, key features, and value features. The two extracted query features are then fused and subjected to a feature fusion attention operation. The feature after the feature fusion attention operation is then combined with the two extracted key features and two value features, respectively, and subjected to window attention operations, resulting in two preliminary features. These two preliminary features are then fused, and subsequently subjected to feature fusion attention operations and a multilayer perceptron operation to obtain global features. This invention employs a dual-path global attention mechanism and enhances shared parameters through feature fusion attention operations, comprehensively improving the model's ability to capture long-range dependencies of complex features. During global feature extraction, the head uses a dual-path approach of query (Q), key (K), and value (V), specifying the number of heads in the multi-head attention mechanism. The shared Q key enables feature complementarity, allowing the model to focus on different subspaces of different inputs. Each head can independently learn different features, thereby improving the model's ability to acquire image context information. The feature fusion attention operation utilizes a grouping mechanism to reduce model complexity, a gating mechanism to adjust feature weights, and learnable scaling and bias parameters to enhance the model's non-linear expressive capabilities. After the dual-path Q-axis undergoes feature fusion attention operations to achieve parameter mixing and sharing, the multi-head attention is reassembled, and finally, feature projection is used to obtain features with rich global information.

[0092] In this embodiment of the invention, the global feature extraction process includes: performing dual-path depthwise convolution on the input features; after each path of depthwise convolution, performing three sets of parallel depthwise separable convolutions to obtain a set of query features, key features, and value features. It can be understood that during the global feature extraction process, two sets of query features, key features, and value features are obtained. In this embodiment of the invention, a matrix size reorganization operation is also performed on each query feature, key feature, and value feature. Taking the process of obtaining a set of query features Q1, key features K1, and value features V1 through one path as an example, the process can be represented by the following formula:

[0093] Q1=R(DFConv(DWSConv(X in )));

[0094] K1=R(DFConv(DWSConv(X in )));

[0095] V1=R(DFConv(DWSConv(X in )));

[0096] Where R represents matrix resizing, DFConv represents depthwise separable convolution, DWSConv represents depthwise convolution, and X represents... in This represents the input features. In this embodiment of the invention, the input features are further normalized before being processed by depthwise convolution. It can be understood that the above operations yield another set of query features Q2, key features K2, and value features V2. After fusing the obtained query features Q1 and Q2, a feature fusion attention operation is performed, followed by window attention operations combined with key features K1 and K2, and value features V1 and V2, respectively, to obtain the first preliminary features. Second preliminary features The above process is as follows:

[0097] ;

[0098] ;

[0099] Where M represents the feature fusion attention operation, d k Indicates the feature dimension.

[0100] For the first preliminary feature respectively Second preliminary features After layer normalization, the two features are fused again, and then the fused features are sequentially subjected to feature fusion attention, Dropout, layer normalization, and multilayer perceptron operations to obtain the global features. The above process can be represented by the following equation:

[0101] ;

[0102] Among them, X out The output global features are represented by MLP, which stands for Multilayer Perceptron Operation, and LN stands for Layer Normalization Operation.

[0103] In this embodiment of the invention, the operation process of the multilayer perceptron is as follows: Figure 6 As shown, after performing convolution and depthwise separable convolution on the input features in sequence, SiLU activation is performed. The features after SiLU activation and depthwise separable convolution are multiplied element-wise, and then batch normalization and convolution are performed on the multiplied features in sequence to obtain the output features.

[0104] In some embodiments, the feature fusion attention operation process is as follows: Figure 7 As shown, during the feature fusion attention operation, the input features are grouped to obtain multiple sub-features; each sub-feature is divided into channels, and the divided features are subjected to a dual-path operation; in each path operation, the features are scaled and shifted by channel attention, and then the operated features are activated by sigmoid, and the obtained features are fused with the input features of the current path operation to obtain the output of the current path operation; after fusing and regrouping the inputs of the dual-path operation, the output features are obtained.

[0105] In this embodiment of the invention, the input feature X is grouped to obtain multiple sub-features. Then each sub-feature Channel segmentation is performed to obtain the first channel sub-feature X. 10 Second channel sub-feature X 11 For the first channel sub-feature X 10 Second channel sub-feature X 11 Perform parallel dual-path operations. For the first channel sub-feature X... 10 Taking the processing branch as an example, for the first channel sub-feature X 10 Group normalization is performed, followed by channel attention scaling and parameter shifting operations on the normalized features, and then sigmoid activation is applied to the processed features. The sigmoid-activated features are then multiplied element-wise with the normalized features to obtain the first channel sub-feature X. 10 Corresponding first intermediate feature The above process can be represented by the following formula:

[0106] ;

[0107] ;

[0108] ;

[0109] Where G represents group normalization, and split represents channel partitioning. and These represent the learnable parameters for the attention scaling and parameter shifting operations, respectively. It is understandable that for the second channel sub-feature X... 11 Perform the same operation as described above to obtain the second intermediate feature. The first intermediate feature With the second intermediate feature After channel splicing, group rewashing is performed to obtain the output features.

[0110] The process of extracting local features and obtaining local features is as follows: Figure 8 As shown, at least two channel-space aggregation operations are performed on the input features to obtain local features. In this embodiment of the invention, residual operations are used in conjunction with the local feature extraction process. Specifically, for the input feature X... in Perform three consecutive channel-spatial aggregation operations on the input feature X. in After performing a 3×3 convolution operation and a ReLU activation operation, a first channel-spatial aggregation operation is performed to obtain the first channel-spatial aggregation feature X1. Then, a second channel-spatial aggregation operation is performed on the first channel-spatial aggregation feature X1, followed by a 3×3 convolution operation and a ReLU activation operation, and finally, the feature X1 is combined with the input feature X. in Combined, the second channel-spatial aggregation feature X2 is obtained; after performing a 3×3 convolution operation on the second channel-spatial aggregation feature X2 and a ReLU activation operation, a third channel-spatial aggregation operation is performed, which is then combined with the input feature X. in Combined with the first channel-spatial aggregation feature X1, we obtain the local feature X. out .

[0111] In each channel-space aggregation operation: spatial attention and channel attention operations are simultaneously performed on the input features, and the features obtained from the two attention operations are fused to obtain the output features. This invention utilizes dense connections as its main framework during local feature extraction to achieve saturated information transmission and sharing, improve the network's feature utilization, and thus obtain richer and more nuanced feature representations. The main framework includes ordinary convolution and channel-space aggregation operations. Ordinary convolution ensures a consistent number of output channels, while channel-space aggregation integrates channel and spatial attention, enabling the model to learn different features more precisely and enhancing its ability to learn complex features. Channel attention uses max pooling to obtain salient features and average pooling to obtain the overall feature distribution, allowing the model to consider both overall features and salient features. Spatial attention reduces feature loss through a grouped feature reshaping strategy and utilizes a dual-branch parallel strategy to achieve cross-spatial feature interaction, improving the model's ability to acquire spatial locations.

[0112] In some embodiments, the process of performing spatial attention operations on input features is as follows: Figure 9 As shown, after adjusting the matrix size of the input features to obtain the reconstructed features, a two-branch operation is performed on the reconstructed features: In the first branch operation, the reconstructed features are simultaneously subjected to average pooling in the X and Y directions, and then the two pooled features are fused. The fused features are then split into channels to obtain a first width feature and a first height feature; the first width feature and the first height feature are then subjected to sigmoid activation operations, and then fused with the reconstructed features; the fused features are subjected to parallel matrix size adjustment and global average pooling. The features after matrix size adjustment are input into the second branch. The features after global average pooling are subjected to softmax operation and then matrix tensor multiplication (Matmul) with the features from the second branch to obtain the output features of the first branch; in the second branch... In the operation, global average pooling, as well as average pooling in the X and Y directions, are performed on the reconstructed features simultaneously. After the global average pooling features are subjected to a softmax operation, they are multiplied by a matrix tensor with the features from the first branch after matrix size adjustment. The features after matrix tensor multiplication are then combined with the output features of the first branch and the reconstructed features to obtain the output features of the spatial attention operation. The two pooled features after average pooling in the X and Y directions are fused, and the fused features are split into channels to obtain the second width feature and the second height feature. The second width feature and the second height feature are activated by sigmoid operation respectively, and then fused with the reconstructed features. After the fused features are subjected to matrix size adjustment, they are input into the first branch for matrix tensor multiplication.

[0113] In this embodiment of the invention, the input feature Xin Adjusting the matrix size yields the reconstructed feature X. g Then, the reconstructed feature X g Perform a two-branch operation. In the first branch operation, reconstruct feature X... g Simultaneously perform average pooling in the X direction (i.e.) Figure 9 In X-AVG) and average pooling in the Y direction (i.e. Figure 9 In Y-AVG (a pooling algorithm), the features after pooling are fused by channel concatenation and convolution. Then, the fused features are split by channel to obtain the first width feature and the first height feature. The above process can be represented as:

[0114] ;

[0115] in, and P represents the first width feature and the first height feature, respectively. X-AVG and P Y-AVG These represent the average pooling in the X direction and the average pooling in the Y direction, respectively.

[0116] For the first width feature and first height features After performing sigmoid activation operations separately, the data is then compared with the reconstructed feature X. g Perform a fusion operation by multiplying corresponding elements. The above process can be represented by the following formula:

[0117] ;

[0118] Among them, X mix This represents the fused features. After group normalization of the fused features, global average pooling (i.e., ...) is performed. Figure 9 In the XY-AVG model, the features after global average pooling are softmaxed and then multiplied by a matrix tensor with the features from the second branch to obtain the output features of the first branch. The above process can be represented by the following formula:

[0119] ;

[0120] in, P represents matrix tensor multiplication. GAP Indicates global average pooling. This indicates a feature originating from the second branch.

[0121] In the operation of the second branch, the reconstructed feature X gFirst, convolution is performed, then global average pooling, as well as average pooling in the X and Y directions, are applied simultaneously to the convolutional features. The two pooled features resulting from the X and Y direction average pooling operations are then fused through channel concatenation and convolution. Finally, the fused features are split into channels to obtain the second width and second height features. The above process can be represented by the following formula:

[0122] ;

[0123] in, and These represent the second width feature and the second height feature, respectively. Regarding the second width feature... Second height features After performing sigmoid activation operations separately, the data is then compared with the reconstructed feature X. g Element-wise multiplication, followed by matrix resizing of the multiplied features, yields the feature vectors. .

[0124] After performing a softmax operation on the features obtained from global average pooling, a matrix tensor multiplication operation is performed with the features obtained from the matrix size adjustment operation in the first branch. The features obtained from the matrix tensor multiplication are then multiplied with the output features of the first branch. After multiplying corresponding elements, then summing them with the reconstructed feature X g The output features of the spatial attention operation are obtained by adding the corresponding elements.

[0125] In some embodiments, the process of performing channel attention operations on input features is as follows: Figure 10As shown, a dual-branch channel feature extraction is performed on the input features: In the first channel feature extraction branch, max pooling is performed on the input features, and the pooled features are then subjected to two fully connected operations to obtain the first channel features; In the second channel feature extraction branch, average pooling is performed on the input features, and the pooled features are then subjected to two fully connected operations to obtain the second channel features; The first and second channel features are then integrated and fused with the input features to obtain the output features of the channel attention operation. In this embodiment of the invention, the process of performing channel attention operation on the input features specifically includes: performing a dual-branch channel feature extraction on the input features: In the first channel feature extraction branch, max pooling is performed on the input features, and batch normalization is performed on the max-pooled features; the processed features are subjected to two fully connected operations, and a ReLU activation operation is performed between the two fully connected operations to obtain the first channel features. In the second-channel feature extraction branch, the input features are subjected to average pooling, and the pooled features are then batch normalized. The processed features are then subjected to two fully connected operations, with a ReLU activation operation performed between the two fully connected operations to obtain the second-channel features. The corresponding elements of the first-channel and second-channel features are added together, followed by sigmoid activation. Finally, the activated features are multiplied by the corresponding elements of the input features to obtain the output features of the channel attention operation.

[0126] Topological feature interaction pyramid such as Figure 11 As shown, the topological feature interaction pyramid comprises at least two stages, performed sequentially: In the first stage, cross-scale feature interactions are performed on multiple local-global features of different scales to obtain the output features of the first stage, and the number of output features in the first stage is the same as the number of input local-global features; cross-scale feature interactions are performed on the output features of the first stage to obtain the input features of the second stage; in the second to penultimate stages, cross-scale feature interactions are performed on the input features of each stage to obtain the output features of each stage, and the number of output features in each stage is the same as the number of input features in each stage; cross-scale feature interactions are performed on the output features of each stage to obtain the input features of the next stage; in the final stage, a Laplacian transform residual operation is performed on the input features to obtain the fused image. In this embodiment of the invention, the topological feature interaction pyramid comprises at least four stages.

[0127] This invention also provides a multimodal image fusion method based on multi-scale residuals and feature aggregation, such as... Figure 12 As shown, it includes:

[0128] S1: Obtain two image datasets of different modalities and the corresponding fused image dataset; preprocess the three image datasets to obtain the training set. In this embodiment of the invention, various data augmentation methods are applied to the three datasets, specifically including: randomly cropping the original image into a sub-image of size 256×256, randomly scaling the sub-image with a scaling factor of [0.5,2], and brightness perturbation strategies.

[0129] S2: Construct a multimodal image fusion system as provided in this invention.

[0130] S3: Use the training set obtained in step S1 to train the system constructed in step S2 to obtain a multimodal image fusion model.

[0131] In step S3, images of two different modalities in the training set are used as training inputs, and fused images in the training set are used as training outputs. During training, the multimodal image fusion system is trained using result loss and inter-layer loss. Result loss calculates the loss of the fused image obtained by the multimodal image fusion model, and inter-layer loss calculates the loss of the features output by the local-global feature extraction group in the multimodal image fusion model. The total loss is obtained by weighted summation of result loss and inter-layer loss.

[0132] The total loss is:

[0133] L total =L res +μL layers ;

[0134] Among them, L total L represents the total loss. res L represents the loss as a result. layers The interlayer loss is represented by μ, where μ represents the weight of the interlayer loss; the final loss is obtained by the following formula:

[0135] L res =αL align +βL gradient +γL foces +ρL ssim ;

[0136] Among them, L align L represents the basic feature alignment loss. gradient L represents the gradient loss. foces L represents the significant enhancement loss. ssim The loss represents the overall structural similarity loss, and α, β, γ and ρ represent the corresponding weights of the four types of loss.

[0137] In the basic feature alignment loss, the larger pixel value from the two different modalities is taken pixel by pixel, and the fused image is constrained using L1 loss based on the larger pixel value. Specifically, the basic feature alignment loss L... align It can be expressed by the following formula:

[0138] L align =L1(I f ,max(I vi ,I ri ));

[0139] Among them, I f Let L1 represent the fused image, and I represent the L1 loss. vi and I vi These represent two different modalities of the input images. In this embodiment of the invention, I vi Represents a visible light image, I ri Represents an infrared image.

[0140] The gradient loss includes forward gradient loss and backward gradient loss. Specifically, gradients are calculated for the two different modalities of the image and the fused image separately, and the maximum gradient value of the two different modalities is taken. Using the maximum gradient value as the first term and the gradient of the fused image as the second term, the L1 loss is calculated to obtain the forward gradient loss. Similarly, using the gradient of the fused image as the first term and the maximum gradient value as the second term, the L1 loss is calculated to obtain the backward gradient loss. The forward and backward gradient losses are combined with the L1 loss between the gradient of the fused image and the gradient of one of the different modalities to obtain the final gradient loss. In this embodiment of the invention, the gradient loss L... gradient It can be expressed by the following formula:

[0141] ;

[0142] in, Represents visible light image I vi The gradient obtained after Laplace transform, Infrared image I ri The gradient obtained after Laplace transform, Represents the fused image I f The gradient obtained after Laplace transform.

[0143] The significant region enhancement loss is obtained by the following formula:

[0144] ;

[0145] Where I represents one of the two different modalities, E represents the mathematical expectation, and M... ir This represents a luminance region mask. In this embodiment of the invention, the luminance region mask M... irSpecifically, it is an adaptively generated infrared brightness region mask. This represents a negative luminance loss term. Luminance region mask M ir We obtain it from the following formula:

[0146] M ir =Sig((max(I1,I2)-0.5)×10);

[0147] Where Sig represents the sigmoid activation function. This relates to the fusion of visible light image I in this embodiment of the invention. vi and infrared image I ri For image fusion, the salient region enhancement loss can be adjusted accordingly as follows:

[0148] .

[0149] The overall structural similarity loss is expressed by the following formula:

[0150] L ssim =(1-SSIM(max(I f ,I1)))+(1-MS_SSIM(max(I f ,I1)));

[0151] Wherein, SSIM represents structural similarity, and MS_SSIM represents multiple structural similarity, for the fused visible light image I in the embodiments of the present invention. vi and infrared image I ri Image fusion, combined with structural similarity loss L ssim It can be adjusted accordingly:

[0152] L ssim =(1-SSIM(max(I f ,I ri )))+(1-MS_SSIM(max(I f ,I ri ))).

[0153] Interlayer loss is obtained by the following formula:

[0154] ;

[0155] Where S represents the Sobel edge gradient operator, and This represents the corresponding features of two different modalities of images in the i-th layer of the system; Let L2 represent the feature map of the i-th layer in the system, and L2 represent the L2 loss. This indicates that the i-th layer in the system takes the maximum value. For the fused visible light image I in this embodiment of the invention... vi and infrared image I riFor image fusion, the interlayer loss can be adjusted accordingly as follows:

[0156] .

[0157] Furthermore, in the training process provided in this embodiment of the invention, the Adam optimizer is used and a cosine annealing scheduler is employed to optimize the learning rate strategy. In the fusion task, the initial learning rate is set to 1e-3, the weight decay is 1e-4, and the learning rate gradually decays to 1e-6. Throughout the experiment, the batch size is set to 1, and the entire framework is trained for 10 epochs. All experiments are run on an NVIDIA 4090 GPU under the PyTorch framework.

[0158] S4: Input the two different modal images to be fused into the multimodal image fusion model obtained in step S3 for fusion to obtain the corresponding fused image.

[0159] To demonstrate the good image fusion effect of the multimodal image fusion method based on multi-scale residuals and feature aggregation provided in this invention, image fusion experiments were conducted on three different modalities. The experimental results are as follows: Figure 13 As shown, where, Figure 13 (a) shows the fusion of the visible light image and the infrared image. Figure 13 (b) shows the fusion of optical and SAR images. Figure 13 (c) shows the fusion of MRI and PET images. From Figure 13 As can be seen, this invention efficiently utilizes the features of different scales of the image by combining the differential feature enhancer, the local-global feature extractor, and the interlayer-outcome loss, making the fusion effect closer to the salient features of different model images and effectively increasing the information entropy of the fused image.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-modal image fusion system based on multi-scale residual and feature aggregation, characterized in that, Comprise: A double-flow residual feature extraction backbone, which respectively performs multi-scale residual processing on two different modal images inputted, and respectively obtains a plurality of scales of first modal features and second modal features; A differential feature enhancement group, which comprises a same number of differential feature enhancers as the number of scales of the first modal features; In each differential feature enhancer, the first modal feature and the second modal feature of the corresponding scale are respectively subjected to channel and spatial attention operations, and then the processed first modal feature and the processed second modal feature are subjected to residual and fusion operations to obtain an enhanced feature of the current scale; A local-global feature extraction group, which comprises a same number of local-global feature extractors as the number of differential feature enhancers; in each local-global feature extractor, the enhanced feature from the corresponding differential feature enhancer is subjected to global feature extraction and local feature extraction, and then the global feature and the local feature are fused to obtain a local-global feature; A topological feature interaction pyramid, which performs feature interaction of different levels and different stages on a plurality of different scales of local-global features outputted by the local-global feature extraction group to obtain a final fused image; The topological feature interaction pyramid comprises no less than two stages, and the stages are sequentially performed: in the first stage, a plurality of different scales of local-global features are subjected to cross-scale feature interaction to obtain output features of the first stage, and the number of the output features of the first stage is the same as the number of the inputted local-global features; the output features of the first stage are subjected to cross-scale feature interaction to obtain input features of the second stage; In the second stage to the second-to-last stage, the input features of each stage are subjected to cross-scale feature interaction to obtain output features of each stage, and the number of the output features of each stage is the same as the number of the input features of each stage; the output features of each stage are subjected to cross-scale feature interaction to obtain input features of the next stage; In the last stage, the input features are subjected to Laplacian transformation residual operation to obtain a fused image.

2. The multi-scale residual and feature aggregation based multi-modal image fusion system of claim 1, wherein, The double-flow residual feature extraction backbone comprises a first modal residual feature extraction branch and a second modal residual feature extraction branch, wherein: In the first modal residual feature extraction branch, the first modal image is subjected to continuous multi-time residual processing, and each time residual processing obtains a scale of first modal features; In the second modal residual feature extraction branch, the second modal image is subjected to continuous multi-time residual processing, and each time residual processing obtains a scale of second modal features. 3.The multi-modal image fusion system based on multi-scale residual and feature aggregation of claim 1, wherein, In each differential feature enhancer: The first modal feature and the second modal feature of the corresponding scale are respectively subjected to spatial attention operation, and then the features after the spatial attention operation are respectively added to the first modal feature and the second modal feature to correspondingly obtain first modal spatial features and second modal spatial features; The first and second modal spatial features are subjected to a difference operation, and the obtained difference feature is subjected to a channel attention operation; the feature obtained through the channel attention operation is multiplied by the first and second modal spatial features, and then added to the first and second modal features respectively, and the two obtained features are fused to obtain the enhanced feature of the current scale.

4. The multi-scale residual and feature aggregation based multi-modal image fusion system of claim 1, wherein, In each local-global feature extractor: The enhanced feature from the corresponding difference feature enhancer is subjected to an encoding operation, and the encoded feature is subjected to global feature extraction and local feature extraction at the same time, and then the extracted global feature and local feature are fused, and the fused feature is subjected to FFN operation to obtain the local-global feature; In the process of global feature extraction, the input feature is subjected to two-way query feature, key feature and value feature extraction at the same time, and the extracted two query features are fused and subjected to feature fusion attention operation; the feature processed by the feature fusion attention operation is subjected to window attention operation with the two extracted key features and value features respectively, and two preliminary features are obtained; the two preliminary features are fused, and then the fused feature is subjected to feature fusion attention operation and multilayer perception operation in turn to obtain the global feature; In the process of local feature extraction, the input feature is subjected to not less than two times of channel-space aggregation operation to obtain the local feature; In each channel-space aggregation operation, the input feature is subjected to spatial attention operation and channel attention operation at the same time, and the features obtained by the two kinds of attention operations are fused to obtain the output feature.

5. The multi-scale residual and feature aggregation based multi-modal image fusion system of claim 4, wherein, In the process of feature fusion attention operation: The input feature is subjected to grouping operation to obtain a plurality of sub-features; each sub-feature is subjected to channel division, and the divided features are subjected to two-way operation; in each way operation, the feature is subjected to channel attention scaling and parameter offset operation, and then subjected to sigmoid activation operation, and the obtained feature is fused with the input feature of the current way operation to obtain the output of the current way operation; the input of the two-way operation is fused and grouped to obtain the output feature.

6. The multi-scale residual and feature aggregation based multi-modal image fusion system of claim 4, wherein, In the process of spatial attention operation on the input feature: The input feature is subjected to matrix size adjustment to obtain a reconstructed feature, and then subjected to two-branch operation: In the operation of the first branch, the reconstructed feature is subjected to X-direction and Y-direction average pooling at the same time, and then the two pooled features are fused, and the fused feature is subjected to channel splitting to obtain a first width feature and a first height feature; the first width feature and the first height feature are subjected to sigmoid activation operation respectively, and then fused with the reconstructed feature; the fused feature is subjected to parallel matrix size adjustment and global average pooling, the feature after the matrix size adjustment operation is input into the second branch, and the feature after the global average pooling operation is subjected to softmax operation, and then subjected to matrix tensor multiplication operation with the feature from the second branch to obtain the output feature of the first branch; In the operation of the second branch, global average pooling is performed on the reconstructed feature, and average pooling is performed in the X direction and the Y direction; after the softmax operation is performed on the feature after the global average pooling operation, matrix tensor multiplication is performed with the feature after the matrix size adjustment operation in the first branch, and the feature after the matrix tensor multiplication is combined with the output feature of the first branch and the reconstructed feature to obtain the output feature of the spatial attention operation; after the fusion of the two pooled features after the average pooling operation in the X direction and the Y direction, channel splitting is performed on the fused feature to obtain a second width feature and a second height feature; after the sigmoid activation operation is performed on the second width feature and the second height feature respectively, fusion is performed with the reconstructed feature; after the matrix size adjustment operation is performed on the fused feature, the matrix tensor multiplication operation in the first branch is input.

7. The multi-scale residual and feature aggregation based multi-modal image fusion system of claim 4, wherein, In the process of performing the channel attention operation on the input feature, two branch channel feature extraction is performed on the input feature: In the first channel feature extraction branch, maximum pooling is performed on the input feature, and two full connection operations are sequentially performed on the pooled feature to obtain a first channel feature; in the second channel feature extraction branch, average pooling is performed on the input feature, and two full connection operations are sequentially performed on the pooled feature to obtain a second channel feature; the first channel feature and the second channel feature are integrated, and then fused with the input feature to obtain the output feature of the channel attention operation.

8. A multi-modal image fusion method based on multi-scale residual and feature aggregation, characterized in that, Comprise: S1: obtaining two different modal image data sets and corresponding fusion image data sets; preprocessing three image data sets to obtain a training set; S2: constructing a multi-modal image fusion system as claimed in any one of claims 1-7; S3: training the system constructed in step S2 using the training set obtained in step S1 to obtain a multi-modal image fusion model, wherein: the two different modal images in the training set are used as the input of the training, and the fusion image in the training set is used as the output of the training; during the training process, the multi-modal image fusion system is trained using a result loss and an inter-layer loss; the result loss calculates the loss of the fusion image obtained by the multi-modal image fusion model, and the inter-layer loss calculates the loss of the feature output by the local-global feature extraction group in the multi-modal image fusion model; the result loss and the inter-layer loss are weighted and summed to obtain a total loss; S4: inputting the two different modal images to be fused into the multi-modal image fusion model obtained in step S3 for fusion to obtain the corresponding fusion image.

9. The multi-modal image fusion method based on multi-scale residual and feature aggregation according to claim 8, characterized in that, In step S3, the total loss is: L total =L res +μL layers ; where L total represents the total loss, L res represents the result loss, L layers represents the inter-layer loss, and μ represents the inter-layer loss weight; The result loss is obtained by the following formula: L res = αL align + βL gradient + γL foces + ρL ssim ; wherein, L align represents the base feature alignment loss, L gradient represents the gradient loss, L foces represents the salient region enhancement loss, L ssim represents the comprehensive structural similarity loss, and a, b, g and p represent the corresponding weights of the four losses. In the basic feature alignment loss, the larger pixel value in the two different modal images is taken pixel by pixel, and the fusion image is constrained based on the larger pixel value using the L1 loss; The gradient loss comprises a forward gradient loss and a reverse gradient loss; wherein, the gradients of the images of the two different modalities and the fusion image are calculated respectively, and the maximum value of the gradients of the images of the two different modalities is calculated; the L1 loss is calculated by taking the maximum value of the gradients as a first item and the gradient of the fusion image as a second item to obtain the forward gradient loss, and the L1 loss is calculated by taking the gradient of the fusion image as a first item and the maximum value of the gradients as a second item to obtain the reverse gradient loss; the gradient loss is obtained by combining the forward gradient loss and the reverse gradient loss, and the L1 loss between the gradient of the fusion image and the gradient of one of the images of the two different modalities; The salient region enhancement loss is obtained by the following formula: ； where I f denotes the fused image, I denotes one of the two different modality images, E denotes the mathematical expectation, M ir denotes the luminance region mask, obtained from the following equation: M ir = Sig((max(I1, I2) - 0.5) x 10); Wherein, Sig represents a sigmoid activation function; The inter-layer loss is obtained by the following formula: ； where S denotes a Sobel edge gradient operator, and denotes the corresponding feature of the image of the two different modalities at the i-th layer in the system; denotes the feature map of the i-th layer in the system, L1 denotes the L1 loss, and L2 denotes the L2 loss, denotes the maximum value of the i-th layer in the system.

Images