Image fusion method based on iterable double-branch attention and modal difference guidance

Abstract

The invention discloses an image fusion method based on iterative double-branch attention and modal difference guidance, and belongs to the technical field of computer image vision. In a feature extraction stage, a CNN-based shallow feature extraction module is used to obtain shallow features, and the output of the features has two branches: one branch continues to be input to a Transform-based depth feature extraction module to extract depth features; the other branch is input to the fusion module to realize effective utilization of multi-stage features; in the feature fusion stage, according to the deep and shallow features obtained in the feature extraction stage, firstly, an attention module capable of iterating double branches is utilized in the fusion stage to capture detail information of the extracted shallow and deep features of the infrared and visible light images, and more important feature information is reserved; and then, the reserved important detail information is input into a fusion module based on modal difference guidance of Transform, and the fusion module can reserve difference information of different modals.

Description

Technical Field

[0001] The present invention relates to the field of computer image vision technology, and in particular to an image fusion method based on iterative dual-branch attention and modality difference guidance. Background Art

[0002] With the rapid development of sensor technology, researchers have begun experimenting with using multiple sensors to obtain different images of the same object or scene. These images vary in content, resolution, and spectral characteristics, but each contains valuable information. To fully leverage this information and improve image quality and usability, researchers have begun researching image fusion. By fusing multi-source image information, image fusion can generate a single image with richer information and higher quality. These images offer enhanced visual quality and information content, providing powerful support for a wide range of applications. For example, in medical imaging, image fusion can improve diagnostic accuracy; in remote sensing, it can enhance the ability to identify surface cover; and in computer vision, image fusion can be used for tasks such as target detection, tracking, and recognition.

[0003] Image fusion can be divided into traditional fusion methods and deep learning-based fusion methods. Among traditional fusion methods, methods based on multi-scale decomposition are more popular. However, most traditional fusion methods use manually defined fusion strategies. Such fusion rules lack the ability to adaptively fuse features, resulting in information loss. Convolutional neural networks (CNNs) dominate deep learning image fusion methods. However, if the core of the fusion strategy is limited to convolution operations, the fusion process will be limited to the local receptive field, making it difficult to establish long-term dependence on global context information. In recent years, with the development of Transformer technology in the field of computer vision, Transformer technology has also been applied to image fusion methods. However, these methods focus on preserving global context information and long-term dependencies, but do not fully consider the differential information between different modalities, making it difficult to achieve information complementarity.

[0004] The above problems need to be solved urgently. To this end, the present invention proposes an image fusion method based on iterative dual-branch attention and modality difference guidance. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to solve the problem that the manually defined fusion strategy used in traditional fusion methods lacks the ability of adaptive feature fusion, and the deep learning method focuses on local and global feature extraction and utilization, does not fully consider the difference information of different modalities, and is difficult to achieve information complementarity. An image fusion method based on iterative dual-branch attention and modality difference guidance is provided.

[0006] The present invention solves the above technical problems through the following technical solutions, which include the following steps:

[0007] S1: Data preprocessing

[0008] Acquire an image data set, and perform normalization processing on the infrared image and the corresponding visible light image in the image data set;

[0009] S2: Get image samples

[0010] The preprocessed image data is segmented into image blocks using a segmentation function, and then the contrast function is used to check the contrast of each image block. If the contrast is not lower than the threshold, it will be retained, and finally infrared and visible light image samples are obtained;

[0011] S3: Feature Extraction

[0012] Select an end-to-end fusion model, set the model parameters, and then set two paths for the infrared and visible light image samples to input into the same feature extraction module to capture the shallow and deep features of the image samples;

[0013] S4: Image Fusion

[0014] The shallow and deep features obtained in the feature extraction stage are input into the attention mechanism module based on iterative dual branches for processing. Then, the different modal difference information is fused through the modal difference guided fusion module based on Transformer. Finally, the fused features are input into the feature reconstruction module based on reversible neural network to obtain the final fused image.

[0015] Furthermore, in step S1, the infrared image and the corresponding visible light image are not labeled, and the preprocessing process is as follows:

[0016] S11: Extract infrared and visible light image data, read the pixel value of each pixel in the image, and convert the pixel value into float32 format;

[0017] S12: Divide each pixel value by 255, so that the pixel value is linearly scaled from the range of 0-255 to the range of 0-1, thereby obtaining normalized image data.

[0018] Furthermore, in step S2, the specific processing process is as follows:

[0019] S21: Obtain normalized image data and input it into the segmentation function Im2Path() for block processing;

[0020] S22: Then use the contrast function to check whether the contrast of each image block is lower than the threshold. If the contrast of the infrared and visible light image blocks is not lower than the threshold, they will be retained, and then the infrared and visible light image samples x will be obtained. * and y * , the contrast function is as follows:

[0021]

[0022] Where I represents the input image, threshold is the threshold of the contrast ratio, percentile is the function for calculating the contrast ratio, and p upper Indicates the upper percentile used when calculating contrast, p lower Indicates the lower percentile to use when calculating contrast.

[0023] Furthermore, in step S3, the specific processing process is as follows:

[0024] S31: Get infrared and visible light image samples x * and y * , the infrared and visible light image samples x * and y * There are two paths: first input into the feature extraction module based on CNN shallow feature extraction module, and obtain the corresponding shallow feature x f1 、y f1 、x f2 、y f2 ;

[0025] S32: Then the shallow feature x is obtained f2 、y f2 They are respectively input into the deep feature extraction module based on Transformer in the feature extraction module to obtain the corresponding deep features x f3 and y f3 .

[0026] Furthermore, in the step S31, the number of shallow feature extraction modules based on CNN is two, which are set in sequence. The first shallow feature extraction module based on CNN includes two convolutional layers, and each convolutional layer is processed by a BN layer and a FReLU activation function in sequence to obtain the shallow feature x f1 and y f1 The second CNN-based shallow feature extraction module also includes two convolutional layers. A BN layer and a FReLU activation function are used after the first convolutional layer. In the second convolutional layer, no BN layer and FReLU activation function are used to process the shallow feature x. f2 and y f2 ;

[0027] In the step S32, the number of deep feature extraction modules based on Transformer is two, and the structures of the two modules are the same and are arranged in parallel. The deep feature extraction module based on Transformer includes an MDTA module and a CDFN module arranged in sequence. The MDTA module uses three convolutional layers and a depth-separable convolutional layer to generate Q, K, and V for the input features, and then multiplies Q and K, and then uses a softmax activation function to generate the weight of each channel, and then multiplies it with V, and finally uses a convolutional layer and adds the input features. The CDFN module uses two convolutional layers and depth-separable convolution, one of which passes through the depth-separable convolution path first uses GELU non-linearity activation, and then multiplies it with the generated result of the other path, and finally passes through a convolutional layer to add the generated result and the input feature to obtain the final deep feature, that is, the deep feature x f3 and y f3 .

[0028] Furthermore, in step S4, the specific processing process is as follows:

[0029] S41: Obtain the shallow features x captured in the feature extraction stage f1 、y f1 、x f2 、y f2 and deep features x f3 、y f3 ;

[0030] S42: x f1 and y f1 Input to the first iterable two-branch attention mechanism module to obtain the feature x f1_1 and y f1 _1 , shallow feature x f2 and y f2 Input to the second iterable two-branch attention mechanism module to obtain the feature x f2_2 and y f2_2 , deep features x f3 and y f3 Input to the third iterable two-branch attention mechanism module to obtain the feature x f3_3 and y f3_3 , and then use concatenate to concatenate features x f1_1 and y f2_2 、y f1_1 and x f2_2 , and get the features x_y respectively f1 and x_y f2, and then input them into the first and second parts of the Transformer-based modality difference guided fusion module to obtain the preliminary fusion feature x_y pf_1 and x_y pf_2 ;

[0031] S43: Output features x_y of the first and second parts of the fusion module guided by the modality difference of Transformer pf_1 and x_y pf_2 The features x extracted by the third iterative two-branch attention mechanism module are respectively f3_3 and y f3_3 Use concatenate to get x_y f1_3 and x_y f2_3 , input into the third part of the Transformer-based modality difference guided fusion module to obtain the final fusion feature x_y f_4 ;

[0032] S44: Using fusion feature x_y f_4 , use the feature reconstruction module based on reversible neural network to fusion feature x_y f_4 Perform feature reconstruction and finally obtain the fused image image fusion .

[0033] Furthermore, in the step S42, the iterative dual-branch attention mechanism module includes two parts set in sequence. The first part first uses two point-by-point convolution layers for processing in sequence, wherein a Rule activation function is used after the first point-by-point convolution layer; after the two point-by-point convolution layers, two paths are used to process the generated features. The first path first uses the GAP module, then uses two point-by-point convolution layers, and finally uses the sigmoid activation function to generate the weight of each channel, and multiplies the obtained weight by the feature of the first pass through the two point-by-point convolution layers; the second path directly uses two point-by-point convolution layers, and then uses the sigmoid activation function to generate the weight of each space, and multiplies the obtained weight by the feature of the first pass through the two point-by-point convolution layers; finally, the features obtained by the two paths are processed. The features are added and then added to the initial infrared and visible light features respectively; the features of the two paths are then concatenated and input into the second part, which is first processed by a point-by-point convolution layer and then divided into two paths again. The first path uses the GAP module and then two point-by-point convolution layers for processing. A Rule activation function is used after the first point-by-point convolution layer; the second path directly uses two point-by-point convolution layers and a Rule activation function is used after the first point-by-point convolution layer; the results of the two paths are then added and finally input into the sigmoid activation function. The sigmoid generates a weight ω and multiplies it with the initial visible light feature, and at the same time generates a weight (1-ω) and multiplies it with the initial infrared feature to obtain the output result of the iterative dual-branch attention mechanism module;

[0034] In step S43, the Transformer-based modality difference guided fusion module includes three parts. The first part has the same structure as the second part and is set in sequence. In the first part, a LN layer and a SA layer are first used to obtain the processed features, and the features are added to the initial input features to obtain the first output features. Then, the features are input to a LN layer and a MLP layer in sequence, and the obtained features are added to the first output features to obtain the preliminary fusion features x_y of the first part. pf_1 In the second part, we also use an LN layer and an SA layer to get the processed features, add the features to the initial input features to get the first output features, and then input them into an LN layer and an MLP layer in sequence. The features obtained are added to the first output features to get the preliminary fusion features x_y of the second part. pf_2 ; The preliminary fusion feature x_y of the first part pf_1 and the output feature x_y of one branch in the third iterable two-branch attention module f1_3 Use the concatenate operation to process the preliminary fusion feature x_y of the second part pf_2And the output feature x_y of the other branch of the third iterable two-branch attention module f2_3 Use the concatenate operation for processing, and then input the two parts of features obtained by the concatenate operation into the third part of the fusion module guided by the modal difference based on the Transformer to obtain the final fusion feature x_y f_4 In the third part, the input features are first processed by a point-by-point convolution layer and a depth-wise separable convolution layer, and then divided into three parts: Q, K, and V according to the number of channels. Q and K of different modes are multiplied, and then processed by the Softmax activation function and multiplied with V of the current mode. The multiplied features are then subtracted from V and added to the original input features before concatenation to obtain the final fusion feature x_y f_4 ;

[0035] In the feature reconstruction module based on the reversible neural network of step S44, four INN layers are sequentially arranged. In the first INN layer, the feature x_y is fused. f_4 First, it is split into two branches, each branch has C / 2 channels, and then the elements are added and multiplied to get the output x_y of the first INN layer. INN_1 , the output x_y INN_1 Input to the second INN layer to get x_y INN_2 , change x_y INN_2 Input to the third INN layer to get x_y INN_3 , change x_y INN_3 Input to the fourth INN layer to get x_y INN_4 , and finally x_y INN_4 After a convolution layer, the final fused image is obtained fusion .

[0036] Furthermore, in the INN layer, the input features are first split into two branches: the first branch has the feature F inn_1 , the second branch is characterized by F inn_2 ; The feature F of the first branch inn_1 First, it is processed through a BRB block to obtain F b_1 , followed by the feature F of the second branch inn_2 Add up to get F i_b_1 , and then add the feature F i_b_1 Then, after a BRB block processing, we get F b_2 , then with the feature F of the first branch inn_1 Multiply to get F i_b_2 , then the added feature F i_b_1 Then go through a BRB block to get Fb_3 The feature F after multiplication i_b_2 Add them together to get F i_b_3 , and finally F i_b_3 and F i _b_1 Add to get the feature F inn_finall , where, in the BRB block, a convolutional layer and Rule6 activation function are first used, followed by depth-wise separable convolution and convolution layers, and finally another convolutional layer is used, and the obtained features are added to the original input to obtain the final output.

[0037] Furthermore, the image fusion method based on iterative dual-branch attention and modality difference guidance further includes the following steps:

[0038] S5: Model parameter optimization

[0039] The overall loss function is used to calculate the loss of the fused image and the source image. The gradient is calculated based on the loss value and the back-propagation algorithm, and the model parameters are adjusted through the Adam optimizer. The image sample is input again and iterative training is repeated until the iteration threshold is reached.

[0040] Furthermore, in step S5, the overall loss function Loss total as follows:

[0041] Loss total =αLoss in +βLoss grad +γLoss msssim

[0042] Among them, Loss in Loss is the strength loss. grad is the gradient loss, Loss msssim is the MSSSIM loss, α, β, γ are the weight parameters of the corresponding loss;

[0043] Strength loss in The calculation formula is as follows:

[0044]

[0045] Where H represents the height of the image, W represents the width of the image; I f Represents the obtained fused image, I vi and I ir Represent the input visible light image and infrared image respectively, ||·||1 represents L1 regularization, and max represents the maximum selection of elements;

[0046] Gradient Loss grad The calculation formula is as follows:

[0047]

[0048] in, represents the sobel operator, |·| represents the absolute value operation;

[0049] MSSSIM Loss msssim The calculation formula is as follows:

[0050] Loss msssim =β1(1-MSSSIM(I f ,I vi ))+β2(1-MSSSIM(I f ,I ir ))

[0051] Among them, β1 and β2 are the weights controlling the two losses respectively, and MSSSIM() represents the calculation of multi-scale structure loss.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] 1. In the feature extraction stage, the present invention improves the existing fusion method that can only utilize the final extracted features. Instead, it performs two-stage feature extraction. First, a shallow feature extraction module based on CNN is used to obtain shallow features. The output of this feature has two branches: one branch continues to input the deep feature extraction module based on Transformer to extract deep features; the other branch is input to the fusion module, realizing the effective utilization of multi-stage features.

[0054] 2. In the feature fusion stage, the present invention first utilizes an iterative two-branch attention module to capture detailed information from the shallow and deep features of the extracted infrared and visible light images based on the deep and shallow features obtained in the feature extraction stage, thereby retaining more important feature information. This retained important detail information is then input into a Transformer-based modality difference-guided fusion module, which is capable of preserving differential information across modalities. Combining CNN and Transformer in the above fusion stage avoids information loss caused by manually designed fusion strategies while also taking into account information complementarity across modalities.

[0055] 3. In the feature reconstruction module of the present invention, by introducing the INN layer to perform image reconstruction, the integrity of the feature information can be effectively maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 1 is a flow chart of an image fusion method based on iterative dual-branch attention and modality difference guidance in the first embodiment of the present invention;

[0057] Figure 2 1 is a schematic diagram of an implementation flow of an image fusion method based on iterative dual-branch attention and modality difference guidance in Example 1 of the present invention;

[0058] Figure 3 Schematic diagram of the overall structure of the end-to-end fusion model of visible light and infrared images in the first embodiment of the present invention;

[0059] Figure 4 Schematic diagram of the structure of the shallow feature extraction module based on CNN in the first embodiment of the present invention;

[0060] Figure 5 Schematic diagram of the structure of the Transformer-based deep feature extraction module in the first embodiment of the present invention;

[0061] Figure 6 Schematic diagram of the structure of the iterative dual-branch attention mechanism module in the first embodiment of the present invention;

[0062] Figure 7 1 is a schematic structural diagram of the first and second parts of the Transformer-based modality difference guided fusion module in Example 1 of the present invention;

[0063] Figure 8 2 is a schematic structural diagram of the third part of the Transformer-based modality difference guided fusion module in the first embodiment of the present invention;

[0064] Figure 9 3 is a schematic diagram of the structure of the feature reconstruction module based on the reversible neural network in the first embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0066] Example 1

[0067] like Figure 1 、 2 As shown, this embodiment provides a technical solution: an image fusion method based on iterative dual-branch attention and modality difference guidance, comprising the following steps:

[0068] Step 1: Obtain an image dataset and use normalization to normalize the image data to a fixed range. The image dataset includes infrared and visible light images, both of which are unlabeled.

[0069] As an example, infrared and visible light images of the MSRS (Multi-Spectral Road Scenarios Dataset) are extracted from the image database. The infrared image itself is a grayscale image, so it is only necessary to grayscale the visible light image, read the pixel value of each pixel in the image, and convert the pixel value to float32; then divide each pixel value by 255. This can linearly scale the pixel value from the range of 0-255 to the range of 0-1, ensuring that all pixel values are within a uniform range.

[0070] Step 2: Use the segmentation function to divide the normalized image data into image blocks, and then use the contrast function to check the contrast of each image block. If the contrast is not lower than the threshold, it will be retained, and finally the samples for training are obtained.

[0071] As an example, extract the normalized grayscale images x and y of the infrared and visible light images of the pedestrian scene from MSRS, use the segmentation function Im2Path() to split x and y into image blocks, and randomly select two image blocks x ′ and y ′ , x ′ and y ′ Input to the contrast function and Where threshold = 0.1, l = 10, u = 90, the 10th percentile and 90th percentile values of the image block can be calculated by the contrast function, and then the two image blocks x are obtained based on the ratio of the difference between these two values and the 90th percentile value. ‘ and y ′ The contrast value, if the value is less than 0.1, the image block is discarded, otherwise the image block is retained, and finally the sample x that can be trained is obtained * and y * .

[0072] It should be noted that in Im2Path(), the new width, height and total number of blocks after segmentation are calculated based on the set window size and step size.

[0073] Step 3: Select an end-to-end fusion model. First, set the model parameters. Then, set two paths for the infrared and visible light image samples and input them into the same feature extraction module to capture the shallow and deep features of the image samples.

[0074] like Figure 3As shown, in this embodiment, the end-to-end image fusion model consists of three parts: feature extraction, feature fusion, and feature reconstruction modules. The feature extraction module first extracts features from the input visible light and infrared image pairs. The feature fusion module then fuses the extracted features. The fused features are then input into the feature reconstruction module for image reconstruction, ultimately yielding a fused image. This end-to-end fusion model eliminates the need for manual design of feature extraction and fusion strategies. It automatically learns multi-level features and fuses image pairs in real time, providing improved adaptability to the quality of the input images.

[0075] As an example, obtain infrared and visible light image samples x * and y * , x * and y * There are two paths, first input to the first CNN-based shallow feature extraction module (such as Figure 4 As shown), after two convolutional layers (the sizes of the two convolution kernels are 7 and 3 respectively), each convolutional layer is followed by a BN layer and a FReLU activation function. f1 and y f1 Input to the second shallow feature extraction module, which still uses two convolution layers (the size of both convolution kernels is 3), and the last convolution layer has no BN and FReLU to obtain the final shallow feature x f2 and y f2 .

[0076] Then the shallow feature x f2 and y f2 They are respectively input into the deep feature extraction module based on Transformer. The Transformer of this deep feature extraction module is applied across channels rather than spatial dimensions to generate channel attention weights, through which long-range dependencies can be established on contextual information to obtain the deep features x of the image. f3 and y f3 .

[0077] It should be noted that, in this embodiment, Figure 5As shown in the figure, the number of deep feature extraction modules based on Transformer is two. The structures of the two modules are the same and are set in parallel. The deep feature extraction module based on Transformer includes a multi-Dconv head transposed attention (MDTA) and a gated-Dconv feed-forward network (GDFN). MDTA is a self-attention mechanism applied across channels. It uses three convolutional layers and depth-separable convolutional layers to generate Q (query vector), K (key vector), and V (value vector) for the input features, and then multiplies Q and K. The softmax activation function is used to generate the weight of each channel, which is then multiplied by V. Finally, a convolution is used to add the input features. CDFN uses two convolutional layers and depth-separable convolutions. One of the paths through the depth-separable convolution is first activated with GELU non-linearity, and then multiplied with the generated result of the other path. Finally, it passes through a convolutional layer to add the generated result and the input feature to obtain the final deep feature, that is, the deep feature x f3 and y f3 .

[0078] Step 4: Obtain the shallow features x captured in the feature extraction stage f1 、y f1 、x f2 、y f2 and deep features x f3 、y f3 , in the fusion stage, three iterative dual-branch attention mechanism modules are used to convert x f1 and y f1 Input to the first iterable two-branch attention mechanism module to get x f1_1 and y f1_1 , x f2 and y f2 Input to the second iterable two-branch attention mechanism module to get x f2_2 and y f2_2 , x f3 and y f3 Input to the third iterable dual-branch attention mechanism module to get x f3_3 and y f3_3 After retaining important detail features, use concatenate to concatenate x f1_1 and y f2_2 、y f1_1 and x f2_2 , and we get x_y respectively f1 and x_y f2, and then input them into the first and second parts of the Transformer-based modality difference guided fusion module to obtain the preliminary fusion feature x_y pf_1 and x_y pf_2 , the first and second parts of the Transformer-based modality difference guided fusion module on the input feature x_y f1 and x_y f2 It can model global context information and avoid the defect that shallow features are limited to local receptive fields.

[0079] For the first and second parts of the Transformer-based modality difference guided fusion module, the preliminary fusion feature x_y is obtained pf_1 and x_y pf_2 x extracted by the third iterative two-branch attention mechanism module f3_3 and y f3_3 Use concatenate to get x_y f1_3 and x_y f2_3 , input to the third part of the Transformer-based modality difference guided fusion module to obtain the final fusion feature x_y f_4 .

[0080] Using the fused feature x_y f_4 , using a feature reconstruction module based on a reversible neural network (such as Figure 9 As shown) for the fusion feature x_y f_4 Perform feature reconstruction. Four INN layers are used in the feature reconstruction stage. f_4 First, it is split into two branches, each branch has C / 2 channels, and then the elements are added and multiplied to get the output x_y of the first INN layer. INN_1 , the output x_y INN_1 Input to the second INN layer to get x_y INN_2 , change x_y INN_2 Input to the third INN layer to get x_y INN_3 , change x_y INN_3 Input to the fourth INN layer to get x_y INN_4 , and finally x_y INN_4 After a convolution layer, the final fused image is obtained fusion .

[0081] In this embodiment, if Figure 6As shown, the iterable dual-branch attention mechanism module consists of two parts, which are set up in sequence. The first part first uses two point-by-point convolution layers in sequence (a BN is used after each convolution layer, the same below). There is a Rule activation function after the first point-by-point convolution layer. After the two point-by-point convolution layers, two paths are used to process the generated features. The first path first uses the global average pooling module (Global Average Pooling, GAP), then uses two point-by-point convolution layers, and finally uses the sigmoid activation function to generate the weight of each channel, and multiplies the obtained weight with the feature of the first pass through the two point-by-point convolution layers; the second path is to directly use two point-by-point convolution layers, and then use the sigmoid activation function to generate the weight of each space, and the obtained weight is still multiplied with the feature of the first pass through the two point-by-point convolution layers; finally, the features obtained by the two paths are added together and then added to the initial infrared and visible light features respectively. The formula is as follows:

[0082] F add =PB(PBR(F vi +F ir )

[0083] ω c =δ(PB(PBR(GAP(F add ))))

[0084] ω s =δ(PB(PBR(F add )))

[0085] F ir =F ir +((ω c *F add )+(ω s *F add ))

[0086] F vi =F vi +((ω c *F add )+(ω s *F add ))

[0087] Then the features of the two paths are concatenated and input into the second part. The second part first uses a point-by-point convolution layer (without BN), and then it is still divided into two paths. The first path uses the GAP module and then two point-by-point convolution layers. There is a Rule activation function after the first point-by-point convolution layer; the second path directly uses two point-by-point convolution layers. There is a Rule activation function after the first convolution layer. The results of the two paths are then added and finally input into the sigmoid activation function. The sigmoid generates a weight ω and multiplies it by the initial visible light feature. At the same time, it generates a weight (1-ω) and multiplies it by the initial infrared feature to obtain the output result of the iterative dual-branch attention mechanism module. The formula is as follows:

[0088] F cat_c =PB(PBR(GAP(PB(CAT(F ir ,F vi )))))

[0089] F cat_s =PB(PBR(PB(CAT(F ir ,F vi ))))

[0090] ω=δ(F cat_c +F cat_s )

[0091] F ir =F ir *(1-ω)

[0092] F vi =F vi *ω

[0093] Among them, F vi Represents the input visible light feature, F ir Represents the infrared features of the input, PB represents the convolution layer and BN, PBR adds the Rule activation function, CAP represents the global average pooling, and δ represents the sigmoid activation function.

[0094] In this embodiment, if Figure 7 、 8 As shown in Figure 1, the Transformer-based modality difference guided fusion module consists of three parts. The first two parts have the same structure, represented by TRM, and are set in sequence. In the first and second parts, an LN layer and a SA layer are first used to obtain the processed features, and the features are added to the initial input features to obtain the first output features. Then, an LN layer and an MLP layer are input in sequence, and the obtained features are added to the first output features to obtain the preliminary fusion features of the first two parts. The formula is as follows:

[0095]

[0096] in, Represents the output of the nth iterable two-branch attention mechanism module, CAT is to concatenate different features.

[0097] In the third part, the input features first pass through a point-by-point convolution layer and a depth-wise separable convolution layer, and are divided into three parts Q, K, and V according to the number of channels. Q and K of different modalities are multiplied, and then processed by the Softmax activation function and multiplied with V of the current modality. The multiplied features are then subtracted from V and added to the original input features. Finally, the processed infrared and visible light features are spliced together to obtain the final fusion feature x_y f_4 , the formula is as follows:

[0098]

[0099] At vi =Softmax(γ vi (K vi *Q vi ))

[0100] At ir =Softmax(γ ir (K ir *Q ir ))

[0101]

[0102] Among them, DPw includes convolutional layer and depth-separable convolutional layer. is the chunking function, γ vi and γ ir is a learnable parameter, Indicates the TRM of the nth section.

[0103] In this embodiment, the feature reconstruction process is implemented by the INN layer. In the INN layer, the fused features are first split into two branches, and the features of the first branch are first processed by a Bottleneck residual block (BRB), and then added to the features of the second branch. The added features are then multiplied with the first branch through a BRB, and then the added features are added to the multiplied features through a BRB. Finally, the features of the two branches are added. One of the BRBs first uses a convolution layer and a Rule6 activation function, then uses a depth-separable convolution and a convolution layer, and finally uses a convolution layer. The obtained features are added to the original input to obtain the final output. The formula is as follows:

[0104] F f,k+1 [c+1:C]=I(F f,k [1:c])+F f,k [c+1:C]

[0105] F f,k+1 [1:c]=F f,k [1:c]*(I(F f,k+1 [c+1:C]))+I(F f,k+1 [c+1:C])

[0106] F f,k+1 ={F f,k+1 [1:c]+F f,k+1 [c+1:C]}

[0107] Where k represents the kth reversible layer, F f,k [1:c] and F f,k [c+1:C] represents the first c channels and the channels from c+1 to C, and I represents BRB.

[0108] Step 5: Use intensity, gradient loss, and MSSSIM loss to calculate the loss of the fused image and the source image, adjust the model parameters according to the loss value and the back propagation algorithm, and repeat the iterative training with the updated fused image until the iteration threshold is reached.

[0109] As an example, the fused image image fusion And the source image x is input into the function of calculating the loss value, through Calculate the source image and the fused image image fusion The strength loss value, Calculate the source image and the fused image image fusion Gradient loss value, Loss msssim =β1(1-MSSSIM(I f ,I vi ))+β2(1-MSSSIM(I f ,I ir ))Calculate the source image and the fused image image fusion The MSSSIM loss value is finally passed through αLoss in +βLoss grad +γLoss msssim Calculate the sum of the loss values Loss total , where α, β, and γ are 1, 10, and 1 respectively. Then the image is obtained by back propagation algorithm fusion The Adam optimization algorithm is then used to update the model parameters.

[0110] In this embodiment, the maximum number of iterations is set to 20. In each iteration, the training sample is input into the fusion model for forward calculation, and the gradient is back-propagated according to the loss function to update the model parameters. This process is repeated until 20 iterations are completed.

[0111] The working principle of the present invention is as follows:

[0112] Obtain an image dataset and use normalization to normalize the image data to a fixed range; use a segmentation function to divide the normalized image data into smaller image blocks, then use a contrast function to check the contrast of each image block. If the contrast is not lower than the threshold, it is retained, and finally a sample for training is obtained; after obtaining the image samples, first set the model parameters, then input the infrared and visible light images into the shallow and deep feature extraction modules based on CNN and Transformer along two paths respectively, and then input them into the attention module based on iterative dual branches to retain important detail information of different modalities. The extracted important detail information is then input into the modality difference guided feature fusion module based on Transformer to fuse the difference information of different modalities, realize the complementarity of different modal information, and obtain the final fusion feature; The fused feature is input into the feature reconstruction module based on the reversible neural network to finally generate a fused image; The gradient, intensity and MSSSIM loss of the generated fused image are calculated, and the gradient of these losses is calculated by the backpropagation algorithm. Then, the Adam optimizer is used for optimization, and the model parameters are updated. The image samples are repeatedly input for training until the iteration threshold is reached.

[0113] Example 2

[0114] In order to further illustrate the fusion performance of the fusion model in Example 1, this example selects MSRS, TNO and M 3 DF three public datasets for experiments, and use existing methods for comparative experiments, select EN, SF, AG, SD, SCD and VIF and other commonly used image fusion indicators as the evaluation indicators of the experiment (the larger the indicator value, the better). Specifically, the training set of MSRS is selected to train the model, and 361 images in the test set are used for evaluation, and TNO and M are used respectively. 3 DF test set of 14 images were evaluated. Specific experimental results are shown in Tables 1 to 3, where “ours” refers to the fusion model proposed in Example 1 of the present invention.

[0115] Table 1 Experimental results of EN, SF, AG, SD, SCD and VIF of each model on the MSRS test set

[0116]

[0117]

[0118] Table 2 Experimental results of EN, SF, AG, SD, SCD and VIF of each model on the TNO test set

[0119]

[0120] Table 3 Models in M 3 Experimental results of EN, SF, AG, SD, SCD and VIF on the DF test set

[0121]

[0122]

[0123] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. Image fusion method based on iterative dual-branch attention and modality difference guidance, characterized by: The following steps are involved: S1: Data preprocessing Acquire an image data set, and perform normalization processing on the infrared image and the corresponding visible light image in the image data set; S2: Get image samples The preprocessed image data is segmented into image blocks using a segmentation function, and then the contrast function is used to check the contrast of each image block. If the contrast is not lower than the threshold, it will be retained, and finally infrared and visible light image samples are obtained; S3: Feature Extraction Select an end-to-end fusion model, set the model parameters, and then set two paths for the infrared and visible light image samples to input into the same feature extraction module to capture the shallow and deep features of the image samples; S4: Image Fusion The shallow and deep features obtained in the feature extraction stage are input into the attention mechanism module based on iterative dual branches for processing. Then, the different modal difference information is fused through the modal difference guided fusion module based on Transformer. Finally, the fused features are input into the feature reconstruction module based on reversible neural network to obtain the final fused image.

2. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 1, characterized in that In step S1, the infrared image and the corresponding visible light image are not labeled. The preprocessing process is as follows: S11: Extract infrared and visible light image data, read the pixel value of each pixel in the image, and convert the pixel value into float32 format; S12: Divide each pixel value by 255, so that the pixel value is linearly scaled from the range of 0-255 to the range of 0-1, thereby obtaining normalized image data.

3. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 1, characterized in that In step S2, the specific processing process is as follows: S21: Obtain normalized image data and input it into the segmentation function Im2Path() for block processing; S22: Then use the contrast function to check whether the contrast of each image block is lower than the threshold. If the contrast of the infrared and visible light image blocks is not lower than the threshold, they will be retained, and then the infrared and visible light image samples x will be obtained. * and y * , the contrast function is as follows: Where I represents the input image, threshold is the threshold of the contrast ratio, percentile is the function for calculating the contrast ratio, and p upper Indicates the upper percentile used when calculating contrast, p lower Indicates the lower percentile to use when calculating contrast.

4. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 1, characterized in that In step S3, the specific processing process is as follows: S31: Get infrared and visible light image samples x * and y * , the infrared and visible light image samples x * and y * There are two paths: first input into the feature extraction module based on CNN shallow feature extraction module, and obtain the corresponding shallow feature x f1 、y f1 、x f2 、y f2 ; S32: Then the shallow feature x is obtained f2 、y f2 They are respectively input into the deep feature extraction module based on Transformer in the feature extraction module to obtain the corresponding deep features x f3 and y f3 .

5. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 4, characterized in that In step S31, the number of shallow feature extraction modules based on CNN is two, which are set in sequence. The first shallow feature extraction module based on CNN includes two convolutional layers, and each convolutional layer is processed by a BN layer and a FReLU activation function in sequence to obtain the shallow feature x f1 and y f1 The second CNN-based shallow feature extraction module also includes two convolutional layers. A BN layer and a FReLU activation function are used after the first convolutional layer. In the second convolutional layer, no BN layer and FReLU activation function are used to process the shallow feature x. f2 and y f2 ; In the step S32, the number of deep feature extraction modules based on Transformer is two, and the structures of the two modules are the same and are arranged in parallel. The deep feature extraction module based on Transformer includes an MDTA module and a CDFN module arranged in sequence. The MDTA module uses three convolutional layers and a depth-separable convolutional layer to generate Q, K, and V for the input features, and then multiplies Q and K, and then uses a softmax activation function to generate the weight of each channel, and then multiplies it with V, and finally uses a convolutional layer and adds the input features. The CDFN module uses two convolutional layers and depth-separable convolution, one of which passes through the depth-separable convolution path first uses GELU non-linearity activation, and then multiplies it with the generated result of the other path, and finally passes through a convolutional layer to add the generated result and the input feature to obtain the final deep feature, that is, the deep feature x f3 and y f3 .

6. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 5, characterized in that: In step S4, the specific processing process is as follows: S41: Obtain the shallow features x captured in the feature extraction stage f1 、y f1 、x f2 、y f2 and deep features x f3 、y f3 ; S42: x f1 and y f1 Input to the first iterable two-branch attention mechanism module to obtain the feature x f1_1 and y f1_1 , shallow feature x f2 and y f2 Input to the second iterable two-branch attention mechanism module to obtain the feature x f2_2 and y f2_2 , deep features x f3 and y f3 Input to the third iterable two-branch attention mechanism module to obtain the feature x f3_3 and y f3_3 , and then use concatenate to concatenate features x f1_1 and y f2_2 、y f1_1 and x f2_2 , and get the features x_y respectively f1 and x_y f2 , and then input them into the first and second parts of the Transformer-based modality difference guided fusion module to obtain the preliminary fusion feature x_y pf_1 and x_y pf_2 ; S43: Output features x_y of the first and second parts of the fusion module guided by the modality difference of Transformer pf_1 and x_y pf_2 The features x extracted by the third iterative two-branch attention mechanism module are respectively f3_3 and y f3_3 Use concatenate to get x_y f1_3 and x_y f2_3 , input into the third part of the Transformer-based modality difference guided fusion module to obtain the final fusion feature x_y f_4 ; S44: Using fusion features x_y f_4 , use the feature reconstruction module based on reversible neural network to fusion feature x_y f_4 Perform feature reconstruction and finally obtain the fused image image fusion .

7. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 6, characterized in that In the step S42, the iterative dual-branch attention mechanism module includes two parts set in sequence. The first part first uses two point-by-point convolution layers for processing in sequence, wherein a Rule activation function is used after the first point-by-point convolution layer; after the two point-by-point convolution layers, two paths are used to process the generated features. The first path first uses the GAP module, then uses two point-by-point convolution layers, and finally uses the sigmoid activation function to generate the weight of each channel, and multiplies the obtained weight by the feature of the first pass through the two point-by-point convolution layers; the second path is to directly use two point-by-point convolution layers, and then use the sigmoid activation function to generate the weight of each space, and multiply the obtained weight by the feature of the first pass through the two point-by-point convolution layers; finally, the features obtained by the two paths are added , and then added to the initial infrared and visible light features respectively; then the features of the two paths are spliced and input into the second part. The second part is first processed by a point-by-point convolution layer, and then divided into two paths again. The first path uses the GAP module and then uses two point-by-point convolution layers for processing. A Rule activation function is used after the first point-by-point convolution layer; the second path directly uses two point-by-point convolution layers and uses a Rule activation function after the first point-by-point convolution layer; then the results of these two paths are added together and finally input into the sigmoid activation function. Sigmoid generates a weight ω and multiplies it with the initial visible light feature, and at the same time generates a weight (1-ω) and multiplies it with the initial infrared feature to obtain the output result of the iterative dual-branch attention mechanism module; In step S43, the Transformer-based modality difference guided fusion module includes three parts. The first part has the same structure as the second part and is set in sequence. In the first part, a LN layer and a SA layer are first used to obtain the processed features, and the features are added to the initial input features to obtain the first output features. Then, the features are input to a LN layer and a MLP layer in sequence, and the obtained features are added to the first output features to obtain the preliminary fusion features x_y of the first part. pf_1 In the second part, we also use an LN layer and an SA layer to get the processed features, add the features to the initial input features to get the first output features, and then input them into an LN layer and an MLP layer in sequence. The features obtained are added to the first output features to get the preliminary fusion features x_y of the second part. pf_2 ; The preliminary fusion feature x_y of the first part pf_1 and the output feature x_y of one branch in the third iterable two-branch attention module f1_3 Use the concatenate operation to process the preliminary fusion feature x_y of the second part pf_2 And the output feature x_y of the other branch of the third iterable two-branch attention module f2_3 Use the concatenate operation for processing, and then input the two parts of features obtained by the concatenate operation into the third part of the fusion module guided by the modal difference based on the Transformer to obtain the final fusion feature x_y f_4 In the third part, the input features are first processed by a point-by-point convolution layer and a depth-wise separable convolution layer, and then divided into three parts: Q, K, and V according to the number of channels. Q and K of different modes are multiplied, and then processed by the Softmax activation function and multiplied with V of the current mode. The multiplied features are then subtracted from V and added to the original input features before concatenation to obtain the final fusion feature x_y f_4 ; In the feature reconstruction module based on the reversible neural network of step S44, four INN layers are used, and in the first INN layer, the fusion feature x_y f_4 First, it is split into two branches, each branch has C / 2 channels, and then the elements are added and multiplied to get the output x_y of the first INN layer. INN_1 , the output x_y INN_1 Input to the second INN layer to get x_y INN_2 , change x_y INN_2 Input to the third INN layer to get x_y INN_3 , change x_y INN_3 Input to the fourth INN layer to get x_y INN_4 , and finally x_y INN_4 After a convolution layer, the final fused image is obtained fusion .

8. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 7, characterized in that: In the INN layer, the input features are first split into two branches: the feature of the first branch is F inn_1 , the second branch is characterized by F inn_2 ; The feature F of the first branch inn_1 First, it is processed through a BRB block to obtain F b_1 , followed by the feature F of the second branch inn_2 Add up to get F i_b_1 , and then add the feature F i_b_1 Then, after a BRB block processing, we get F b_2 , then with the feature F of the first branch inn_1 Multiply to get F i_b_2 , then the added feature F i_b_1 Then go through a BRB block to get F b_3 The feature F after multiplication i_b_2 Add them together to get F i_b_3 , and finally F i_b_3 and F i_b_1 Add to get feature F inn_finall , where, in the BRB block, a convolutional layer and Rule6 activation function are first used, followed by depth-wise separable convolution and convolution layers, and finally another convolutional layer is used, and the obtained features are added to the original input to obtain the final output.

9. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 1 or 8, characterized in that: The image fusion method based on iterative dual-branch attention and modality difference guidance also includes the following steps: S5: Model parameter optimization The overall loss function is used to calculate the loss of the fused image and the source image. The gradient is calculated based on the loss value and the back-propagation algorithm, and the model parameters are adjusted through the Adam optimizer. The image sample is input again and iterative training is repeated until the iteration threshold is reached.

10. The image fusion method based on iterative dual-branch attention and modality difference guidance according to claim 9, characterized in that: In step S5, the overall loss function Loss total as follows: Loss total =αLoss in +βLoss grad +γLoss msssim Among them, Loss in Loss is the strength loss. grad is the gradient loss, Loss msssim is the MSSSIM loss, α, β, γ are the weight parameters of the corresponding loss; Strength loss in The calculation formula is as follows: Where H represents the height of the image, W represents the width of the image; I f Represents the obtained fused image, I vi and I ir Represent the input visible light image and infrared image respectively, ||·||1 represents L1 regularization, and max represents the maximum selection of elements; Gradient Loss grad The calculation formula is as follows: in, represents the sobel operator, |·| represents the absolute value operation; MSSSIM Loss msssim The calculation formula is as follows: Loss msssim =β1(1-MSSSIM(I f ,I vi ))+β2(1-MSSSIM(I f ,I ir )) Among them, β1 and β2 are the weights controlling the two losses respectively, and MSSSIM() represents the calculation of multi-scale structure loss.

Images