A distributed infrared and visible light image fusion network

Through the distributed structure of infrared and visible image fusion network, the MDDC multi-scale dense hollow convolution and CMB coordinate average attention fusion module are used, combined with the ECB edge assisted correction module, the lack of texture details and thermal target features in the existing fusion method is solved. The problem of the problem is that high-quality fusion images are generated, which significantly improves the visual effects and evaluation indicators.

CN117935012BActive Publication Date: 2024-11-01GUANGZHOU WEIZHUO INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410134382.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-31
Publication Date
2024-11-01
Estimated Expiration
2044-01-31

AI Technical Summary

Technical Problem

The existing infrared and visible light image fusion methods lack effective texture details and thermal target characteristics, and do not fully consider context information, resulting in incomplete fusion results.

Method used

Using an infrared and visible image fusion network based on a distributed structure, the MDDC multi-scale dense cavity convolution module and the CMB coordinate average attention fusion module are used for feature extraction and fusion, and the edge information is refined through the ECB edge-assisted correction module to construct the total Loss function to constrain the fusion target.

Benefits of technology

A more comprehensive information feature fusion is achieved, the texture details of visible light images and thermal information of infrared images are retained, noise is suppressed, and high-quality fusion images are generated, which significantly improves visual effects and evaluation indicators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117935012B_ABST
    Figure CN117935012B_ABST
Patent Text Reader

Abstract

The present invention provides an infrared and visible light image fusion network based on a distributed structure. The topological structure of the network consists of an upper channel, a lower channel and an intermediate channel. The upper channel and the lower channel are both composed of an MDDC multi-scale dense hole convolution module and an ECB edge auxiliary correction module. The intermediate channel includes a CMB coordinate average attention fusion module. The MDDC multi-scale dense hole convolution module is used to obtain information features of infrared images and visible light images. The ECB edge auxiliary correction module is used to refine the edge details of the information features extracted by the upper channel and the lower channel. The CMB coordinate average attention fusion module is used to perform feature fusion. The upper channel and the lower channel are used to extract features of visible light and infrared images. The intermediate channel fuses the features extracted by the upper channel and the lower channel with the fusion results of the previous step layer by layer, and the last layer generates a fused image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of infrared and visible light image fusion, and in particular to an infrared and visible light image fusion network based on a distributed structure. Background Art

[0002] Image fusion is a key image processing technology that aims to overcome the dilemma that images taken by a single-mode sensor cannot effectively and comprehensively describe the image scene due to the limitations of hardware theory and technology. Among the existing image fusion scenarios, the most widely used is the fusion of visible light and infrared images. Infrared images can show the thermal distribution of objects and can effectively highlight targets even in adverse weather conditions. However, the background details of infrared images are not clear enough. Visible light images provide rich color information and high-resolution texture details, but their visual quality and resolution are highly dependent on lighting and environmental conditions. Fusion of infrared and visible light images can combine the rich details of visible light with the target characteristics of infrared to generate a comprehensive image that contains more useful information than any single source image, providing more favorable assistance for subsequent advanced visual tasks such as target detection, target tracking, and semantic segmentation. At present, the fusion results obtained by most existing methods usually have imperfect texture details and lack thermal target features. Existing fusion methods are usually artificially designed and unlearnable, and do not fully consider contextual information. To address the above problems, this paper proposes a distributed architecture network based on attention mechanism and dense dilated convolution. Summary of the invention

[0003] The purpose of the present invention is to provide an infrared and visible light image fusion network based on a distributed structure to solve the problems raised in the above background technology.

[0004] The present invention is achieved through the following technical solutions:

[0005] The invention discloses an infrared and visible light image fusion network based on a distributed structure. The topological structure of the network consists of an upper channel, a lower channel and an intermediate channel. The upper channel and the lower channel are both composed of sequentially cascaded MDDC multi-scale dense hole convolution modules and ECB edge auxiliary correction modules. The intermediate channel includes a plurality of convolution blocks and a CMB coordinate average attention fusion module. The MDDC multi-scale dense hole convolution module is used to obtain information features of infrared images and visible light images. The ECB edge auxiliary correction module is used to refine the edge details of the information features extracted by the upper channel and the lower channel. The CMB coordinate average attention fusion module is used to perform feature fusion. The MDDC multi-scale dense hole convolution module and the ECB edge auxiliary correction module in the upper channel and the lower channel are used to extract features of visible light and infrared images. The CMB coordinate average attention fusion module in the intermediate channel is used to fuse the features extracted by the two channels with the fusion results of the previous step layer by layer, and a fused image is generated at the last layer.

[0006] Specifically, the MDDC multi-scale dense dilated convolution module consists of four output branches with different receptive fields, a MAT multi-scale attention module, and an SCM space and channel attention fusion module. The MAT multi-scale attention module is used to weight the input features, and then feed the weighted features to four different convolution branches. Multi-scale features are obtained through convolution kernels with different receptive fields, and fused through the SCM space and channel attention fusion module. The fused information features are the final output of the MDDC multi-scale dense dilated convolution module.

[0007] Specifically, the specific process of calculating the spatial attention weight and channel attention weight for the input features is as follows:

[0008]

[0009]

[0010] in, represents the input features, It represents the nth layer of the MDDC multi-scale dense dilated convolution module. represents the convolution operation, (·) represents the sigmoid activation function, (·) represents the global pooling operation, (·) represents the l1 regularization operation, is the weight feature map obtained by the channel attention module, is the weight feature map obtained by the spatial attention module, =1, 2, indicating one of the input features;

[0011] And the attention fusion operation can be expressed as:

[0012]

[0013]

[0014] in, , and They represent the channel attention weighted graph and spatial attention weighted graph calculated by the soft-max operation, respectively. Then, , and It can be obtained by the following formula:

[0015]

[0016]

[0017]

[0018] in, and They represent the enhanced channel attention feature map and spatial attention feature map respectively. is the final output, represents the channel attention feature, represents the spatial attention feature, Represents the final output features.

[0019] Specifically, the CMB coordinate average attention fusion module enhances and extracts the information features through a parallel one-dimensional maximum pooling operation based on the one-dimensional average pooling operation of the CA coordinate attention module. The specific process is as follows:

[0020] Step 1: Input feature map X and perform two parallel one-dimensional pooling operations, where , C represents the number of channels of the feature map, H represents the height of the feature map, W represents the width of the feature map, the one-dimensional pooling operation includes one-dimensional average pooling and one-dimensional maximum pooling, and each parallel pooling method is performed along the width direction and along the height direction;

[0021] Step 2: Transpose the results of the two pooling operations and concatenate them along the spatial dimension to obtain a feature map focusing on the central area of ​​the salient target and a feature map focusing on the edge area with significant changes. Then, perform a 1x1 convolution transformation on these two feature maps, followed by a nonlinear activation operation to increase the nonlinear fitting capability.

[0022]

[0023]

[0024] in, represents the ReLU activation function, represents 1×1 convolution, and Represents the feature map obtained after average pooling operation along the height and width direction of the feature map, and represents the feature map obtained after pooling operations along the height and width directions of the feature map, T represents the transposition operation, [·;·] represents the connection operation along the spatial dimension after transposition, H represents the height of the feature map, W represents the width of the feature map, is the enhanced feature map of average pooling, It is the enhanced feature map of maximum pooling;

[0025] Step 3: Then use the element maximum operation to fuse the two activated feature map information;

[0026]

[0027] in, for and The feature map is fused by element-wise maximum operation;

[0028] Step 4: Decompose the fused features in the spatial dimension to obtain feature vectors in the height and width directions. Then, perform 1x1 convolution transformation and nonlinear activation operations on these two vectors to obtain attention weights in the height and width directions.

[0029]

[0030]

[0031] in, and Represent the two decomposition vectors related to the height and width of F, respectively. represents the sigmoid activation function, represents a 1×1 convolution operation, and are the attention weights in height and width directions;

[0032] Step 5: Weight the feature map X with the attention weights in the height direction and width direction to generate an enhanced feature map;

[0033]

[0034] in, and Indicates the specific position in the height and width directions. represents the enhanced feature map;

[0035] Step 6: Calculate the coordinate average attention map for the two input features input into the CMB coordinate average attention fusion module, perform a soft-max operation on the coordinate average attention map to obtain their respective probability weights, and multiply them with the input features to weight the important features;

[0036]

[0037] in, represents the Ath input feature, represents the Bth input feature, and is the probability weight generated by performing soft-max on the coordinate average attention map of the input feature, It is the final output result of the CMB coordinate average attention fusion module.

[0038] Specifically, the specific process of refining the edge details of the information feature by the ECB edge auxiliary correction module is as follows;

[0039] Get the edge information used to correct the image, whose size is m × n edge gradient map , It is expressed as:

[0040]

[0041]

[0042]

[0043] in, represents the linear operator used to calculate the level, h represents the horizontal direction, represents the linear operator used to calculate the vertical first layer difference, v represents the vertical direction, and Represents the source pixels Left and below The nearest neighbor pixel of

[0044] The spatial gradient filter is used to optimize the edge details to obtain more obvious gradient information. The calculation formula is as follows:

[0045]

[0046] in, is a spatial gradient filter, M and N are both constants, is the horizontal pixel of the feature map, is the vertical pixel of the feature map, and max represents the maximum value operation.

[0047] Specifically, the network constrains the fusion target by constructing a total loss function including intensity loss and detail loss, so as to retain the texture details in the visible light image and highlight the infrared thermal information of the target in the fused image. The total loss function is as follows:

[0048]

[0049] in, represents the intensity loss function, represents the detail loss function, is the hyperparameter of the intensity loss function, is the hyperparameter of the detail loss function.

[0050] Specifically, the strength loss function is calculated by the following formula: :

[0051]

[0052] The detail loss function is calculated by the following formula :

[0053]

[0054] Among them, H represents the height of the feature map, W represents the width of the feature map, max(·) represents the maximum selection operation of the element, |·| represents the absolute value, is the pixel value of the fusion result of infrared and visible light images, is the pixel value of the infrared image, is the pixel value of the visible light image, is the sobel gradient operator.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention provides an infrared and visible light image fusion network based on a distributed structure. The distributed fusion framework is different from the existing infrared and visible light image fusion framework. Three channels are used to realize image fusion, wherein the upper and lower channels are used to extract information features, and the other middle channel is used to fuse information features. In order to obtain more comprehensive information features, an MDDC multi-scale dense hole convolution module is added for more efficient feature extraction. In addition, a CMB coordinate average attention fusion module is added to more specifically fuse the extracted multi-scale information. Therefore, our fusion result can retain more useful information features. The network can obtain a rough image through the cooperation of the MDDC multi-scale dense hole convolution module and the CMB coordinate average attention fusion module, and then refine the edge information in the fusion result through the ECB edge auxiliary correction module, so as to suppress the influence of noise on the fusion result and obtain the final fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A schematic diagram of the structure of an infrared and visible light image fusion network based on a distributed structure provided by the present invention.

[0059] Figure 2 Schematic diagram of the structure of the SCM space and channel attention fusion module.

[0060] Figure 3 Schematic diagram of the specific steps of the CMB coordinate average attention fusion module.

[0061] Figure 4 Schematic diagram of the process of calculating attention features of input features for the CMB coordinate average attention fusion module.

[0062] Figure 5 This is an objective comparison result diagram of different methods on the TNO dataset in the comparative experiment.

[0063] Figure 6 This is a diagram showing the subjective comparison results of different methods on the TNO dataset in the comparative experiment.

[0064] Figure 7 The figure shows the subjective comparison results of different parameters in the ablation experiment.

[0065] Figure 8This is an objective comparison result diagram of different parameters in the ablation experiment.

[0066] Fig. 9 The figure shows the subjective comparison results with and without the ECB module on the RoadScene dataset.

[0067] Fig.10 The figure shows the objective comparison results with and without the ECB module on the RoadScene dataset. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0069] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0070] It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0071] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0072] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0073] See also Figure 1 A distributed structure-based infrared and visible light image fusion network, the topological structure of the network consists of an upper channel, a lower channel and an intermediate channel, the upper channel and the lower channel are both composed of sequentially cascaded MDDC multi-scale dense hole convolution modules and ECB edge auxiliary correction modules, the intermediate channel includes a plurality of convolution blocks and a CMB coordinate average attention fusion module, the MDDC multi-scale dense hole convolution module is used to obtain information features of infrared images and visible light images, the ECB edge auxiliary correction module is used to refine the edge details of the information features extracted by the upper channel and the lower channel, the CMB coordinate average attention fusion module is used to perform feature fusion, the MDDC multi-scale dense hole convolution module and the ECB edge auxiliary correction module in the upper channel and the lower channel are used to extract features of visible light and infrared images, the CMB coordinate average attention fusion module in the intermediate channel is used to fuse the features extracted by the two channels with the fusion results of the previous step layer by layer, and the last layer generates a fused image.

[0074] The infrared and visible light image fusion network based on distributed structure provided by the present invention is a distributed architecture network based on attention mechanism and dense hole convolution, which realizes data fusion based on three channels. The network adopts a distributed fusion framework to make full use of the fusion output of the previous step. It can make full use of the information features of infrared images and visible light images. First, the two channels use the MDDC multi-scale dense hole convolution module to obtain rich context information from the source image. Then, a fusion strategy based on the CMB coordinate average attention fusion module is adopted to achieve the fusion between the results of the two channels. Then, the extracted features and the fusion results of the previous step are sent to the intermediate channel for fusion to reduce the loss of information features in the infrared image and the visible light image. Furthermore, an ECB edge auxiliary correction module is added, which can refine the edge details of the fusion result and effectively suppress noise. The subjective qualitative and objective quantitative comparison results of the proposed method and other mainstream and classic fusion methods on public data sets show that our fusion image results have better performance in visual effects and evaluation indicators.

[0075] For details, see Figure 2 The MDDC multi-scale dense hole convolution module consists of four output branches with different receptive fields, a MAT multi-scale attention module, and an SCM space and channel attention fusion module. The MAT multi-scale attention module is used to weight the input features, and then feed the weighted features into four different convolution branches. Multi-scale features are obtained through convolution kernels with different receptive fields, and fused through the SCM space and channel attention fusion module. The fused information features are the final output of the MDDC multi-scale dense hole convolution module.

[0076] Specifically, the specific process of calculating the spatial attention weight and channel attention weight for the features of the input SCM space and channel attention fusion module is as follows:

[0077]

[0078]

[0079] in, represents the input features, It represents the nth layer of the MDDC multi-scale dense dilated convolution module. represents the convolution operation, (·) represents the sigmoid activation function, (·) represents the global pooling operation, (·) represents the l1 regularization operation, is the weight feature map obtained by the channel attention module, is the weight feature map obtained by the spatial attention module, =1, 2, indicating one of the input features;

[0080] And the attention fusion operation can be expressed as:

[0081]

[0082]

[0083] in, , and They represent the channel attention weighted graph and spatial attention weighted graph respectively calculated by the soft-max operation on the input features. Then, , and It can be obtained by the following formula:

[0084]

[0085]

[0086]

[0087] in, and They represent the enhanced channel attention feature map and spatial attention feature map respectively. is the final output, represents the channel attention feature, represents the spatial attention feature, Represents the final output features.

[0088] For details, see Figure 3 and Figure 4 The CMB coordinate average attention fusion module enhances the extraction of the information features through a parallel one-dimensional maximum pooling operation based on the one-dimensional average pooling operation of the CA coordinate attention module. The specific process is as follows:

[0089] Step 1: Input feature map X and perform two parallel one-dimensional pooling operations, where , C represents the number of channels of the feature map, H represents the height of the feature map, W represents the width of the feature map, the one-dimensional pooling operation includes one-dimensional average pooling and one-dimensional maximum pooling, and each parallel pooling method is performed along the width direction and along the height direction;

[0090] Step 2: Transpose the results of the two pooling operations and concatenate them along the spatial dimension to obtain a feature map focusing on the central area of ​​the salient target and a feature map focusing on the edge area with significant changes. Then, perform a 1x1 convolution transformation on these two feature maps, and then perform a nonlinear activation operation to increase the nonlinear fitting capability.

[0091]

[0092]

[0093] in, represents the ReLU activation function, represents 1×1 convolution, and Represents the feature map obtained after average pooling operation along the height and width direction of the feature map, and represents the feature map obtained after pooling operations along the height and width directions of the feature map, T represents the transposition operation, [·;·] represents the connection operation along the spatial dimension after transposition, H represents the height of the feature map, W represents the width of the feature map, is the enhanced feature map of average pooling, It is the enhanced feature map of maximum pooling;

[0094] Step 3: Then use the element maximum operation to fuse the two activated feature map information;

[0095]

[0096] in, for and The feature map is fused by element-wise maximum operation;

[0097] Step 4: Decompose the fused features in the spatial dimension to obtain feature vectors in the height and width directions. Then, perform 1x1 convolution transformation and nonlinear activation operations on these two vectors to obtain attention weights in the height and width directions.

[0098]

[0099]

[0100] in, and Represent the two decomposition vectors related to the height and width of F, respectively. represents the sigmoid activation function, represents a 1×1 convolution operation, and are the attention weights in height and width directions;

[0101] Step 5: Weight the feature map X with the attention weights in the height direction and width direction to generate an enhanced feature map;

[0102]

[0103] in, and Indicates the specific position in the height and width directions. represents the enhanced feature map;

[0104] Step 6: Calculate the coordinate average attention map for the two input features input into the CMB coordinate average attention fusion module, perform a soft-max operation on the coordinate average attention map to obtain their respective probability weights, and multiply them with the input features to weight the important features;

[0105]

[0106] in, represents the Ath input feature, represents the Bth input feature, and is the probability weight generated by performing soft-max on the coordinate average attention map of the input feature, It is the final output result of the CMB coordinate average attention fusion module.

[0107] Specifically, before outputting the fused image, the information features are input into the ECB edge auxiliary correction module to correct the edge details. The specific correction process is as follows:

[0108] Get the edge information used to correct the image, whose size is m × n edge gradient map , It is expressed as:

[0109]

[0110]

[0111]

[0112] in, represents the linear operator used to calculate the level, h represents the horizontal direction, represents the linear operator used to calculate the vertical first layer difference, v represents the horizontal direction, and Represents the source pixels nearest neighbor pixels to the left and below;

[0113] The spatial gradient filter is used to optimize the edge details to obtain more obvious gradient information. The calculation formula is as follows:

[0114]

[0115] in, is a spatial gradient filter, M and N are both constants, is the horizontal pixel of the feature map, is the vertical pixel of the feature map, and max represents the maximum value operation.

[0116] Specifically, the network constrains the fusion target by constructing a total loss function including intensity loss and detail loss, so as to retain the texture details in the visible light image and highlight the infrared thermal information of the target in the fused image. The total loss function is as follows:

[0117]

[0118] in, represents the intensity loss function, represents the detail loss function, is the hyperparameter of the intensity loss function, is the hyperparameter of the detail loss function.

[0119] Specifically, the strength loss function is calculated by the following formula: :

[0120]

[0121] The detail loss function is calculated by the following formula :

[0122]

[0123] Among them, H represents the height of the feature map, W represents the width of the feature map, max(·) represents the maximum selection operation of the element, |·| represents the absolute value, is the pixel value of the fusion result of infrared and visible light images, is the pixel value of the infrared image, is the pixel value of the visible light image, is the sobel gradient operator.

[0124] Furthermore, the image fusion effect of the network was evaluated, and the evaluation indicators included: MI mutual information, EN entropy, VIF visual information fidelity, SD forest stand deviation, SF spatial frequency and AG average gradient.

[0125] Among them, the MI mutual information quantifies the amount of information obtained by the fused image from the source image, the EN entropy evaluates the amount of information contained in the fused image based on information theory, the VIF visual information fidelity mainly calculates the information fidelity in the fused image, which is consistent with human visual perception, the SD stand deviation is a reflection of the image contrast based on the static concept, the larger the SD stand deviation value, the better the contrast distribution in the image, and the more information the image carries, the SF spatial frequency reflects the rate of change of the image grayscale, and the AG average gradient can measure the clarity of the fused image, and the higher the values ​​of these indicators, the better the performance of the fusion network.

[0126] The fusion results of the network model provided by the present invention are compared with the currently advanced fusion network models. The test phase is carried out on the TNO and RoadScene data sets. Advanced infrared and visible light image fusion open source methods such as fusion frameworks based on deep learning, such as DDcGAN, Densefuse, Nestfuse, IFCNN, RFNNest, SeAfusion, U2Fusion, SDNet, and STDFusionNet, are selected for comparison with the method of the present invention; the details are as follows:

[0127] Comparative experiment: Objective evaluation index calculation and comparative experiment were performed on the fusion results of the above fusion network model on 20 images in the TNO dataset. The comparison results can be found in Figure 5 and Figure 6As can be seen from the figure, the fusion network of the present invention has the best results in AG average gradient and EN entropy, which shows that the fusion network of the present invention has a good ability to retain information from the source image. In addition, the highest SD forest stand deviation result also indicates that the fusion result of the image fusion network of the present invention has superior contrast. The best result based on VIF visual information fidelity shows that the proposed method can generate high-quality images that are more in line with the human visual system. Although the proposed image fusion network is not optimal in MI mutual information, the fusion performance of this network is the best after comprehensive evaluation of multiple objective indicators, which is consistent with the subjective evaluation results.

[0128] Ablation experiment: The optimal parameters of the loss function are determined in the experiment, and the effectiveness of the ECB edge auxiliary correction module and the CMB coordinate average attention fusion module in improving the fusion network is verified. The experimental results can be found in Figure 7 and Figure 8 , Figure 7 The calculation results of the objective indicators of the fusion results under different parameters are given. Among the 7 objective evaluation indicators, when the combination of α=1 and β=1 is taken, several indicators of the fusion results are optimal. Therefore, compared with other loss parameters, this set of parameters has better fusion performance. In addition, in order to facilitate the observation of the impact of loss parameters on fusion performance, Figure 8 The image comparison of the fusion results of different parameter groups is given. When the parameter combination is α=10, β=1, the weight of the intensity loss is large, so more intensity information features are retained. It can be observed that the edges of the poles and leaves are covered by strong light. When the parameter combination is α=1, β=10, the edge details are very blurred, the overall image becomes darker, and less pixel intensity information is retained. When the combination of α=1, β=1 is taken, the edge details of the poles and leaves in the fused image are clearer, the image contrast is higher, and the visual quality is better. In summary, compared with the other two sets of parameters, the parameter combination of α=1, β=1 has better fusion performance and is the parameter used in subsequent experiments.

[0129] The difference between the two models is tested under the same input using a model with an ECB edge-assisted correction module and a model without the ECB edge-assisted correction module. Specifically, a visible light image and an infrared image of a street scene in the street dataset are used as input to obtain the fusion result. Fig. 9The fusion results and regional comparison of the two models are shown. From the first row of comparison images, it can be seen that in the daytime street scene, the street lights in the fusion results processed by the ECB edge-assisted correction module have better texture details and are easier to distinguish. In addition, as shown in the boxes in the second row of comparison images, it is easier to find manhole covers in the fusion results processed by an ECB edge-assisted correction module, and the manhole covers have more obvious edge features, while the manhole covers in the fusion results not processed by the ECB edge-assisted correction module are difficult to distinguish. In addition, the present invention also calculates objective indicators and the influence of human subjective factors, such as Fig.10 The results processed by the ECB edge-assisted correction module have higher EN entropy, AG average gradient, SD forest stand deviation, and SF spatial frequency on the RoadScene dataset, which shows that the fusion results processed by the ECB edge-assisted correction module not only retain rich information and features from the source image, but also the objects in it have excellent advantages in contrast with the environment and can be better identified.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on a distributed structure, characterized in that: The invention comprises an infrared and visible light image fusion network based on a distributed structure, wherein the topological structure of the network consists of an upper channel, a lower channel and an intermediate channel, wherein the upper channel and the lower channel are both composed of sequentially cascaded MDDC multi-scale dense hole convolution modules and ECB edge auxiliary correction modules, wherein the intermediate channel comprises a plurality of convolution blocks and a CMB coordinate average attention fusion module, wherein the MDDC multi-scale dense hole convolution module is used to obtain information features of infrared images and visible light images, wherein the ECB edge auxiliary correction module is used to refine edge details of information features extracted from the upper channel and the lower channel, wherein the CMB coordinate average attention fusion module is used to perform feature fusion, wherein the MDDC multi-scale dense hole convolution module and the ECB edge auxiliary correction module in the upper channel and the lower channel are used to extract features from visible light and infrared images, wherein the CMB coordinate average attention fusion module in the intermediate channel is used to fuse the features extracted from the two channels with the fusion results of the previous step layer by layer, and a fused image is generated at the last layer; The MDDC multi-scale dense hole convolution module consists of four output branches with different receptive fields, a MAT multi-scale attention module, and an SCM space and channel attention fusion module. The MAT multi-scale attention module is used to weight the input features, and then feed the weighted features into four different convolution branches. Multi-scale features are obtained through convolution kernels with different receptive fields, and fused through the SCM space and channel attention fusion module. The fused information features are the final output of the MDDC multi-scale dense hole convolution module. The CMB coordinate average attention fusion module uses a one-dimensional average pooling operation in the CA coordinate attention module, and then performs enhanced extraction of the information features through a parallel one-dimensional maximum pooling operation. The specific process is as follows: Step 1: Input feature map X and perform two parallel one-dimensional pooling operations, where X∈R CxHxW , C represents the number of channels of the feature map, H represents the height of the feature map, W represents the width of the feature map, the one-dimensional pooling operation includes one-dimensional average pooling and one-dimensional maximum pooling, and each parallel pooling method is performed along the width direction and along the height direction; Step 2: Transpose the results of the two pooling operations and concatenate them along the spatial dimension to obtain a feature map focusing on the central area of ​​the salient target and a feature map focusing on the edge area with significant changes. Then, perform a 1x1 convolution transformation on these two feature maps, followed by a nonlinear activation operation to increase the nonlinear fitting capability. in, ReLU activation function, Conv1 1×1 convolution, avgz H and avgz W Represents the feature map obtained after average pooling along the height and width of the feature map, maxz H and maxz W represents the feature map obtained by performing the maximum pooling operation along the height and width directions of the feature map, T represents the transposition operation, [·;·] represents the connection operation along the spatial dimension after transposition, H represents the height of the feature map, W represents the width of the feature map, and F avg is the enhanced feature map of average pooling, F max It is the enhanced feature map of maximum pooling; Step 3: Then use the element maximum operation to fuse the two activated feature map information; F=max(F avg ,F max ) Where F is F avg and F max The feature map is fused by element-wise maximum operation; Step 4: Decompose the fused features in the spatial dimension to obtain feature vectors in the height and width directions. Then, perform 1x1 convolution transformation and nonlinear activation operations on these two vectors to obtain attention weights in the height and width directions. S H =σ(Conv1(F H )) S W =σ(Conv1(F w )) Among them, F H and F w Represent two decomposition vectors related to the height and width of F, σ represents the sigmoid activation function, Conv1 represents a 1×1 convolution operation, and S H and S W are the attention weights in height and width directions; Step 5: Weight the feature map X with the attention weights in the height direction and width direction to generate an enhanced feature map; O(i,j)=X(i,j)×S H (i)×S W (j) Among them, i∈{1,2,…,H} and j∈{1,2,…,W} represent the specific positions in the height direction and width direction, and O(i,j) represents the enhanced feature map; Step 6: Calculate the coordinate average attention map for the two input features input into the CMB coordinate average attention fusion module, perform the soft-max operation on the coordinate average attention map to obtain the respective probability weights, and multiply it with the input features to weight the important features; X out =X A ×ω A +X B ×ω B Among them, X A represents the Ath input feature, X B represents the Bth input feature, ω A and ω B is the probability weight generated by performing soft-max on the coordinate average attention map of the input feature, X out It is the final output result of the CMB coordinate average attention fusion module; The specific process of calculating the spatial attention weight and channel attention weight for the features of the input SCM spatial and channel attention fusion module is as follows: in, Represents the input features, N represents the Nth layer in the MDDC multi-scale dense hole convolution module, Conv represents the convolution operation, represents the sigmoid activation function, P(·) represents the global pooling operation, L(·) represents the regularization operation, is the weight feature map obtained by the channel attention module, is the weighted feature map obtained by the spatial attention module, k = 1, 2, representing one of the input features; And the attention fusion operation is expressed as: Where, M = 2, and They represent the channel attention weighted graph and spatial attention weighted graph respectively calculated by the soft-max operation on the input features. Then, and It is obtained by the following formula: in, and They represent the enhanced channel attention feature map and spatial attention feature map respectively. is the final output, fc represents the channel attention feature, fs represents the spatial attention feature, and f represents the final output feature.

2. The infrared and visible light image fusion method based on a distributed structure according to claim 1, characterized in that: The specific process of refining the edge details of information features by the ECB edge auxiliary correction module is as follows: Get the edge information used to correct the image, whose size is m×n edge gradient map It is expressed as: in, represents the linear operator used to calculate the level, h represents the horizontal direction, Represents a linear operator for calculating the vertical first layer difference, v represents the vertical direction, u l(a) and u b(a) They represent the nearest neighbor pixels to the left l and below b of the a-th source pixel respectively; The spatial gradient filter is used to optimize the edge details to obtain more obvious gradient information. The calculation formula is as follows: in, is a spatial gradient filter, m and n are both constants, e is the horizontal pixel of the feature map, q is the vertical pixel of the feature map, and max represents the maximum value operation.

3. The infrared and visible light image fusion method based on a distributed structure according to claim 2 is characterized in that: The network constrains the fusion target by constructing a total loss function including intensity loss and detail loss, so as to retain the texture details in the visible light image and highlight the infrared thermal information of the target in the fused image. The total loss function is as follows: L fusion =αL intensity +βL detail Among them, L intensity represents the intensity loss function, L fusion represents the detail loss function, α is the hyperparameter of the intensity loss function, and β is the hyperparameter of the detail loss function.

4. The infrared and visible light image fusion method based on a distributed structure according to claim 3 is characterized in that: The strength loss function L is calculated by the following formula intensity : The detail loss function L is calculated by the following formula detail : Among them, H represents the height of the feature map, W represents the width of the feature map, max(·) represents the maximum selection operation of the element, |·| represents the absolute value, is the pixel value of the fusion result of infrared and visible light images, is the pixel value of the infrared image, is the pixel value of the visible light image, is the sobel gradient operator.