An image fusion method based on difference saliency aggregation and joint gradient constraint

Abstract

The present invention discloses an image fusion method based on difference saliency aggregation and joint gradient constraint, comprising the following steps: inputting two types of source images, infrared and visible light, into a fusion network, integrating through a regional aggregation strategy to obtain a difference joint saliency map; inputting the generated difference joint saliency map and the two types of source images into a feature fusion subnetwork together, reconstructing the features through convolution to obtain a preliminary fusion image; constructing a joint gradient map containing complementary texture information of the source images through a dual-channel gradient aggregation module in the fusion network; calculating the content loss between the preliminary fusion image and the two types of source images, infrared and visible light, respectively, in the generator of the fusion network; calculating the adversarial loss between the joint gradient map and the gradient map of the preliminary fusion image in the discriminator of the fusion network; using the content loss and the adversarial loss together to train the fusion network to generate a final fusion image. The present invention realizes the fusion of infrared and visible light images.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to an image fusion method based on difference significance aggregation and joint gradient constraint. Background Art

[0002] Image fusion technology is to integrate the data collected by multiple sensors or shooting conditions in the same scene to form an image with different descriptions of the same scene, which is convenient for the computer to further identify and process. In the task of infrared and visible light image fusion, due to the hardware conditions of the imaging equipment, interference in the imaging and transmission process, the influence of natural environment and other factors, the final image will inevitably have problems such as distortion and blur. Image fusion technology can well solve the defect of insufficient single image information, improve the richness of image content, make the expression of the picture more substantial, reduce information redundancy, and achieve better visual effects.

[0003] The purpose of image fusion is to obtain images with higher visual effects. However, due to technical conditions and environmental factors, background high-brightness information such as smoke, strong light, and strong light like fog at night is usually retained in visible light images. Once there is a target in the visible light image that is blocked by the background highlight information, the integrity of the texture information and infrared intensity characteristics of the foreground target in the fused image will inevitably be affected. At the same time, it is difficult to retain meaningful background highlight information while displaying the complete foreground target in the fused image. If the fusion process cannot retain meaningful image content, the redundant information in the fused image will weaken the expression of effective information. Summary of the invention

[0004] The purpose of the present invention is to provide an image fusion method based on difference significance aggregation and joint gradient constraint, which realizes the fusion of infrared and visible light images.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: an image fusion method based on difference significance aggregation and joint gradient constraint, comprising the following steps:

[0006] S1. Input the infrared and visible light source images into a fusion network; the fusion network includes a generator, a dual-channel gradient aggregation module and a discriminator, wherein the generator includes a saliency difference perception aggregation subnetwork and a feature fusion subnetwork; the infrared and visible light source images are first input into a saliency difference perception aggregation subnetwork, which includes a multi-scale stereo attention module and a regional aggregation strategy, and the attention information of the saliency regions of the infrared and visible light source images are respectively extracted through the multi-scale stereo attention module, and the difference joint saliency map is obtained through integration through the regional aggregation strategy;

[0007] S2, inputting the generated difference joint saliency map and the two types of source images into a feature fusion subnetwork; wherein the feature fusion subnetwork includes a gradient residual module of a chain structure with added jump connections, and the gradient residual modules are sequentially connected to extract the features of the difference joint saliency map and the two types of source images of infrared and visible light, and then reconstructing the features through convolution to obtain a preliminary fused image;

[0008] S3, construct a joint gradient map containing complementary texture information of the source images through the dual-channel gradient aggregation module in the fusion network; combine the joint gradient map with the gradient of the preliminary fusion image Figure 1 The images are input into the discriminator of the fusion network to enhance the texture details of the preliminary fusion image.

[0009] S4. In the generator of the fusion network, the content loss between the preliminary fusion image and the two types of source images, infrared and visible light, is calculated respectively; in the discriminator of the fusion network, the adversarial loss between the joint gradient map and the gradient map of the preliminary fusion image is calculated; the content loss and the adversarial loss are used together to train the fusion network to optimize the effect of image fusion; when the number of training rounds reaches the preset number, the network training is completed and the final fusion image is generated.

[0010] The S1 is specifically:

[0011] S11, inputting two types of source images, infrared and visible light images, into the saliency difference perception aggregation sub-network respectively; using the multi-scale stereo attention module to perceive the spatial and channel attention information of the salient regions of the two types of source images at different scales, and obtaining the saliency feature map of the corresponding source image;

[0012] S12, integrate the significant feature maps through the regional aggregation strategy to obtain the difference joint significant map I mask .

[0013] The S2 is specifically:

[0014] S21. In the feature fusion subnetwork, the difference is combined with the saliency map I mask Multiply the infrared and visible light images element by element to obtain the salient target area map I t and significant background area Figure I d ; to I t and infrared images, I d The shallow features of the image and visible light image are extracted using 3x3 convolutional layers in two parallel feature extraction branches respectively; in each parallel feature extraction branch, the extracted shallow features are connected in series in the channel dimension, and the gradient residual module with a chain structure of jump connection is added in the feature fusion sub-network to further extract deep features;

[0015] S22, the shallow features after concatenation are further extracted through sequentially connected gradient residual modules, and the adjacent gradient residual modules are connected through jump connections to avoid the loss of context information; in the gradient residual module, the mainstream is densely connected by two 3x3 convolutions to extract deep features, and the residual stream is extracted and retains the fine-grained features of the source image through gradient operation using the Sobel gradient operator. The deep features and fine-grained features obtained by the mainstream and residual streams are respectively integrated after splicing in the channel dimension; the deep features and fine-grained features extracted from the gradient residual module are reconstructed through 4 3x3 convolution layers. Except for the last layer that uses the Tanh activation function to obtain a preliminary fused image, the remaining layers all use BN normalization and LRelu activation function;

[0016] The S3 is specifically:

[0017] S31, constructing a joint gradient map through a dual-channel gradient aggregation module: first, the gradients of the infrared and visible light channels are calculated separately through the Sobel operator; then, the gradient information of the infrared and visible light channels is integrated through a dual-channel aggregation strategy to obtain a joint gradient map I containing complementary texture information of the source image grad , this process is expressed as:

[0018]

[0019] in, represents gradient operation, specifically, the sobel operator calculates the gradient; abs(·) represents absolute value operation, and max(·) represents the maximum selection strategy at the pixel level;

[0020] S32, the gradient of the joint gradient map with complementary texture features and the gradient of the preliminary fusion image Figure 1 The discriminator is a four-layer network structure. The first three layers use 3x3 convolution kernels, and the last layer is a fully connected layer, and the sigmoid function is used to output the discrimination probability. The discriminator is used to calculate the similarity of texture information between the joint gradient map and the gradient map of the preliminary fusion image, so as to further characterize the texture details of the foreground target and background semantic information in the preliminary fusion image.

[0021] The S4 is specifically:

[0022] S41, respectively calculating the content loss between the preliminary fused image and the two types of source images in the generator; wherein the content loss at least includes pixel intensity loss, structural similarity loss, and adversarial loss of the generator;

[0023] S42. In the discriminator, the adversarial loss between the joint gradient map and the preliminary fused image gradient map is calculated; the network is trained together with the content loss to optimize the image fusion effect; the total number of training rounds is preset, and when the number of training rounds reaches the preset total number of training rounds, the network training is considered to be completed, and the final fused image is generated.

[0024] The working mechanism of the multi-scale stereo attention module in S11 is:

[0025] In this module, the input image is extracted through 3x3 depth-wise separable convolution to obtain the general feature F0 of each channel; F0 is divided into different scale features through 3x3 depth-wise separable convolution with different dilation rates; the attention weights of different branches of multi-scale features in space and channel dimensions are calculated through stereo attention; the calculation formula of stereo attention is expressed as:

[0026]

[0027] Among them, v represents the stereo attention weight, c and s represent channel attention and spatial attention respectively. represents element-wise multiplication;

[0028] The stereo attention weights are processed by the softmax function to obtain the final stereo attention weights, which integrate features of different scales. The fused feature F is adjusted through a 1x1 convolution channel and then summed with the input image to obtain the saliency feature map of the corresponding source image.

[0029] The S12 is specifically:

[0030] The salient feature maps of the corresponding infrared and visible light images are integrated through the regional aggregation strategy: all salient regions of the source image are integrated through pixel-level maximum selection:

[0031]

[0032] Among them, I joint (i, j) represents the pixel value at position (i, j) of the salient feature map after integration, I ir_mask (i, j) and I vi_mask (i, j) are the pixel values ​​at position (i, j) of the salient feature maps of the infrared and visible light images respectively;

[0033] The integrated salient feature map is binarized by the OTSU threshold segmentation method to obtain the difference joint saliency map, which is expressed as:

[0034] I mask =Threshold OTSU (I joint )

[0035] Among them, Threshold OTSU (·) indicates the OTSU threshold segmentation method.

[0036] The S41 is specifically:

[0037] The content loss of the generator includes pixel intensity loss, similarity loss and generator adversarial loss, which are expressed as: L int ,L sim ,L adv ;in,

[0038] Pixel intensity loss L int The definition is expressed as:

[0039]

[0040] Where H and W are the height and width of the image, respectively, ||·||1 represents the L1 norm, and max(·) represents the maximum selection at the image pixel level. The intensity loss integrates the pixel intensity distributions of the infrared and visible images through the maximum selection strategy to constrain the pixel intensity distribution of the fused image.

[0041] Similarity loss L sim The definition is expressed as:

[0042]

[0043] Among them, SSIM(I f ,I ir ) represents image I f and I ir The structural similarity measure between ir and I vi Infrared and visible light images respectively;

[0044] Generator’s adversarial loss L adv The definition is expressed as:

[0045]

[0046] Where c is the probability label with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images.

[0047] Constructing the generator content loss L for the image fusion task G , which is defined as:

[0048] L G =λ1L int +λ2L SSIM +λ3L adv

[0049] Among them, λ1,λ2,λ3 are the weights controlling each loss;

[0050] The S42 is specifically:

[0051] S42, the similarity between the fused image gradient map and the joint gradient map is calculated by the adversarial loss function of the discriminator, so as to reduce the difference in texture features between the preliminary fused image and the joint gradient map; the adversarial loss function of the discriminator Defined as:

[0052]

[0053] Where a is the label of the gradient map of the fused image, with a value of 0; b is the label of the joint gradient map, with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images.

[0054] The total number of training rounds is preset to E. When the number of training rounds reaches E, the network training is considered completed and the final fused image is generated.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) This paper proposes a novel generative adversarial network with difference saliency aggregation and joint gradient constraint to achieve the fusion of infrared and visible light images. This method well coordinates the relationship between fully representing the target and retaining meaningful highlight information. The generated fused image has richer semantic information, which helps to meet the needs of advanced visual tasks.

[0057] (2) The present invention designs a saliency difference perception aggregation subnetwork. The multi-scale stereo attention mechanism is used to perceive the spatial position information and channel attention information of the saliency regions of the source images of different modalities. The regional aggregation strategy can stereoscopically integrate the saliency region differences in the images of different modalities and construct a difference joint saliency map. The difference joint saliency map can effectively alleviate the difficulty of retaining the target features obscured by the background highlight information.

[0058] (3) The present invention designs a chained gradient residual module with added skip connections, in which the gradient residual modules are sequentially connected to enhance the ability to extract image target features and fine-grained information, and skip connections are added between adjacent gradient residual modules to avoid the loss of contextual information. In the gradient residual module, the mainstream uses dense connections to enhance the ability of network feature reuse, and the residual flow uses gradient operators to improve the description of fine-grained information;

[0059] (4) The dual-channel gradient aggregation module strengthens the connection between the gradient information of infrared and visible light images, generating a joint gradient map containing complementary texture information of the source images. Under the adversarial constraints of the joint gradient map, the fused image can effectively show rich texture details that contain both foreground objects and background semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of a multi-scale stereo attention module structure in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a fusion network structure in an embodiment of the present invention;

[0064] Figure 5 Schematic diagram of the structure of the gradient residual module in an embodiment of the present invention;

[0065] Figure 6 Schematic diagram of a discriminator network structure in an embodiment of the present invention;

[0066] Figure 7 This is a fusion effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0068] See also Figure 1-Figure 7 , the technical solution of the present invention is:

[0069] An image fusion method based on difference significance aggregation and joint gradient constraint, the method comprises the following steps:

[0070] 1. Input the two types of source images, infrared and visible light, into the constructed fusion network. The fusion network mainly consists of three parts: generator, dual-channel gradient aggregation module and discriminator. Among them, the generator is composed of a saliency difference perception aggregation subnetwork and a feature fusion subnetwork. The two types of source images, infrared and visible light, are first input into the saliency difference perception aggregation subnetwork, which includes a multi-scale stereo attention module and a regional aggregation strategy. The attention information of the salient areas of the image is extracted separately through the multi-scale stereo attention module, and the difference joint saliency map is obtained through integration of the regional aggregation strategy;

[0071] (1) The two types of source images, infrared and visible light images, are input into the saliency difference perception aggregation subnetwork respectively. First, a multi-scale stereo attention module is used to perceive the channel and spatial position information of the salient regions of the two types of source images, and obtain the saliency feature map of the corresponding source image:

[0072] In the multi-scale stereo attention module, the input image is first convolved with a 3x3 depthwise separable convolution (DSConv3x3) to obtain the general feature F0 of each channel. On this basis, DSConv3x3 convolution with different expansion rates is used to divide F0 into different scale features. In order to make full use of the relationship between multi-scale features in space and channel dimensions, stereo attention is used to calculate the attention weights of different branches. The calculation formula of stereo attention is: Where v represents the stereo attention weight, c and s represent channel attention and spatial attention respectively. Represents element-by-element multiplication. The final stereo attention weight is obtained after being processed by the softmax function, integrating features of different scales. The fusion feature F is adjusted through a 1x1 convolution channel and then summed with the image feature to obtain the final output result.

[0073] (2) The salient feature maps are integrated three-dimensionally through a simple and effective regional aggregation strategy to obtain the difference joint saliency map I mask The process is:

[0074] The salient feature maps of the corresponding infrared and visible light images are integrated through a region aggregation method. First, all the salient regions of the source image are integrated through pixel-level maximum selection:

[0075]

[0076] Among them, I joint (i, j) represents the pixel value at position (i, j) of the salient feature map after integration, I ir_mask (i, j) and I vi_mask(i, j) are the pixel values ​​at the position (i, j) of the salient feature maps of the infrared and visible light images respectively. Then, the integrated salient feature map is binarized by OTSU threshold segmentation to obtain the final difference joint saliency map:

[0077] I mask =Threshold OTSU (I joint )

[0078] Threshold OTSU (·) indicates the OTSU threshold segmentation method.

[0079] 2. The generated difference joint saliency map and the two types of source images are input into the feature fusion sub-network. The feature fusion sub-network contains a chained gradient residual module with added jump connections, where the gradient residual modules are connected in sequence to extract the difference joint saliency map and the features of the two types of source images, infrared and visible light. The jump connections between adjacent modules avoid the loss of context information; then the features are reconstructed through convolution to obtain a preliminary fused image. This process is:

[0080] In the feature fusion sub-network, the difference is combined with the saliency map I mask Multiply the infrared and visible light images element by element to obtain the salient target area map I t and significant background area Figure I d ; to I t and infrared images, I d The shallow features of the image and visible light image are extracted using 3x3 convolutional layers in two parallel feature extraction branches respectively; in each parallel feature extraction branch, the extracted shallow features are connected in series in the channel dimension, and the chained gradient residual module with jump connection in the feature fusion subnetwork is used to further extract deep features;

[0081] The shallow features after concatenation are further extracted through the gradient residual modules connected in sequence. At the same time, the adjacent gradient residual modules are connected through jump connections to avoid the loss of context information. In the gradient residual module, the mainstream is densely connected by two 3x3 convolutions to extract deep features, and the residual stream is extracted and retains the fine-grained features of the source image through gradient operation using the Sobel gradient operator. The deep features and fine-grained features obtained by the mainstream and residual streams are respectively integrated after splicing in the channel dimension. The deep features and fine-grained features extracted from the gradient residual module are reconstructed through 4 3x3 convolution layers. Except for the last layer that uses the Tanh activation function to obtain a preliminary fused image, the remaining layers use BN normalization and LRelu activation function.

[0082] 3. Construct a joint gradient map containing complementary texture information of the source images through the dual-channel gradient aggregation module in the fusion network; combine the joint gradient map with the gradient of the constructed preliminary fusion image Figure 1 The initial fusion image is then input into the discriminator of the fusion network to enhance the texture details of the preliminary fusion image. The specific steps include:

[0083] The joint gradient map is constructed through the dual-channel gradient aggregation module: first, the gradients of the infrared and visible light channels are calculated separately through the Sobel operator; then the gradient information of the infrared and visible light channels is integrated through the dual-channel aggregation strategy to obtain the joint gradient map I containing the complementary texture information of the source image. grad , this process is expressed as:

[0084]

[0085] in, represents gradient operation, specifically, the sobel operator calculates the gradient; abs(·) represents absolute value operation, and max(·) represents the maximum selection strategy at the pixel level;

[0086] The joint gradient map with complementary texture features and the gradient of the preliminary fusion image are Figure 1 The discriminator is a four-layer network structure. The first three layers use 3x3 convolution kernels, and the last layer is a fully connected layer, and the sigmoid function is used to output the discrimination probability. The discriminator is used to calculate the similarity of texture information between the joint gradient map and the gradient map of the preliminary fusion image, so as to further characterize the texture details of the foreground target and background semantic information in the preliminary fusion image.

[0087] 4. In the generator of the fusion network, the content loss between the generated preliminary fusion image and the two types of source images is calculated respectively; in the discriminator of the fusion network, the adversarial loss between the joint gradient map and the preliminary fusion image gradient map is calculated; the two losses are used together to train the fusion network to optimize the image fusion effect; when the number of training rounds reaches the preset number, the network training is completed and the final fusion image is generated. The specific steps include:

[0088] (1) The content loss of the generator includes pixel intensity loss, similarity loss and generator adversarial loss, which are: L int ,L sim ,L adv . Among them, the pixel intensity loss L int The definition is as follows:

[0089]

[0090] Where H and W are the height and width of the image respectively, ||·||1 represents the L1 norm, Represents the maximum selection at the image pixel level. The intensity loss integrates the pixel intensity distributions of the infrared and visible images through a maximum selection strategy to constrain the pixel intensity distribution of the fused image.

[0091] The role of similarity loss is to maintain the overall structure of the fused image consistent with the source image. Similarity loss L sim The definition is as follows:

[0092]

[0093] Among them, SSIM(I1,I2) represents the structural similarity measure between images I1 and I2. ir and I vi Infrared and visible light images respectively.

[0094] The adversarial loss of the generator aims to make the fused image retain more texture information in the source image. adv The definition is as follows:

[0095]

[0096] Where c is the probability label with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images.

[0097] Constructing the generator content loss L for the image fusion task G , and its loss function is defined as follows:

[0098] L G =λ1L int +λ2L SSIM +λ3L adv

[0099] Among them, λ1,λ2,λ3 are the weights controlling each loss.

[0100] (2) The adversarial loss function of the discriminator is used to calculate the similarity between the fused image gradient map and the joint gradient map, so as to reduce the difference in texture features between the preliminary fused image and the joint gradient map. Dadv Defined as:

[0101]

[0102] Where a is the label of the gradient map of the fused image, with a value of 0; b is the label of the joint gradient map, with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images.

[0103] The total number of training rounds is preset to E. When the number of training rounds reaches E, the network training is considered completed and the final fused image is generated.

[0104] Figure 6 The fusion results of the test image of the present invention are shown. It can be seen from the results in the figure that the present invention can well preserve the target blocked by the high-brightness information, and at the same time, the meaningful high-brightness information in the image is also preserved, thereby enriching the semantic information of the fused image.

[0105] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image fusion method based on difference significance aggregation and joint gradient constraint, characterized in that: The following steps are involved: S1. Input the infrared and visible light source images into a fusion network; the fusion network includes a generator, a dual-channel gradient aggregation module and a discriminator, wherein the generator includes a saliency difference perception aggregation subnetwork and a feature fusion subnetwork; the infrared and visible light source images are first input into a saliency difference perception aggregation subnetwork, which includes a multi-scale stereo attention module and a regional aggregation strategy, and the attention information of the saliency regions of the infrared and visible light source images are respectively extracted through the multi-scale stereo attention module, and the difference joint saliency map is obtained through integration through the regional aggregation strategy; The S1 is specifically: S11, inputting two types of source images, infrared and visible light images, into the saliency difference perception aggregation sub-network respectively; using the multi-scale stereo attention module to perceive the spatial and channel attention information of the salient regions of the two types of source images at different scales, and obtaining the saliency feature map of the corresponding source image; S12, integrate the significant feature map through the regional aggregation strategy to obtain the difference joint significant map I mask ; S2, inputting the generated difference joint saliency map and the two types of source images into a feature fusion subnetwork; wherein the feature fusion subnetwork includes a gradient residual module of a chain structure with added jump connections, and the gradient residual modules are sequentially connected to extract the features of the difference joint saliency map and the two types of source images of infrared and visible light, and then reconstructing the features through convolution to obtain a preliminary fused image; The S2 is specifically: S21. In the feature fusion subnetwork, the difference is combined with the saliency map I mask Multiply the infrared and visible light images element by element to obtain the salient target area map I t and significant background area Figure I d ; to I t and infrared images, I d The shallow features of the image and visible light image are extracted using 3x3 convolutional layers in two parallel feature extraction branches respectively; in each parallel feature extraction branch, the extracted shallow features are connected in series in the channel dimension, and the gradient residual module with a chain structure of jump connection is added in the feature fusion sub-network to further extract deep features; S22, the shallow features after concatenation are further extracted through sequentially connected gradient residual modules, and the adjacent gradient residual modules are connected through jump connections to avoid the loss of context information; in the gradient residual module, the mainstream is densely connected by two 3x3 convolutions to extract deep features, and the residual stream is extracted and retains the fine-grained features of the source image through gradient operation using the Sobel gradient operator. The deep features and fine-grained features obtained by the mainstream and residual streams are respectively integrated after splicing in the channel dimension; the deep features and fine-grained features extracted from the gradient residual module are reconstructed through 4 3x3 convolution layers. Except for the last layer that uses the Tanh activation function to obtain a preliminary fused image, the remaining layers all use BN normalization and LRelu activation function; S3, constructing a joint gradient map containing complementary texture information of the source images through a dual-channel gradient aggregation module in the fusion network; inputting the joint gradient map together with the gradient map of the preliminary fusion image into the discriminator of the fusion network to enhance the texture details of the preliminary fusion image; The S3 is specifically: S31, construct a joint gradient map through a dual-channel gradient aggregation module: first, the gradients of the infrared and visible light channels are calculated separately through the Sobel operator; then the gradient information of the infrared and visible light channels is integrated through a dual-channel aggregation strategy to obtain a joint gradient map I containing complementary texture information of the source image grad , this process is expressed as: in, represents gradient operation, specifically, the sobel operator calculates the gradient; abs(·) represents absolute value operation, and max(·) represents the maximum selection strategy at the pixel level; S32, inputting the joint gradient map with complementary texture features and the gradient map of the preliminary fused image into the discriminator together, the discriminator is a four-layer network structure, the first three layers use 3x3 convolution kernels, the last layer is a fully connected layer, and the sigmoid function is used to output the discrimination probability; the discriminator is used to calculate the similarity of texture information between the joint gradient map and the gradient map of the preliminary fused image, so as to further characterize the texture details of the foreground target and background semantic information in the preliminary fused image; S4. In the generator of the fusion network, the content loss between the preliminary fusion image and the two types of source images, infrared and visible light, is calculated respectively; in the discriminator of the fusion network, the adversarial loss between the joint gradient map and the gradient map of the preliminary fusion image is calculated; the content loss and the adversarial loss are used together to train the fusion network to optimize the effect of image fusion; when the number of training rounds reaches the preset number, the network training is completed and the final fusion image is generated.

2. The image fusion method based on difference significance aggregation and joint gradient constraint according to claim 1, characterized in that: The S4 is specifically: S41, respectively calculating the content loss between the preliminary fusion image and the two types of source images in the generator; wherein the content loss at least includes pixel intensity loss, structural similarity loss and adversarial loss of the generator; S42. In the discriminator, the adversarial loss between the joint gradient map and the preliminary fused image gradient map is calculated; the network is trained together with the content loss to optimize the image fusion effect; the total number of training rounds is preset, and when the number of training rounds reaches the preset total number of training rounds, the network training is considered to be completed, and the final fused image is generated.

3. The image fusion method based on difference significance aggregation and joint gradient constraint according to claim 1, characterized in that: The working mechanism of the multi-scale stereo attention module in S11 is: In this module, the input image is extracted through 3x3 depth-wise separable convolution to obtain the general feature F0 of each channel; F0 is divided into different scale features through 3x3 depth-wise separable convolution with different dilation rates; the attention weights of different branches of multi-scale features in space and channel dimensions are calculated through stereo attention; the calculation formula of stereo attention is expressed as: Among them, v represents the stereo attention weight, c and s represent channel attention and spatial attention respectively. represents element-wise multiplication; The stereo attention weights are processed by the softmax function to obtain the final stereo attention weights, which integrate features of different scales. The fused feature F is adjusted through a 1x1 convolution channel and then summed with the input image to obtain the saliency feature map of the corresponding source image.

4. The image fusion method based on difference significance aggregation and joint gradient constraint according to claim 1, characterized in that: The S12 is specifically: The corresponding saliency feature maps of infrared and visible light images are integrated through regional aggregation strategy: Integrate all salient regions of the source image by pixel-wise maximum selection: Among them, I joint (i, j) represents the pixel value at position (i, j) of the salient feature map after integration, I ir_mask (i, j) and I vi_mask (i, j) are the pixel values ​​at position (i, j) of the salient feature maps of the infrared and visible light images respectively; The integrated salient feature map is binarized by the OTSU threshold segmentation method to obtain the difference joint saliency map, which is expressed as: I mask =Threshold OTSU (I joint ) Among them, Threshold OTSU (·) indicates the OTSU threshold segmentation method.

5. The image fusion method based on difference significance aggregation and joint gradient constraint according to claim 2, characterized in that: The S41 is specifically: The content loss of the generator includes pixel intensity loss, similarity loss and generator adversarial loss, which are expressed as: L int ,L sim ,L adv ; in, Pixel intensity loss L int The definition is expressed as: Where H and W are the height and width of the image, respectively, ||·||1 represents the L1 norm, and max(·) represents the maximum selection at the image pixel level. The intensity loss integrates the pixel intensity distributions of the infrared and visible images through the maximum selection strategy to constrain the pixel intensity distribution of the fused image. Similarity loss L sim The definition is expressed as: Among them, SSIM(I f ,I ir ) represents image I f and I ir The structural similarity measure between ir and I vi Infrared and visible light images respectively; Generator’s adversarial loss L adv The definition is expressed as: Where c is the probability label with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images; Constructing the generator content loss L for the image fusion task G , which is defined as: L G =λ1L int +λ2L SSIM +λ3L adv Among them, λ1,λ2,λ3 are the weights controlling each loss.

6. The image fusion method based on difference significance aggregation and joint gradient constraint according to claim 2, characterized in that: The S42 is specifically: The adversarial loss function of the discriminator is used to calculate the similarity between the fused image gradient map and the joint gradient map, so as to reduce the difference in texture features between the preliminary fused image and the joint gradient map. Dadv Defined as: Where a is the label of the gradient map of the fused image, with a value of 0; b is the label of the joint gradient map, with a value of 1; D(·) represents the output of the discriminator; n represents the nth image, and N represents the total number of images; The total number of training rounds is preset to E. When the number of training rounds reaches E, the network training is considered completed and the final fused image is generated.

Images