Infrared and visible light image fusion system and method

Abstract

The present invention discloses an infrared and visible light image fusion system and method, wherein the image fusion system comprises a multi-scale feature extraction unit for respectively extracting features of infrared images and visible light images, a feature screening unit and an image reconstruction unit located after the multi-scale feature extraction unit. Firstly, the multi-scale features of the infrared image and the visible light image are respectively extracted by the multi-scale feature extraction unit, and then the multi-scale features are feature screened by a dual attention mechanism, and then the image reconstruction unit completes the image fusion, thereby obtaining an infrared and visible light fusion image with clear details.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and relates to infrared and visible light image fusion technology, and in particular to an infrared and visible light image fusion system and method based on dense hole convolution and double attention mechanism. Background Art

[0002] Image fusion is the organic integration of image information of the same scene obtained by two or more sensors. Fused images are more conducive to human and machine visual perception and other image processing tasks, such as image enhancement, face recognition, feature extraction and target detection. Since image fusion technology can enrich image information and visual effects, it has been widely used in many fields such as computer vision, medical imaging, visual monitoring, target detection and tracking, and joystick measurement.

[0003] Infrared images are captured by infrared sensors that sense infrared bands and convert thermal radiation information into grayscale images; infrared images have strong contrast but low resolution. Visible light images are generated by visible light sensors capturing reflected light with rich spatial information. The generated visible light images have rich texture details but are easily disturbed by factors such as light intensity and dense fog. In view of the complementary characteristics of the two, infrared and visible light image fusion technology has become an important development direction in the research and application of image fusion technology.

[0004] Image fusion aims to integrate multi-source heterogeneous information to the greatest extent possible, fusing images from different sources into a single image with rich content. According to the image category, image fusion can be divided into multimodal image fusion, multi-exposure image fusion, multi-focus image fusion and multispectral image fusion. Infrared and visible light image fusion is an important branch in the field of multimodal image fusion. In recent years, related fusion algorithms based on traditional frameworks and learning have been proposed. From the analysis of the algorithm processing flow, infrared and visible light image fusion can basically be divided into three parts: heterogeneous feature information extraction, fusion rule construction and fused image reconstruction.

[0005] Typical traditional fusion methods measure the activity level of pixels or regions in the spatial domain or transform domain, and realize image fusion according to specific fusion rules. They can be roughly divided into sparse representation-based methods, multi-scale transformation-based methods, subspace-based methods, saliency-based methods, and hybrid methods. Traditional fusion methods have been explored for a long time. Although remarkable results have been achieved in fusion performance and image quality, there are still some difficulties or shortcomings that need to be solved. First, due to the diversity of infrared and visible light images, the manually designed fusion rules in existing traditional methods generally limit the fusion performance, and some fused images will have edge artifacts, feature overlap, or missing details [; secondly, in terms of fusion efficiency, some current fusion methods have the disadvantages of redundant fusion information and high computational intensity, which are not suitable for situations with high real-time requirements such as large-scale and high-frequency generation of fusion objects.

[0006] Recently, deep learning-based methods have made some progress in this field. They use powerful nonlinear neural units to fit the target image, and are increasingly attracting the attention of researchers in terms of fusion performance. Learning-based image fusion methods basically use convolutional neural networks, and the typical network structure is reflected in the encoder model and the generative adversarial model. The former extracts representative features of the source image through a filter with learnable parameters to complete feature-level image reconstruction; the latter uses the adversarial game between the generator and the adversary to learn the intensity, texture and contrast of the source image. Although these methods have achieved some good performance to a certain extent, they either extract single-scale features or extract multi-scale features by downsampling. Part of the source image information is lost in the fusion stage, thus affecting the image fusion quality. Moreover, in terms of fusion strategies, the rules adopted are such as taking the average, taking the maximum, or introducing an attention mechanism based on the softmax operator, but none of them have a wide range of adaptive image fusion capabilities. Summary of the invention

[0007] In view of the problems existing in traditional image fusion technology, such as poor fused image quality, low efficiency and lack of adaptability, the purpose of the present invention is to provide an infrared and visible light image fusion system and method, which is based on dense hole convolution and dual attention mechanism, can achieve adaptive refinement of features while acquiring dense multi-scale features, and finally reconstruct a high-quality fused image.

[0008] The inventive idea of ​​the present invention is: first, based on the principle of dense atrous convolution, feature extraction is performed to obtain dense multi-scale features; then, based on the dual attention mechanism, the aggregated pyramid features are adaptively refined; finally, the feature information transmitted above and the compensation information from the jump connection are combined to reconstruct and generate a high-quality fused image.

[0009] Based on the above-mentioned inventive concept, the infrared and visible light image fusion system provided by the present invention comprises a multi-scale feature extraction unit for extracting features of infrared images and visible light images respectively, a feature screening unit and an image reconstruction unit located after the multi-scale feature extraction unit;

[0010] The multi-scale feature extraction unit is used to perform multi-layer dilated convolution processing on the infrared image and the visible light image respectively to obtain multi-scale features corresponding to the infrared image and the visible light image;

[0011] The feature screening unit screens the multi-scale features extracted by the multi-scale feature extraction unit based on a double attention mechanism;

[0012] The image reconstruction unit is used to fuse the filtered infrared image features and visible light image features.

[0013] The multi-scale feature extraction unit includes an input layer and a multi-layer cascade of dense dilated convolution modules with increasing expansion rates; the input layer is used to extract original features from infrared images and visible light images respectively; the dense dilated convolution module is used to perform dilated convolution processing on the input features; the input of the latter layer of dense dilated convolution modules is obtained by splicing the outputs of all dense dilated convolution modules before the current layer and the original features extracted by the input layer in the channel dimension. The input layer can use a two-dimensional convolution layer to convert the infrared image / visible light image into the feature space. Multi-layer cascade of dense dilated convolution modules is used to aggregate multi-scale features. Compared with the pooling operation, the dilated convolution can increase the receptive field while keeping the resolution of the feature map unchanged. With the help of dense network DenseNets connection, the output of the dilated convolution layer with a small dilation rate is fed to the dilated convolution layer with a larger dilation rate. These feature maps from different sampling scales and the original feature maps obtained through the input layer are uniformly spliced ​​in the channel dimension and serve as the input of the next layer. This stacked dilated convolution method solves the problem of sparse sampling on pixels in a single dilated convolution, and can obtain a dense feature pyramid with diverse scales.

[0014] For the convenience of explanation, the output of the l-th layer dense hole convolution module is represented as Y l , which can be expressed as the following formula:

[0015]

[0016] In the formula, [Y l-1 ,Y l-2 ,…,Y0] represents the output of all the previous dense hole convolution modules and the input layer of the concatenated lth layer. It is worth noting that the output of the convolution layer as the input layer (whose expansion rate d = 1) is represented as Y0. Indicates that the convolution kernel size is K and the expansion rate is dl The l-th layer of the dilated convolution operator can be briefly described as:

[0017]

[0018] Where (·) represents a certain pixel information, x represents the input of the dense hole convolution module, and w[n] represents the nth parameter of the convolution window.

[0019] The receptive field R of the l-th layer dense hole convolution module can be expressed as:

[0020]

[0021] The upper and lower limits of the sampling scale of each dense hole convolution module of the present invention are:

[0022] Range=[R min ,R max ] (4)

[0023] In the formula, R min Take the receptive field of the first layer of dense hole convolutional network R max Take the sum of the receptive fields of the dense atrous convolution modules in each layer, that is,

[0024] Such a distribution of sampling scales can ensure that the network learns the feature distribution at multiple scales, making the fusion result more similar to the source image in terms of structure. In addition, although the multi-scale feature extraction units used for infrared image and visible light image feature extraction are consistent in network structure, the independent characteristics of infrared image and visible light image during training will result in different parameter distributions for the two corresponding multi-scale feature extraction units.

[0025] In the feature screening unit, many visual tasks use attention mechanisms to weight the areas of interest in the visual scene. The goal of infrared and visible light image fusion is to retain visual details and significant thermal radiation targets at the same time. The dense features obtained by the multi-scale feature extraction unit have different important information. In order to reasonably utilize the features of each scale, the present invention introduces a cascaded channel attention and spatial attention mechanism behind the multi-scale feature extraction unit. The feature screening unit is mainly composed of two consecutive ECA Net modules and a spatial submodule. The local cross-channel interaction strategy without dimensionality reduction and the method of adaptively selecting the size of the one-dimensional convolution kernel are adopted to realize the attention selection on the channel, and the input data compensation of the ECA Net module and the spatial submodule are connected to the output end for addition, specifically: the extracted multi-scale features are input into the ECA Net module, the output value of the ECA Net module is added to the input value of the ECA Net module and then input into the spatial submodule; the output value of the spatial submodule is added to the input value, and the result is used as the final output of the feature screening unit. The ECA Net includes a global average pooling layer, a one-dimensional convolution layer and a sigmoid function arranged in sequence; the spatial submodule includes a parallel maximum pooling layer and a mean pooling layer, a two-dimensional convolution layer, and a sigmoid function. At the same time, the input compensation of the two attention mechanisms is connected to the output end and added. Since there is information loss in the network flow process through the convolution attention mechanism, this approach can ensure the ability of the original attention branch feature screening on the one hand, and make up for the loss of significant features caused by the deepening of the network on the other hand.

[0026] The above-mentioned image reconstruction unit includes several continuous convolutional layers. In the present invention, several continuous convolutional layers with the same structure can be used. After each convolutional layer, a batch normalization module and a RELU function are sequentially set. The last convolutional layer outputs a fused image. Although the image fusion system provided by the present invention does not include upsampling and downsampling, the convolution and filtering effects will more or less lead to the loss of texture details of the source image. Therefore, in the present invention, the spliced ​​outputs of the dense hole convolution modules of the multi-scale feature extraction unit except the last layer of dense hole convolution modules (that is, the outputs of the current layer, all the dense hole convolution modules before the current layer, and the outputs of the input layer are spliced ​​in the channel dimension) are jump-connected to the symmetrical convolutional layer of the image reconstruction unit to transmit more useful potential texture feature information for reconstruction. Here, the symmetrical jump connection method is adopted, for example, the spliced ​​outputs of the second-to-last dense hole convolution module in the channel dimension are jump-connected to the second convolution layer of the image reconstruction unit, and the spliced ​​outputs of the third-to-last dense hole convolution module in the channel dimension are jump-connected to the third convolution layer of the image reconstruction unit, and so on. Considering the different characteristics of infrared images and visible light images, this paper designs a feedforward operator to perform propagation feature selection on the channel concatenation output of the dense dilated convolution module, which can be expressed as:

[0027]

[0028] In the formula, Φ represents the operation through element selection average method; and They represent the feature maps of the spliced ​​output of the channel dimension of the l-th layer dense hole convolution module in the visible light image and the infrared image, respectively. Represents the compensation features of the corresponding dense hole convolution module. These jump connections speed up the convergence of the framework and make the performance of image fusion more satisfactory.

[0029] The above-mentioned infrared and visible light image fusion system also includes a parameter optimization unit, which is used to obtain a loss value according to the loss function based on the fusion image of the infrared image and the visible light image obtained by the image reconstruction unit, and then optimize the system parameters based on the gradient descent algorithm. The loss function can determine the fitting direction of the network training. In the loss function provided by the present invention, two loss terms are set: pixel loss L pixel and the structural similarity loss L ssim The pixel loss term ensures that the fused image contains the intensity information of the source image, and the structural similarity loss term ensures the content consistency of the fused image and the source image at three levels: brightness, structure, and contrast. These two loss terms jointly restrict the learning direction of the network and make up for the shortcomings of a single loss term.

[0030] The loss function formula is as follows:

[0031] L total =λ1L pixel +L ssim (6)

[0032] In the formula, λ1 represents the equal coefficient, which is used to balance the weights of each loss, L pixel represents the pixel difference between the fused image and the source image, L ssim Indicates the structural similarity between the fused image and the source image.

[0033] L pixel Determined according to the following formula:

[0034]

[0035] In the formula, I ir ,I vis and O represent the input infrared image matrix, visible light image matrix and fused image matrix respectively, w ir Indicates the dynamic measurement I ir Adaptive image weights for the loss part, represents the square of the F norm;

[0036] Lssim Determined according to the following formula:

[0037] L ssim =w ir [1-ssim(O,I iR )]+(1-w ir )[1-ssim(O,I vis )] (8)

[0038] In the formula, ssim(·) calculates the structural similarity between images (see Wang Z, Bovik AC, Sheikh HR, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612).

[0039] When evaluating the weight of each source image, the primary consideration is to retain more information in images with higher weights. This is reflected in areas with less noise, areas with rich texture information, and areas with prominent pixel intensity in visible light images and infrared images. The noise and texture of an image are closely related to image quality, while pixel intensity represents the amount of information in the image. Therefore, the present invention comprehensively considers the amount of information and quality of the image, and dynamically calculates the information retention weight of each pair of infrared images and visible light images during the training process.

[0040] The present invention uses the two-dimensional entropy of an image to measure its information content, and the formula is as follows:

[0041]

[0042] Where (j, m) represents a binary group, which is composed of the gray value of the j-th pixel position in the image and the gray mean of m pixels in its neighborhood. The binary group is obtained on the image by sampling a rectangular window. jmrepresents the frequency ratio of (j, m) in all binary groups obtained in the image, size = C*H*W represents the total number of pixels, C represents the number of channels, H represents the height of the image, and W represents the width of the image; the window size can be set according to the image size, and the value of m is the number of all pixels in the rectangular window minus 1. Compared with the one-dimensional entropy, the two-dimensional entropy can highlight the grayscale information of the pixel position in the image and the comprehensive characteristics of the grayscale distribution in the pixel neighborhood. In terms of image quality, the present invention adopts the no-reference image quality evaluator BRISQUE (see Mittal A, Moorthy AK, Bovik A C. No-Reference Image Quality Assessment in the Spatial Domain [J]. Image Processing, IEEE Transactions on, 2012, 21 (12): p. 4695-4708), and divides the quality of the image into 0-100 scores, and the larger the number, the worse the image quality. In the present invention, IQA is used to represent image quality, which is obtained by taking the inverse of the BRISQUE score of the no-reference image quality assessor and expanding it to the same numerical order of the two-dimensional entropy value. ir It can be expressed as:

[0043]

[0044] Among them, ir and IQA ir They represent the information content and quality of the infrared image in the image pair, respectively. k is ir (infrared) or vis (visible light). λ2 is a given coefficient, which is 100 in the present invention. ir It represents the proportion of the information volume and quality of the infrared image in the image pair to the total information volume and quality of the image pair. The final generated weight w ir The loss term is used to constrain the network to learn close to the best fusion performance.

[0045] The present invention further provides the above-mentioned infrared and visible light image fusion method, which comprises the following steps:

[0046] S1 performs multi-layer dilated convolution processing on the infrared image and the visible light image to obtain the multi-scale features corresponding to the infrared image and the visible light image;

[0047] S2 screens the extracted multi-scale features based on the dual attention mechanism, thereby adjusting the feature weights of infrared images and visible light images respectively;

[0048] S3 fuses the infrared image features and the visible light image features screened out in step S2, and outputs a fused image of the infrared image and the visible light image.

[0049] In the above step S1, the process of processing the infrared image and the visible light image is the same, the purpose is to extract features based on different sampling scales, and specifically includes the following sub-steps:

[0050] S21 extracts features from the infrared image and the visible light image respectively to obtain original features of the infrared image and the visible light image;

[0051] S22 performs multi-layer dilated convolution processing on the extracted original features of the infrared image and the visible light image respectively to obtain corresponding multi-scale features; the input of the next layer of dilated convolution processing is obtained by splicing all the dilated convolution processing outputs before the current dilated convolution processing and the original features of the infrared image or the visible light image in the channel dimension.

[0052] In the above step S2, the purpose is to screen out features with important information from the extracted multi-scale features. The dual attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the input data processed based on the channel attention mechanism and the spatial attention mechanism are compensated and added to the output data.

[0053] In the above step S3, the purpose is to fuse the filtered features to obtain a fused image of the infrared image and the visible light image features. The specific implementation method is: the infrared image features and visible light image features filtered out in step S2 are subjected to multiple continuous convolution processes to obtain a fused image. In order to improve the image fusion effect and reduce the loss of image texture details, in the further fusion process, the splicing output jump connection of the results of the hole convolution processing at all levels except the last layer of hole convolution processing in step S1 is introduced into the symmetrical convolution processing of this step.

[0054] When training the infrared and visible light image fusion system given above, in addition to constructing a training set using the collected infrared images and the corresponding visible light images and inputting them into the infrared and visible light image fusion system to obtain a fused image according to the above steps, the following steps are further included:

[0055] S4 obtains a loss value according to the following loss function based on the fused image obtained in step S3, and optimizes the system parameters based on a gradient descent algorithm;

[0056] L total =λ1L pixel +L ssim (6)

[0057] In the formula, λ1 represents the equal coefficient, which is used to balance the weights of each loss, L pixel Indicates the pixel difference between the fused image and the source image (including infrared image and visible light image), L ssimIndicates the structural similarity between the fused image and the source images (infrared image and visible light image).

[0058] During the training process, steps S1-S4 are repeated until the system converges, that is, the image fusion system performance converges to a good and stable level (ie, the loss value tends to be stable).

[0059] In order to improve the training effect of the system, the collected infrared images and the corresponding visible light images can be preprocessed by cropping (such as top, bottom, left, right, center, etc.), vertical rotation and / or horizontal flipping to achieve data augmentation. The preprocessed images are converted to grayscale space, and the image data obtained is used to construct the training set.

[0060] The above L pixel and L ssim The specific determination method is as described above.

[0061] The present invention can use conventional optimization algorithms disclosed in the art to optimize the network parameters of the image fusion system. The present invention uses the Adam algorithm to find the optimal solution based on the gradient descent algorithm.

[0062] In summary, the present invention draws on the method of multi-scale context information aggregation and designs a densely connected hole convolution module to extract multi-scale features. The feature extractor obtains a wide range of receptive fields and finally generates dense pyramidal features without losing image resolution. In the feature fusion stage, the present invention abandons the cumbersome manual fusion rules and adopts the attention cascade module of the ECA Net module and the spatial submodule to learn and obtain the important weights of each scale feature from a large amount of data, and then the enhanced or suppressed features are spliced ​​in the same spatial dimension for image reconstruction. The dual attention module selectively emphasizes or weakens the local or global information of the feature map, which promotes the information flow of the network to a certain extent and accelerates the convergence speed. In terms of detail retention, the present invention draws more than two jump connections from the feature extraction module and passes them to the corresponding image reconstruction convolution layer through the information feedforward operator to compensate for information loss. On the other hand, considering the different amounts of information contained in different infrared and visible light images, the present invention designs an information retention strategy for hybrid image second-order entropy and image parameter-free quality assessment to generate loss weights to guide the network fitting direction.

[0063] The infrared and visible light image fusion system and method provided by the present invention have the following beneficial effects:

[0064] (1) The present invention supports image input of any scale. First, dense multi-scale features are extracted through a dense dilated convolution module mainly composed of multiple layers of cascades. Then, feature screening is performed through an ECA Net module and a spatial submodule based on a dual attention mechanism. Finally, an image reconstruction unit composed of several convolutional layers completes image reconstruction, thereby obtaining an infrared and visible light fusion image with clear details.

[0065] (2) The present invention conducts qualitative and quantitative experiments on two public infrared and visible light image datasets, TNO and VIFB, and has excellent performance on the corresponding evaluation indicators.

[0066] (3) The present invention combines a dual-branch feature extraction network with a jump connection of an additional feedforward operator. The input of the feature extraction network is an infrared and visible image pair. The jump connection branch selectively retains the information transmission amount of the infrared and visible light images through the feedforward operator, making the network easier to converge.

[0067] (4) The present invention provides a new information retention strategy for comprehensively evaluating image quality. The direction of image fusion can be dynamically determined during the training phase, thereby making the image fusion system more generalizable and applicable to both daytime and nighttime scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of the structure of the infrared and visible light image fusion system based on dense dilated convolution and dual attention mechanism provided in this embodiment.

[0069] Figure 2 Schematic diagram of the feature screening unit structure provided in this embodiment; K′ represents the adaptive convolution window size, F eP (·),F MP,AP (·) represents the global pooling operation and the parallel maximum pooling and average pooling operations, σ represents the sigmoid operation, and Add represents the addition operation.

[0070] Figure 3 A fused image obtained by fusing the infrared image and the visible light image of Kaptein_1123 on the TNO dataset by the image fusion system provided for MSVD, GTF, FPDE, TIF, FusionGAN, GANMcC, U2Fusion and Example 2; it contains background markers and salient target markers (people), and the following is an enlarged view of the background marker area.

[0071] Figure 4A fused image obtained by fusing the infrared image and the visible light image of Street on the TNO dataset by the image fusion system provided for MSVD, GTF, FPDE, TIF, FusionGAN, GANMcC, U2Fusion and Example 2; it contains background markers and salient target markers (people), and the following is an enlarged view of the background marker area.

[0072] Figure 5 The image fusion systems provided for MSVD, GTF, FPDE, TIF, FusionGAN, GANMcC, U2Fusion and Example 2 are tested on 21 pairs of images randomly selected from the TNO dataset; wherein, the triangle in the box-shaped wireframe represents the mean, and the horizontal line represents the median.

[0073] Figure 6 The image fusion system provided by MSVD, GTF, FPDE, TIF, FusionGAN, GANMcC, U2Fusion and Example 2 fuses the infrared image and visible light image of the running sequence in the VIFB data set to obtain a fused image; it contains background markers and significant target markers (people), and the following is an enlarged image of the background marker area.

[0074] Figure 7 The image fusion system provided for MSVD, GTF, FPDE, TIF, FusionGAN, GANMcC, U2Fusion and Example 2 is tested on 21 pairs of images on the VIFB dataset; wherein, the triangle in the box-shaped wireframe represents the mean, and the horizontal line represents the median.

[0075] Figure 8 The fused images are obtained by fusing the infrared images and visible light images of two pairs of soldier images with jeeps and balls on the TNO dataset by Ours w / o D, Ours w / o A and Ours image fusion systems respectively; they contain background markers and salient target markers (people in the car window), and the enlarged image of the background marker area is shown below.

[0076] Fig. 9 The video sequence face tracking results based on the VOT-RGBT2019 benchmark; the three rows from top to bottom are infrared, RGB, and fused images respectively; each column shows the tracking status of the sequence frame, where the results of the ground truth, infrared, RGB, and fused images are marked with boxes of different sizes, and a better view can be obtained by zooming in.

[0077] Fusion Assessment Strategy

[0078] Image evaluation is a complex issue, and different image types require different image evaluation strategies. So far, there is no special evaluation index applied to the field of infrared and visible light image fusion. Therefore, in order to objectively reflect the quality of the fused image, the present invention evaluates the image quality through qualitative and quantitative evaluation strategies. Among them, the qualitative evaluation method adopts the same state-of-the-art method to give a quality description of the fused image from the perspective of human visual perception. The quantitative evaluation method uses four indicators including structural similarity SSIM, visual information fidelity VIF, sum of difference correlation SCD and spatial frequency SF to comprehensively evaluate the image quality from multiple perspectives such as image content, visual quality and frequency domain. The larger the values ​​of these four indicators, the better the performance of the fused image.

[0079] (1) The visual information fidelity (VIF) metric measures the information fidelity of the fused image, which is consistent with the human visual system. For details, see (see Y. Han, Y. Cai, Y. Cao, and X. Xu, A new image fusion performance metric based on visual information fidelity, Inf. Fusion, vol. 14, no. 2, pp. 127–135, Apr. 2013).

[0080] (2) SSIM consists of the multiplication of brightness, structure and contrast, and calculates the structural similarity between images by simulating image loss and distortion:

[0081]

[0082] Among them, x, f correspond to the image blocks of images X, F in the sliding window respectively, σ xf and σ x ,σ f are the covariance and variance in the corresponding sliding window, u x ,u f are the means in the corresponding sliding windows, respectively, while C1, C2, and C3 are parameters. The determination method can be found in Wang Z, Bovik AC, Sheikh HR, et al. Image quality assessment: from error visibility to structural similarity [J]. IEEE transactions on image processing, 2004, 13 (4): 600-612. Here, X represents the infrared and visible light images, F represents the fused image, and the average of the two structural similarities between the infrared image and the visible light image and the fused image is taken as the evaluation index SSIM.

[0083] (3) SCD measures the amount of information transferred from the source image to the fused image and is a measure of correlation.

[0084] SCD is defined as:

[0085] SCD=r(D1,S1)+r(D2,S2) (12)

[0086] Among them, r(·) represents the correlation coefficient, which can be calculated by the following formula:

[0087]

[0088] Among them, S1 and S2 represent infrared images and visible light images respectively, D1 and D2 represent fused images, and D k (i,j) and S k (i,j) represents any pixel in the image. Respectively represent D k ,S k The average pixel value.

[0089] (4) SF measures the gradient distribution of an image, which can effectively reveal the texture and details of the image and is defined as follows:

[0090]

[0091] in,

[0092] F(i,j) represents the fused image, M and N represent the height and width of the fused image respectively. DETAILED DESCRIPTION

[0093] The technical solutions of the embodiments of the present invention are clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the present invention.

[0094] Example 1

[0095] The infrared and visible light image fusion system based on dense hole convolution and dual attention mechanism provided in this embodiment is as follows: Figure 1 As shown, it includes two multi-scale feature extraction units for extracting features of infrared images and visible light images respectively, a feature screening unit and an image reconstruction unit located after the multi-scale feature extraction units;

[0096] A multi-scale feature extraction unit, used to perform multi-layer dilated convolution processing on the infrared image and the visible light image to obtain multi-scale features corresponding to the infrared image and the visible light image;

[0097] The feature screening unit screens the multi-scale features extracted by the multi-scale feature extraction unit based on the dual attention mechanism;

[0098] The image reconstruction unit is used to fuse the filtered infrared image features with the visible light image features.

[0099] like Figure 1 As shown, the multi-scale feature extraction unit includes an input layer and three-layer cascaded dense dilated convolution modules with increasing dilation rates. The input layer is used to extract original features of infrared images and visible light images respectively; the dense dilated convolution module is used to perform dilated convolution processing on the input features. In this embodiment, the two-dimensional convolution layer a is used as the input layer, and the convolution kernel size of the two-dimensional convolution layer a is 3*3. The dilation rates of the three-layer cascaded dense dilated convolution modules are d=1, 2, and 3 respectively, and the convolution kernel size is fixed and applicable to K=3. The number of convolutions in each layer is 16, and the padding value is the same as the dilation rate of the corresponding convolution layer to ensure that the resolution of the output feature map remains unchanged. The input of the dense dilated convolution module of the next layer is obtained by splicing the outputs of all dense dilated convolution modules before the current layer and the output of the convolution layer a in the channel dimension.

[0100] In this embodiment, the output of the l-th layer dense hole convolution module is represented as Y l , which can be expressed as the following formula:

[0101]

[0102] In the formula, [Y l-1 ,Y l-2 ,…,Y0] represents the output of all the previous dense hole convolution modules and convolution layer a of the concatenated lth layer. It is worth noting that the output of convolution layer a is represented by Y0. Indicates that the convolution kernel size is K and the expansion rate is d l The l-th layer of the dilated convolution operator can be briefly described as:

[0103]

[0104] Where (·) represents a certain pixel information, x represents the input of the dense hole convolution module, and w[n] represents the nth parameter of the convolution window.

[0105] The receptive field R of the l-th layer dense hole convolution module can be expressed as:

[0106]

[0107] The upper and lower limits of the sampling scale of the feature extractor of the present invention are:

[0108]

[0109] In the formula, R min Take the receptive field of the first layer of dense hole convolutional network R max Take the sum of the receptive fields of the dense atrous convolution modules in each layer, that is,

[0110] The distribution of the sampling scale range [3,15] can ensure that the multi-scale feature extraction unit learns the feature distribution at multiple scales, making the fusion result more similar to the source image in terms of structure. In addition, although the dual-branch feature extraction circuit is consistent in network structure, different parameter distributions will be obtained during training based on the independent characteristics of infrared and visible light images.

[0111] like Figure 1 and Figure 2 As shown, the feature screening unit includes a continuous ECA Net module and a spatial submodule. The ECA Net module includes a global average pooling layer, a one-dimensional convolution layer b and a sigmoid function set in sequence. The spatial submodule includes a parallel maximum pooling layer and a mean pooling layer, a two-dimensional convolution layer c, and a sigmoid function set in sequence. The global average pooling layer is used to perform a global pooling operation on the input feature data ( Figure 2 F GP (·) indicates that the convolution kernel size of the one-dimensional convolution layer b is 3. The feature data output by the one-dimensional convolution layer b and the input data of the ECA Net module are subjected to sigmoid operation ( Figure 2 The input data of the spatial submodule is obtained by adding the input data of the ECA Net module to the input data of the spatial submodule. The maximum pooling layer and the mean pooling layer of the spatial submodule are set in parallel to perform maximum pooling and average pooling operations on the feature data output by the ECA Net module in sequence ( Figure 2 F MP,AP The outputs of the maximum pooling layer and the mean pooling layer are concatenated in the channel dimension to form the input of the two-dimensional convolution layer c. The output of the two-dimensional convolution layer c and the output data of the ECA Net module are subjected to a sigmoid operation ( Figure 2 The output data of the spatial submodule is obtained by adding the output data of the ECA Net module to the output data of the spatial submodule.

[0112] like Figure 1As shown, the image reconstruction unit includes five consecutive two-dimensional convolutional layers (convolutional layer d to convolutional layer h, the convolution kernel size of each convolutional layer is 3*3, and the padding and step size of each layer are set to 1. A batch normalization module and a RELU function are sequentially set after each convolutional layer. In this embodiment, two jump connections of the channel splicing outputs of the first two layers of dense hole convolution modules are introduced into the second and third layers of the feature reconstruction part of the image reconstruction unit (wherein the channel splicing output of the first layer of dense hole convolution module (i.e., the third-to-last dense hole convolution module) is jump-connected to the convolution layer f of the image reconstruction unit; the channel splicing output of the second layer of dense hole convolution module (i.e., the second-to-last dense hole convolution module) is jump-connected to the convolution layer e of the image reconstruction unit) to transmit more useful potential texture feature information for reconstruction. Taking into account the different characteristics of infrared images and visible light images, this embodiment designs a feedforward operator to select propagation features, which can be expressed as:

[0113]

[0114] In the formula, and They represent the feature map pixel matrices of the channel concatenation output of the l-th layer dense dilated convolution module in the visible light image and infrared image, respectively. represents the compensation feature of the corresponding dense hole convolution module, Φ represents the operation by element selection averaging, which refers to and The pixel values ​​at corresponding positions are added and then averaged.

[0115] Example 2

[0116] The training data used in this embodiment comes from representative scenes in the FLIR dataset, totaling 221 pairs (see Xu H, Ma J, Jiang J, et al. U2Fusion: A Unified Unsupervised Image Fusion Network [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020).

[0117] This embodiment provides a training method for the infrared and visible light image fusion system based on dense dilated convolution and dual attention mechanism in Embodiment 1. The training set used is obtained according to the following steps:

[0118] S01 pre-processes the image by cropping (such as top, bottom, left, right and center cropping), vertical rotation and horizontal flipping to achieve data augmentation and obtain 4420 pairs.

[0119] S02 converts the preprocessed image into a grayscale space, and the image is converted into a grayscale image;

[0120] S03 crops the image to a size of 128*128 using conventional nearest neighbor interpolation method.

[0121] Then use the obtained training set to train the image fusion system according to the following steps:

[0122] S1 inputs the infrared image and visible light image in the training set into two multi-scale feature extraction units respectively, and uses the two multi-scale feature extraction units to perform multi-scale feature extraction on the infrared image and the visible light image respectively.

[0123] This step includes the following sub-steps:

[0124] S11 uses the input layer to extract features of the input infrared image / visible light image to obtain original features of the infrared image / visible light image;

[0125] S12 sends the extracted original features to three-layer cascaded dense dilated convolution modules with increasing expansion rates for multi-scale feature extraction; the input of the dense dilated convolution module of the next layer is obtained by splicing the output of all dense dilated convolution modules before the current layer and the output of the input layer in the channel dimension; the output of all dense dilated convolution modules and the output channels of the input layer are spliced ​​to obtain the multi-scale features of the final output.

[0126] S2 uses the ECA Net module and the spatial submodule of the feature screening unit to screen the multi-scale features extracted from the infrared image / visible light image in turn, thereby adjusting the feature weights of the infrared image and the visible light image respectively.

[0127] This step is specifically as follows: the extracted multi-scale features are input into the ECA Net module, the output value of the ECA Net module is added to the input value of the ECA Net module and then input into the spatial submodule; the output value of the spatial submodule is added to the input value, and the result is used as the final output of the feature screening unit.

[0128] S3 uses an image reconstruction unit to fuse the infrared image features and the visible light image features screened out in step S2, and outputs a fused image of the infrared image and the visible light image.

[0129] This step is specifically as follows: the infrared image features and visible light image features screened out in step S2 are input to five consecutive convolutional layers of the image reconstruction unit for processing. During the processing, the channel splicing output of the second-layer dense hole convolution module (i.e., the second-to-last dense hole convolution module) is jump-connected to the convolution layer e of the image reconstruction unit, and is spliced ​​with the output of the convolution layer d in the channel dimension to obtain the input data of the convolution layer e; the channel splicing output of the first-layer dense hole convolution module (i.e., the third-to-last dense hole convolution module) is jump-connected to the convolution layer f of the image reconstruction unit, and is spliced ​​with the output of the convolution layer e in the channel dimension to obtain the input data of the convolution layer f.

[0130] S4 obtains a loss value according to the following loss function based on the fused image obtained in step S3, and optimizes the system parameters based on a gradient descent algorithm;

[0131] L total =λ1L pixel +L ssim (6)

[0132] In the formula, λ1 represents the equal coefficient, L pixel represents the pixel difference between the fused image and the source image, L ssim Indicates the structural similarity between the fused image and the source image.

[0133] L pixel Determined according to the following formula:

[0134]

[0135] In the formula, I ir ,I viS and O represent the input infrared image matrix, visible light image matrix and fused image matrix respectively, w ir Indicates the dynamic measurement I ir The weight of the loss part, Represents the square of the F-norm.

[0136] L ssim Determined according to the following formula:

[0137] L ssim =w ir [1-ssim(O,I iR )]+(1-w ir )[1-ssim(O,I vis )] (8)

[0138] In the formula, ssim(·) calculates the structural similarity between images (see Wang Z, Bovik AC, Sheikh HR, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612).

[0139] Adaptive image weight w iP It can be expressed as:

[0140]

[0141] Among them, iR and IQA iR They represent the information content and quality of the infrared image in the image pair respectively, k is ir or vis, λ2 is a given coefficient, and its value is 100.

[0142] This embodiment uses the two-dimensional entropy of an image to measure its information content, and the formula is as follows:

[0143]

[0144] Where (j, m) represents a binary group, which is composed of the gray value of the j-th pixel position in the image and the gray mean of m pixels in its neighborhood. The binary group is obtained on the image by sampling a rectangular window. jm represents the frequency ratio of (j, m) in all binary groups obtained in the image, size = C*H*W represents the total number of pixels, C represents the number of channels, H represents the height of the image, and W represents the width of the image; in this embodiment, the window size is 3*3, and m = 8. In terms of image quality, this embodiment adopts the no-reference image quality assessor BRISQUE (see Mittal A, Moorthy AK, Bovik A C. No-Reference Image Quality Assessment in the Spatial Domain [J]. Image Processing, IEEE Transactions on, 2012, 21 (12): p. 4695-4708), and divides the quality of the image into scores of 0-100, and the larger the number, the worse the image quality. In this embodiment, IQA represents the image quality, which is obtained by taking the inverse of the no-reference image quality assessor BRISQUE score and expanding it to the same numerical order of the two-dimensional entropy value.

[0145] In this step, the Adam algorithm is used to optimize the parameters of the image fusion system, and the learning rate lr is 1×10 -4 , there are two parameters α and β in this algorithm, and their values ​​are α=25 and β=0.1. The specific optimization process is as follows:

[0146] 1.

[0147] 2.

[0148] Among them, the first item represents the gradient obtained by derivation of the loss function with respect to the parameter θ of the image fusion system, the second item represents the optimization update of the parameter θ using the Adam algorithm and the gradient, and lr represents the learning rate.

[0149] Steps S1-S4 are repeated until the system converges, that is, the image fusion system performance converges to a good and stable level (ie, the loss value tends to be stable).

[0150] Application Example 1

[0151] Input Kaptein_1123 and Street in the TNO dataset (see Toet A. TNO Image fusion dataset. Figshare. data, 2014. https: / / figshare.com / articles / TN Image FusionDataset / 1008029) into the image fusion system trained in Example 2 and operate according to the following steps:

[0152] S1 inputs the infrared image and the visible light image to be fused into two multi-scale feature extraction units respectively, and uses the two multi-scale feature extraction units to perform multi-scale feature extraction on the infrared image and the visible light image respectively;

[0153] S2 uses a feature screening unit to screen the multi-scale features extracted from the infrared image and the visible light image through the feature screening unit;

[0154] S3 uses an image reconstruction unit to fuse the infrared image features and the visible light image features screened out in step S2, and outputs a fused image of the infrared image and the visible light image.

[0155] The above steps S1-S3 are the same as the steps S1-S3 in the image fusion system training method in Example 2, and will not be repeated here.

[0156] The final fused image is as follows Figure 3 and Figure 4 shown.

[0157] In order to comprehensively evaluate the fusion image effect obtained by the present invention, the infrared images and visible light images of Kaptein_1123 and Street in the TNO dataset are fused by using the multi-scale variation method MSVD, the subspace decomposition method FPDE, the salient target method TIF, the total variation method GTF, and three deep learning-based methods FusionGAN, GANMcC and U2Fusion. The fused images are shown in Figure 2. Figure 3 and Figure 4 shown.

[0158] from Figure 3 It can be seen that the overall style of the fused images generated by FusionGAN and GTF is darker, with more infrared image background, especially in the enlarged marked area, the brightness information in the corresponding area of ​​the visible light image is lost. The fused images generated by the methods MSVD, FPDE, U2Fusion and GANMcC introduce different degrees of noise, which can be seen in the enlarged marked area (it is best to zoom in and observe), and such noise ultimately affects the image quality. In comparison, the fused image generated by the TIF method has a high degree of retention of significant targets (i.e., people), but there are black artifacts around the people, and there is a small bright spot between the two trees in the enlarged marked area. The fused image obtained by the image fusion method of the present invention has less noise and richer details while ensuring the contrast of the infrared target, and the marked area can be seen when zoomed in and observed.

[0159] from Figure 4 It can be seen that the methods FusionGAN and GTF still retain infrared significant targets, but the brightness of the billboard in the enlarged marked area is too low. The method FPDE has more obvious noise in the enlarged marked area, and the methods MSVD and GANMcC present a foggy visual effect on the billboard due to the noise, while U2Fusion is more focused on the background color of the visible light image as a whole, and the infrared significant information incorporated is insufficient. The red frame area of ​​the method TIF has higher brightness and contrast, but a heavier black artifact is introduced in the lower left corner of the billboard, which loses image quality. In contrast, the fused image obtained by the image fusion method of the present invention lacks brightness in the enlarged marked area, but has moderate contrast, less noise, and better overall performance. In order to more clearly reveal the superiority of the present invention. The relevant experimental data are shown in Table 1.

[0160] This application example further uses the image fusion system trained in Example 2 to perform infrared image and visible light image fusion tests on 21 randomly selected images from the TNO dataset, and uses four indicators, SSIM, SCD, SF, and VIF, for quantitative analysis. The test results are shown in Tables 1 and Figure 5 shown.

[0161] In order to comprehensively evaluate the fusion image effect obtained by the present invention, MSVD (see Naidu, V.: Image fusion technique using multi-resolution singular value decomposition. Defense Science Journal 61(5), 479–484(2011)), FPDE based on subspace decomposition (see Bavirisetti, DP, Xiao, G., Liu, G.: Multi-sensor image fusion based on fourth order partial differential equations. In: 2017 20th International Conference on Information Fusion (Fusion), pp. 1–9(2017). DOI 10.23919 / ICIF.2017.8009719), and TIF based on salient target (see Bavirisetti, DP, Dhuli, R.: Two-scale image fusion of visible and infrared images using saliency detection. Infrared Physics & Technology 76,52–64(2016)), total variation based method GTF (see Ma, J., Chen, C., Li, C., Huang, J.: Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fusion 31,100–109(2016)), and three deep learning based methods FusionGAN (see Ma, J., Wei, Y., Liang, P., Chang, L., Jiang, J.: Fusiongan: Agenerative adversarial network for infrared and visible image fusion. Information Fusion 48,11–26(2019)), GANMcC (see Ma, J., Zhang, H., Shao, Z., Liang, P., Xu, H.:Ganmcc:Agenerative adversarial network withmulticlassification constraints for infrared and visible image fusion.IEEE Transactions on Instrumentation and Measurement PP(99),1–1(2020)) and U2Fusion (see Xu,H.,Ma,J.,Jiang,J.,Guo,X.,Ling,H.:U2fusion:Aunified unsupervised image fusion network.IEEE Transactions on Pattern Analysis and Machine Intelligence(2020)) were used to test the fusion of infrared and visible images on 21 pairs of images selected from the TNO dataset. The test results are shown in Tables 1 and . Figure 5 shown.

[0162] Table 1 Quantitative comparison results of 21 pairs of images on two public datasets

[0163]

[0164] From Table 1 and Figure 5 From the statistical results, it can be seen that the present invention has the highest average value and the highest median value in the three indicators of SSIM, SF and VIF. The evaluation of SSIM shows that the fused image obtained by the present invention well continues the structural information of the input source image, the evaluation of SF shows that the fused image of the present invention has rich spatial domain information, and the evaluation of VIF shows that the fused image of the present invention has excellent performance in details and is more in line with human visual perception. For the indicator SCD, although the method of the present invention failed to achieve the best performance, it can be seen from the qualitative comparison that the fused image of the present invention can simultaneously have the infrared thermal radiation information and the texture detail information of the visible light image. These results show that the image fusion system and method provided by the present invention can efficiently and complementary utilize the advantageous information of infrared and visible light images, while having the least noise, to obtain a more robust fused image.

[0165] Application Example 2

[0166] In order to verify that the image fusion system and method provided by the present invention have high-quality generalization capabilities, this application example conducted the same qualitative and quantitative analysis experiments on the challenging VIFB dataset. The VIFB dataset is the first benchmark in the field of visible and infrared image fusion, proposed by Zhang et al. (see Zhang X, Ye P, Xiao G. VIFB: AVisible and Infrared Image Fusion Benchmark [C] / / IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2020). The dataset integrates 21 image pairs, covering a wide range of environments and working backgrounds. Since the visible light images in the VIFB dataset are color images, the image fusion system trained by Example 2 is used in the present invention to execute the following fusion strategy: convert the visible light image to the YCbCr color space and extract the Y channel information and fuse it with the infrared image according to steps S1-S3 given in Application Example 1. Finally, the obtained fused image is converted into an RGB image by combining it with the Cb and Cr channels of the visible light image through inverse transformation. The test results are shown in Table 1. Figure 6 and Figure 7 shown.

[0167] In order to comprehensively evaluate the fusion image effect obtained by the present invention, the fusion test of infrared image and visible light image is performed on 21 image pairs selected from the VIFB dataset using MSVD, FPDE based on subspace decomposition, TIF based on salient target, GTF based on total variation, and three deep learning-based methods FusionGAN, GANMcC and U2Fusion. The test results are shown in Table 1. Figure 6 and Figure 7 shown.

[0168] For qualitative comparison results, this application example uses Figure 6 The image sequence shown is discussed. Compared with the visible light image, it can be observed that the results of the seven methods in the prior art have different degrees of color distortion, which is largely due to the inclusion of infrared information. However, when zooming in on the marked area, it can be clearly seen that the method provided by the present invention is able to retain more color and texture details present in the visible image, followed by the competing method FPDE with a more obvious sense of blur. The target area represented by the highlight target, the human target in the method provided by the present invention effectively combines important infrared radiation information and has higher characteristic texture information. The quantitative comparison results of the VIFB dataset are shown in Tables 1 and Figure 7As shown. Compared with the overall performance of the TNO dataset, the image fusion system provided by the present invention only obtained two optimal average values ​​(SSIM and VIF) and one suboptimal value (SF). The evaluation results of SSIM and VIF show that this method can most effectively retain the structural information of the source image and obtain the best visual effect. SF calculates the sum of gradient values, which has a more objective evaluation for noise-free images, while noisy images will have higher values ​​under the same content. This can be proved by the visual results of TIF and U2Fusion. Obviously, there is more noise in the enlarged marked area, which will have a positive impact on the SF value. Overall, the image fusion system provided by the present invention achieves the best performance on the challenging VIFB dataset and has strong generalization ability.

[0169] Application Example 3

[0170] This application example is based on the data set and method provided in Application Example 1.

[0171] In the image fusion system provided in Example 1, the dense multi-scale feature extractor is composed of dense hole convolution modules with successively increasing dilation rates, and the adaptive weight adjustment of the feature map is completed by a feature screening unit based on a dual attention mechanism. In order to verify the impact of these two functions on the fusion performance, this application example performs an ablation experiment on the TNO dataset.

[0172] In order to conduct ablation experiments, this application example constructs two comparison systems (Ours w / o D and Ours w / o A). Ours w / o D lacks a multi-scale feature extraction unit (D) relative to the image fusion system (Ours) provided in Example 1. Ours w / o A lacks a feature screening unit (A) relative to the image fusion system (Ours) provided in Example 1. w / o means (without).

[0173] The two contrast systems are trained according to the training method provided in Example 2, and the two pairs of soldier images with jeeps and balls in the TNO dataset are tested using the trained two contrast systems and the image fusion system trained in Example 2. The test results are shown in Table 2 and Figure 8 shown.

[0174] Table 2 Quantitative comparison results of three image fusion systems on two pairs of images on the TNO dataset

[0175]

[0176] As shown in Table 2, the presence of D will significantly improve the SSIM and SCD indicators of the fused image, and A plays a positive role in affecting SF and VIF. When there is a lack of dense multi-scale feature extraction units, the image fusion system cannot utilize multi-scale feature information, resulting in insufficient pixel-level contrast of the branch and leaf information captured by the fused image, such as Figure 8 As shown. This visual effect also acts on the clouds in the sky and the people in the car windows, making them difficult to separate from the surrounding background. In addition, the fused image acquires less texture details without the use of module A, which focuses on the differential adjustment of feature information. It can be easily observed from the enlarged area in the red box that, without module A, the sky cloud information transmitted by the infrared image is basically invisible. Another fact that cannot be ignored is that SCD has the highest score without module A. This may be because when certain features are enhanced or suppressed, some relevant information of the source image is lost. When modules D and A are present in the network, Figure 8 It can be clearly seen that the fused image obtained by the image fusion system provided by the present invention inherits the most consistent contrast and details. In summary, the above experiments verify the effectiveness of the multi-scale feature extraction unit and feature screening unit in the image fusion system provided by the present invention.

[0177] Application Example 4

[0178] Image fusion serves higher-level visual tasks. In order to test the feasibility of further application of the image fusion system and method provided by the present invention, the present invention implements a visual object tracking experiment (VOT).

[0179] In the experiment of the present invention, the open source SiamRPN composed of backbone (AlexNet structure) AlexNet of the SenseTime team was selected as the short-term single target tracking experiment on the latest VOT-RGBT2019 dataset. VOT-RGBT2019 is the RGB and thermal infrared short-term tracking sub-challenge newly introduced by the committee in 2019, which contains a total of 60 short video sequences. This application example randomly selected 30 video sequences and used the image fusion system trained in Example 2 to execute the following fusion strategy: convert the visible light image to the YCbCr color space and extract the Y channel information and the infrared image according to steps S1-S3 given in Application Example 1. Finally, the obtained fused image is converted into an RGB image by inverse transformation combined with the Cb and Cr channels of the visible light image, and the experimental results of some sequence frames with faces as the target are provided, such as Fig. 9 shown.

[0180] In terms of quantitative indicator evaluation, we use the three indicators selected by the SiamRPN algorithm (Accuracy, Robustness, and EAO (expected average overlap rate)). The evaluation results are shown in Table 3. In the table, Accuracy counts the Intersection-over-Union (overlap rate) of the bounding box (object tracking box) predicted in each frame and gt (true value), which is used to evaluate the accuracy of the tracker in tracking the target in a single test sequence. The larger the value, the higher the accuracy. Robustness evaluates stability. The larger the value, the worse the stability, which is positively correlated with the number of times each video sequence is lost. EAO is the abbreviation of expected average overlap rate, which is the expected value of non-reset overlap of the tracker on a short-term image sequence.

[0181] In the VOT-RGBT2019 benchmark experiment results, it can be seen that the fusion results have achieved the highest values ​​in all three indicators. Obviously, the fusion results have the main information of visible light and infrared images, and have more obvious features, so the tracker can detect the target more easily and with higher accuracy. This shows that the image fusion system provided by the present invention also has considerable advantages in fusing dynamic infrared targets and avoiding distortion.

[0182] Table 3 Image fusion test results of 30 random video sequences on the VOT-RGBT2019 dataset

[0183]

[0184] In addition, the operating efficiency of the image fusion system provided by the present invention is explained based on the time complexity of the TNO dataset in Application Example 1 and the VIFB dataset in Application Example 2 (the analysis results are shown in Table 4).

[0185] Table 4 Time complexity analysis results of 21 random image pairs on TNO and VIFB datasets

[0186]

[0187] In the present invention, all visible images are converted to grayscale space and then processed, so all the methods involved in the comparison only consider the time consumption in the fusion process, without the computational overhead of source image channel separation or fusion. In general, the traditional methods have higher time costs due to the complex arithmetic operations performed on the CPU. Among the deep learning-based methods, compared with U2Fusion and the image fusion method provided by the present invention, GANS uses a relatively simple convolutional network to achieve higher computational efficiency. In order to further evaluate the difference between the two, when the input image has a fixed size (such as 128*128), the image fusion method provided by the present invention has lower computational complexity than U2Fusion. Specifically, the method provided by the present invention has a total of 164k training parameters and 2.9G triggers, while the corresponding values ​​of U2Fusion are 659k and 216G, respectively. In summary, although the method provided by the present invention failed to beat U2Fusion in the VIFB dataset, it achieved the best performance in the TNO dataset, which still proves that the image fusion method provided by the present invention has considerable efficiency.

[0188] In summary, the infrared and visible light image fusion system based on dense hole convolution and dual attention mechanism provided by the present invention is an end-to-end infrared and visible light image fusion network DFDAFuse. First, our fusion network expands the convolution block through densely connected hole convolution modules to capture the pyramidal features of the source image without losing the resolution of the feature map. Then, the aggregated multi-scale feature information is adaptively refined through the sequence ECA Net and the spatial channel attention module. During the training process, the designed image information retention strategy can dynamically measure the weight of the source image and guide the fitting direction of the network. The qualitative and quantitative experiments conducted by the present invention on two public datasets TNO and VIFB show that the method can effectively fuse the prominent information in infrared and visible light images and generate the fused image with the highest comprehensive performance. The fused image combines the main information of infrared and visible light images, and can effectively extract and utilize the advantageous information of the source image. For example, the salient targets in the above-mentioned application examples cannot be seen in the visible light source image, but can be clearly shown in the fused image, indicating that the fused image includes infrared information. Similarly, the fused image also contains information that is not in the infrared source image but in the visible light image. In addition, ablation experiments prove that each module of the network of the present invention has a positive impact on the fused image, and extended experiments further show that the present invention still has application potential in other visual tasks and has good generalization.

[0189] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. An infrared and visible light image fusion system, It is characterized in that It includes a multi-scale feature extraction unit for extracting features of infrared images and visible light images respectively, a feature screening unit and an image reconstruction unit located after the multi-scale feature extraction unit; The multi-scale feature extraction unit is used to perform multi-layer dilated convolution processing on the infrared image and the visible light image respectively to obtain multi-scale features corresponding to the infrared image and the visible light image; the multi-scale feature extraction unit includes an input layer and a multi-layer cascaded dense dilated convolution module with increasing dilation rates; the input layer is used to extract original features from the infrared image and the visible light image respectively; the dense dilated convolution module is used to perform dilated convolution processing on the input features; the input of the dense dilated convolution module of the next layer is obtained by splicing the outputs of all the dense dilated convolution modules before the current layer and the original features extracted by the input layer in the channel dimension; The feature screening unit screens the multi-scale features extracted by the multi-scale feature extraction unit based on a double attention mechanism; The feature screening unit includes a continuous ECA Net module and a spatial submodule; the input data of the ECA Net module and the spatial submodule are connected to the output end for compensation and addition; The image reconstruction unit is used to fuse the filtered infrared image features and visible light image features; the image reconstruction unit includes a plurality of continuous convolutional layers; the channel splicing output jump connections of the dense void convolution modules of the multi-scale feature extraction unit except the last layer of dense void convolution module are introduced into the symmetrical convolution layer of the image reconstruction unit.

2. The infrared and visible light image fusion system according to claim 1, It is characterized in that It also includes a parameter optimization unit, which is used to obtain a loss value according to the following loss function based on the fused image of the infrared image and the visible light image obtained by the image reconstruction unit, and then optimize the system parameters based on the gradient descent algorithm: The loss function formula is as follows: L total =λ 1 L pixel +L ssim (6) In the formula, λ 1 Represents equal coefficients, used to balance the weights of each loss, L pixel represents the pixel difference between the fused image and the source image, L ssim Indicates the structural similarity between the fused image and the source image.

3. The infrared and visible light image fusion system according to claim 2, It is characterized in that L pixel Determined according to the following formula: In the formula, I ir ,I vis and O represent the input infrared image matrix, visible light image matrix and fused image matrix respectively, w ir Indicates the dynamic measurement of I ir Adaptive image weights for the loss part, represents the square of the F norm; L ssim Determined according to the following formula: L ssim =w ir [1-yes(O,I ir )]+(1-w ir )[1-yes(O,I vis )] (8) Where ssim(·) calculates the structural similarity between images; Adaptive image weight w ir It can be expressed as: Among them, ir and IQA ir Respectively represent the information content and quality of the infrared image in the image pair, k is ir or vis, λ 2 For a given coefficient, the final generated weight w ir Influencing the loss term to constrain the network to approach the best fusion performance learning; The two-dimensional entropy of an image is used to measure its information content. The formula is as follows: Where P jm It represents the gray value of pixel j in the image and the gray mean of m pixels in its neighborhood. size = C*H*W represents the total number of pixels, C represents the number of channels, H represents the height of the image, and W represents the width of the image.

4. A method for fusion of infrared and visible light images, It is characterized in that The infrared and visible light image fusion system of claim 1 is used in accordance with the following steps: S1 performs multi-layer dilated convolution processing on the infrared image and the visible light image to obtain the multi-scale features corresponding to the infrared image and the visible light image; S2 screens the extracted multi-scale features based on the dual attention mechanism, thereby adjusting the feature weights of infrared images and visible light images respectively; S3 fuses the infrared image features and the visible light image features screened out in step S2, and outputs a fused image of the infrared image and the visible light image.

5. The infrared and visible light image fusion method according to claim 4, It is characterized in that In step S1, the input of the next layer of dilated convolution processing is obtained by concatenating all dilated convolution processing outputs before the current dilated convolution processing and the original features of the infrared image or visible light image in the channel dimension.

6. The infrared and visible light image fusion method according to claim 4, It is characterized in that The dual attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the input data processed based on the channel attention mechanism and the spatial attention mechanism are compensated and added to the output data.

7. The infrared and visible light image fusion method according to claim 5, It is characterized in that In step S3, the infrared image features and visible light image features screened out in step S2 are subjected to multiple consecutive convolution processes to obtain a fused image; during the fusion process, the splicing output jump connection of the results of the hole convolution processes at all levels except the last layer of hole convolution process in step S1 is introduced into the symmetrical convolution process of this step.

Images