A method, system and device for feature-level fusion of infrared and visible light images

Abstract

The present application relates to the technical field of computer vision, and proposes a method, system and device for infrared and visible light feature level fusion; the feature extraction of infrared and visible light images is made to have a focus by combining Haar transformation and feature extraction of a convolutional neural network; a cross attention mechanism is designed to enable the model to process low-frequency common information and high-frequency unique information of visible light and infrared images respectively; a new dense connection form is proposed to combine an edge operator and a convolution block attention mechanism, thereby reducing the amount of calculation and improving the reusability of shallow features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a feature-level fusion method, system, and device for providing infrared and visible light fused images for downstream target detection tasks. Background Art

[0002] In modern society, the importance of images has become increasingly prominent across all industries. In current image processing and research, single-modality imaging often fails to provide comprehensive and accurate information. Image fusion technology addresses this issue by fusing image information from different sensors to produce a fused image that contains more information and is more suitable for downstream object detection tasks. Visible light images are formed by light reflected from objects, which can provide rich texture and color information, and have outstanding visual expression capabilities. It is the main way for human visual perception to obtain information. However, its imaging is very dependent on lighting conditions. When encountering situations with low light intensity such as wind, snow, heavy rain, and night, it will cause it to lose a lot of image information. Infrared images are formed through thermal radiation imaging, which is not easily affected by lighting conditions. It can still complete imaging under conditions where visible light is difficult to image. At the same time, it can still image some targets that are blocked by other objects. Although the detailed information of the image is lost, it has more prominent information expression for targets such as pedestrians and vehicles in bad weather. In the field of security, since we need to conduct 24-hour all-weather monitoring, the fusion of visible light images and infrared images can comprehensively utilize the detailed texture information of visible light and the significant information of infrared, which can improve the performance of downstream target detection tasks.

[0003] At present, image fusion can be divided into three types based on the different stages of fusion: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion starts directly from each pixel of the image and processes the pixels by weighted averaging, etc. However, this method requires too much data to be processed, which will lead to a long fusion time and affect the real-time performance of the system. Decision-level fusion is a high-level fusion. After analyzing, processing and making decisions on each source image separately, these decision results are fused to obtain the final fusion decision. This method is highly dependent on the accuracy of the decision results, so the accuracy is poor. Feature-level fusion refers to extracting features from the source image, making processing decisions at the extracted feature level, etc., and fusing different source images at the feature level, which is a higher level than the pixel level. This reduces the amount of calculation and the accuracy can still be maintained. However, there are still three problems. First, the focus of infrared images and visible light images in the fusion process is different. Due to the influence of the imaging method, the infrared image's details, such as texture and other high-frequency information, are not clearly expressed. Therefore, the fused image is more likely to focus on the low-frequency information such as the contour and shape of the infrared image, while the visible light image has better detail expression ability, so it is more necessary to focus on its details and other high-frequency information. Second, during the fusion process, infrared and visible light share low-frequency information, such as outlines and shapes, but their high-frequency signals are unique, such as visible light texture and infrared temperature. Therefore, improving the correlation of low-frequency information and the independence of high-frequency information is a challenge that needs to be addressed. Third, deepening the model will lead to the loss of shallow features and increase the computational complexity, which also needs to be addressed.

[0004] In summary, it is necessary to design an image fusion method, system and equipment that can fully consider the differences between infrared and visible light images to solve the problem. Summary of the Invention

[0005] To address the challenges of existing technologies, this paper proposes a method, system, and device for feature-level fusion of infrared and visible light images. By combining the Haar transform with feature extraction using a convolutional neural network, feature extraction for infrared and visible light images is prioritized. A Haar cross-attention mechanism is designed to enable the model to separately process low-frequency shared information and high-frequency unique information in visible and infrared images. A new dense connection scheme is proposed, combining edge operators with a convolutional block attention mechanism to reduce computational complexity and improve the reusability of shallow features.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] A method for feature-level fusion of infrared and visible light, comprising the following steps:

[0008] S1: Obtain infrared and visible light image datasets, preprocess the datasets, and obtain training sets;

[0009] S101: Obtain a sensor data set; use visible light and infrared cameras to shoot, create a data set D, divide the data set D into proportions, and obtain a training set, where the data set D contains both visible light and infrared data;

[0010] S2: Construct a feature-level infrared and visible light image fusion network model, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haar Image Enhancement, and the cross attention module Haar Cross Attention;

[0011] The shallow feature extraction module, Haar Feature Extraction, uses Haar transform to decompose the image and divide the visible light and infrared images into high- and low-frequency sub-bands. The shallow feature extraction module includes an infrared feature extraction module, which consists of two 3*3 convolutional layers and one 1*1 convolutional layer; and a visible light feature extraction module, which consists of two 3*3 convolutional layers and one 1*1 convolutional layer.

[0012] The infrared image enhancement module Haar Image Enhancement decomposes the image through Haar transform, enhances the infrared background contour information, fits it with the detail information of visible light, and outputs an infrared feature enhanced image.

[0013] The dense connection module SADense achieves the reuse of shallow features by densely connecting shallow features. It includes a Sobel operator, a 1*1 convolutional layer, two 3*3 convolutional layers and a convolutional block attention mechanism.

[0014] The Haar CrossAttention module includes a channel-wise attention module consisting of a spatial average pooling layer, two 1x1 convolutional layers, two ReLU activation functions, and a sigmoid activation function. It also includes a spatial attention module consisting of a channel-wise average pooling layer, a 7x7 convolutional layer, and a sigmoid activation function. It also includes a transposed attention module consisting of an image transposition layer and two 1x1 convolutional layers. Haar transform is used to process high- and low-frequency information separately.

[0015] S3: Set network parameters. The specific contents are as follows: set the initial learning rate Ir0, training rounds epochs, and batch size batch-size.

[0016] S4: Based on the neural network model reconstructed in steps 2 and 3, the training set in step 1 is used for training to obtain the optimal weight file of the network model.

[0017] S5:,The obtained weight file is applied to image fusion, and finally the obtained fused image is input into the downstream object detection task.

[0018] Furthermore, the specific contents of step S2 are as follows:

[0019] Step 201: Construct a visible light shallow feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency subband LL and three high-frequency subbands HL, LH, and HH. The visible light image focuses more on high-frequency information such as details. Perform a second-order Haar transform on the HH high-frequency subband. Perform a 3*3 convolution operation on the four subbands obtained from the second-order transform to make the model focus on learning and using detail information. Perform an inverse Haar transform on the obtained feature map to restore the image. A 1*1 convolution is connected in series to adjust the number of channels to obtain a visible light shallow feature extraction feature map F that focuses on high-frequency signals. vi1 , so that the next layer of network can process it.

[0020] Step 202: Construct an infrared shallow feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency sub-band LL and three high-frequency sub-bands HL, LH, and HH. Infrared images focus more on low-frequency information such as contour shape. Perform a second-order Haar transform on the low-frequency sub-band LL. Perform a 3*3 convolution operation on the four sub-bands obtained from the second-order transform to make the model focus on learning and using information such as contours. Perform an inverse Haar transform on the obtained feature map to restore the image. A 1*1 convolution is connected in series to adjust the number of channels to obtain an infrared shallow feature extraction feature map F that focuses on low-frequency signals. ir1 , so that the next layer of network can process it.

[0021] Step 203: Construct a dense connection module SADense, and combine the feature maps F obtained in steps 201 and 202 above vi1 and F ir1 Feature reuse on each branch, on each single branch, connected in each subsequent layer, the first layer feature map F of size W*H*C vi1 and F ir1 The weight information is added through 3*3 convolution respectively, as follows:

[0022]

[0023] Where m, n represent the spatial dimensions of the convolution kernel, K(m,n,c,k) represents the learned weight information, after the convolution operation, i, j represent the corresponding pixel coordinates on the feature map, 0≤i≤W, 0≤j≤H, and the weighted learned feature map F is obtained. vi2 and F ir2 , and then compared with the feature map F obtained in step 201 and step 202 above vi1 and Fir1 Splicing:

[0024]

[0025] As shown above, we get a new feature map F of size W*H*(C+C1) viconcat1 and F irconcat1 , convolution weighted to obtain feature map F vi3 and F ir3 , and then reuse the feature map F obtained in step 201 and step 202 vi1 and F ir1 , and the feature map F vi3 and F ir3 Splice as follows:

[0026]

[0027] The size of the visible light and infrared branches is W*H*(C+C3) feature map F viconcat2 and F irconcat2 Input them into the convolution block attention mechanism respectively to obtain the output feature map M vis and M irs , as follows:

[0028]

[0029] M vis =σ(Conv(F vic ))

[0030]

[0031] M irs =σ(Conv(F irc ))

[0032] Among them, F vic It's F viconcat2 After the middle layer output feature map of the convolution attention mechanism, F irc It's F irconcat2 After the middle layer output feature map of the convolution attention mechanism, it is added with nonlinear features through the activation function to obtain M vis and M irs ;σ represents the activation function, M vis and the feature map F obtained in step 201 vi1 The feature map F obtained by Sobel operator and convolution operation viSout Multiplication, M irs and the feature map F obtained in step 202 ir1 The feature map F obtained by Sobel operator and convolution operation irSout Multiply, as shown below:

[0033] F visout =Conv(Concat(Sobel x (F vi1 ),Sobel y (F vi1 )),K conv )

[0034] F vifinal =F SCout ·F sout

[0035] F irsout =Conv(Concat(Sobel x (F ir1 ),Sobel y (F ir1 )),K conv )

[0036] F irfinal =F SCout ·F sout

[0037] Among them Sobel x Represents the Sobel operator in the horizontal direction, Sobel y Represents the Sobel operator in the vertical direction, concatenated convolution, and finally obtains the above SADense dense block output feature map F vifinal and F irfinal .

[0038] Step 204: Infrared image enhancement module Haar Image Enhancement, decomposes the image through Haar transform, uses the low-frequency signal LL to enhance the infrared background contour information, and uses the high-frequency signals HL, LH, HH to enhance the detail information of the visible light. The enhancement module is the module described in step 205, and outputs an infrared feature enhanced image.

[0039] Step 205: Construct a Haar CrossAttention module. The infrared and visible light feature maps are first transformed through Haar transform to obtain low-frequency sub-bands and high-frequency sub-bands. The low-frequency signal emphasizes correlation, and the low-frequency sub-band feature maps are added. The high-frequency signal emphasizes irrelevance, and the high-frequency sub-band feature maps are subtracted. The resulting feature maps are input into the cross-attention module. The input feature maps are processed in parallel, focusing on the spatial and channel dimensions respectively. The channel dimension is first average pooled, as shown in the following formula:

[0040]

[0041] The resulting feature map is 1*1*C in size. Each channel contains the global channel information of its own space. After 1*1 convolution, the linear transformation of each channel is completed, and the channel dimension is reduced and redundant channel dimensions are removed, as shown in the following formula:

[0042]

[0043] Introduce nonlinear transformation and implement it through activation function Relu:

[0044]

[0045] Then, through a layer of 1*1 convolution operation and the introduction of nonlinear activation function, the dimension of the feature map is restored while the channels are reintegrated and reconstructed for subsequent Sigmoid normalization processing, compressing the weights to the range of [0,1].

[0046] In the spatial dimension, the channel dimension of the feature map is first average pooled, as shown in the following formula:

[0047]

[0048] A feature map of size w*h*1 is obtained. Each pixel in the space contains global channel information. The spatial dimension requires a larger receptive field to enhance the connection between features in different spaces. Therefore, a large kernel convolution of 7*7 is used, as shown in the following formula:

[0049]

[0050] Among them, in order to prevent the convolution process from crossing the boundary and still outputting the feature map of size w*h, padding is required. p represents the extra pixels added to the edge of the data. Here p is set to 3. The feature map processed in the spatial dimension is input into the Sigmoid activation function, the weight is normalized, and multiplied with the tensor in the channel dimension to generate a feature map that focuses on both the channel and spatial dimensions, as shown in the following formula:

[0051] Y(w,h,c)=A(1,1,c)×B(w,h,1)

[0052] Finally, we get the feature map F processed in both spatial and channel dimensions. SC , starting from its own global information, and then processing in parallel with the transposed attention module, by transposing the image itself, then splicing, and then convolving it in the form of appropriate weight ratio with F SC Splicing, completing weighting, and then performing inverse Haar transform to restore the image.

[0053] An infrared and visible light feature-level fusion system, comprising:

[0054] Data processing module: uses a camera to collect images and obtain the processed data.

[0055] Network building module: Construct a network model based on feature-level infrared and visible light image fusion, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haarimage enhancement, and the cross attention module Haar CrossAttention

[0056] Network training module: used to train on the training set according to the set network parameters, output the training weight file, and obtain the required optimal image fusion weight file.

[0057] Output module: Deploy the obtained image fusion model on the device, build a post-processing unit, and output the fused image to the downstream target detection model.

[0058] An infrared and visible light feature-level fusion device, comprising:

[0059] Machine executable instructions: a computer program storing the infrared and visible light feature-level fusion method, which is a computer-readable executable instruction;

[0060] Processor: A processor used to execute the computer program to implement the infrared and visible light feature-level fusion method.

[0061] A computer-readable storage medium: The computer-readable storage medium stores a computer program, and when the computer program is processed by a processor, the storage medium can store the infrared and visible light feature-level fusion method. The storage medium can be any physical storage medium that can include or store information, such as RAM, EMMC, ROM, SD, DVD, etc.

[0062] The beneficial effects of the present invention are mainly manifested in:

[0063] 1) The present invention designs a shallow feature extraction module that combines Haar transform to focus on high-frequency information such as texture in visible light feature extraction and on low-frequency information such as contour shape in infrared feature extraction, so that the fused image can better combine the advantages of both.

[0064] 2) The present invention constructs a new type of dense connection module, which enables the reuse of shallow features and uses edge operators to improve the expression of edge information, reducing the computational complexity of traditional dense connections and reducing the occurrence of the problem of shallow features not being able to be learned during model training.

[0065] 3) The present invention proposes a cross-attention mechanism, which cross-processes the common information of low-frequency signals and the unique information of high-frequency signals of visible light images and infrared images, thereby improving the relevance of the common low-frequency information and the irrelevance of the unique high-frequency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a diagram of the shallow feature extraction structure of the present invention;

[0067] Figure 2 It is the infrared image enhancement diagram of the present invention;

[0068] Figure 3 It is a cross attention structure diagram of the present invention;

[0069] Figure 4 This is a parallel structure diagram of the cross-attention space and channel dimensions of the present invention;

[0070] Figure 5 This is a diagram of the cross-attention transposition attention structure of the present invention;

[0071] Figure 6 It is the overall network structure diagram of the present invention;

[0072] Figure 7 Schematic diagram of the fused image of the present invention;

[0073] Figure 8 Schematic diagram of infrared image enhancement according to the present invention;

[0074] Figure 9 It is a system block diagram of the present invention;

[0075] Figure 10 It is a hardware structure diagram of the electronic equipment of the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described below with reference to the accompanying drawings.

[0077] Reference Figures 1 to 6 , a feature-level fusion method of infrared and visible light, comprising the following steps:

[0078] S1: Use visible light and infrared cameras to shoot, obtain infrared and visible light image datasets, preprocess the datasets, and obtain the training set.

[0079] S2: Construct a network model based on feature-level infrared and visible light image fusion, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haar Image Enhancement, and the cross attention module Haar Cross Attention.

[0080] The shallow feature extraction module Haar Feature Extraction uses Haar transform to decompose the image and divide the visible light and infrared images into high- and low-frequency sub-bands. The infrared feature extraction module consists of two 3*3 convolutional layers and one 1*1 convolutional layer. The visible light feature extraction module consists of two 3*3 convolutional layers and one 1*1 convolutional layer.

[0081] The dense connection module SADense achieves the reuse of shallow features by densely connecting shallow features. It includes a Sobel operator, a 1*1 convolutional layer, two 3*3 convolutional layers and a convolutional block attention mechanism.

[0082] The infrared image enhancement module Haar image enhancement decomposes the image through Haar transform, enhances the infrared background contour information, fits it with the detail information of visible light, and outputs an infrared feature enhanced image.

[0083] The Haar CrossAttention module includes a channel-wise attention module consisting of a spatial average pooling layer, two 1x1 convolutional layers, two ReLU activation functions, and a sigmoid activation function. It also includes a spatial attention module consisting of a channel-wise average pooling layer, a 7x7 convolutional layer, and a sigmoid activation function. It also includes a transposed attention module consisting of an image transposition layer and two 1x1 convolutional layers. Haar transform is used to process high- and low-frequency information separately.

[0084] The specific steps of S2 process are as follows:

[0085] Step 201: Construct a visible light shallow feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency subband LL and three high-frequency subbands HL, LH, and HH. The visible light image focuses more on high-frequency information such as details. Perform a second-order Haar transform on the HH high-frequency subband. Perform a 3*3 convolution operation on the four subbands obtained from the second-order transform to make the model focus on learning and using detail information. Perform an inverse Haar transform on the obtained feature map to restore the image. A 1*1 convolution is connected in series to adjust the number of channels to obtain a visible light shallow feature extraction feature map F that focuses on high-frequency signals. vi1 , so that the next layer of network can process it.

[0086] Step 202: Construct an infrared shallow feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency sub-band LL and three high-frequency sub-bands HL, LH, and HH. Infrared images focus more on low-frequency information such as contour shape. Perform a second-order Haar transform on the low-frequency sub-band. Perform a 3*3 convolution operation on the four sub-bands obtained from the second-order transform to make the model focus on learning and using information such as contours. Perform an inverse Haar transform on the obtained feature map to restore the image. A 1*1 convolution is connected in series. The number of channels is adjusted to obtain an infrared shallow feature extraction feature map F that focuses on low-frequency signals. ir1 , so that the next layer of network can process it.

[0087] The Haar transform expression is:

[0088] Low-Low (LL) sub-band:

[0089]

[0090] Low-High (LH) sub-band:

[0091]

[0092] High-Low (HL) sub-band:

[0093]

[0094] High-High (HH) sub-band:

[0095]

[0096] The convolutional layer parameter settings are shown in the following table:

[0097] Table WT convolution kernel parameter settings

[0098] Table Convolution Kernel Parameter Settings

[0099]

[0100] Step 203: Construct a dense connection module SADense, and combine the feature maps F obtained in steps 201 and 202 above vi1 and F ir1 Feature reuse on each branch, on each single branch, connected in each subsequent layer, the first layer feature map F of size W*H*C vi1 and F ir1 The weight information is added through 3*3 convolution respectively, as follows:

[0101]

[0102]

[0103] Where m, n represent the spatial dimensions of the convolution kernel, K(m,n,c,k) represents the learned weight information, after the convolution operation, i, j represent the corresponding pixel coordinates on the feature map, 0≤i≤W, 0≤j≤H, and the weighted learned feature map F is obtained. vi2 and F ir2 , and then compared with the feature map F obtained in step 201 and step 202 above vi1 and F ir1 Splicing:

[0104]

[0105] As shown above, we get a new feature map F of size W*H*(C+C1) viconcat1 and F irconcat1 , convolution weighted to obtain feature map F vi3 and F ir3 , and then reuse the feature map F obtained in step 201 and step 202 vi1 and F ir1 , and the feature map F vi3 and F ir3 Splice as follows:

[0106]

[0107] The size of the visible light and infrared branches is W*H*(C+C3) feature map F viconcat2 and F irconcat2 Input them into the convolution block attention mechanism respectively to obtain the output feature map M vis and M irs , as follows:

[0108]

[0109] M vis =σ(Conv(F vic ))

[0110]

[0111] M irs =σ(Conv(F irc ))

[0112] Where σ represents the activation function, M vis and the feature map F obtained in step 201 vi1 The feature map F obtained by Sobel operator and convolution operation viSout Multiplication, M irs and the feature map F obtained in step 202 ir1The feature map F obtained by Sobel operator and convolution operation irSout Multiply, as shown below:

[0113] F visout =Conv(Concat(Sobel x (F vi1 ),Sobel y (F vi1 )),K conv )

[0114] F vifinal =F SCout ·F sout

[0115] F irsout =Conv(Concat(Sobel x (F ir1 ),Sobel y (F ir1 )),K conv )

[0116] F irfinal =F SCout ·F sout

[0117] Among them Sobel x Represents the Sobel operator in the horizontal direction, Sobel y Represents the Sobel operator in the vertical direction, concatenated convolution, and finally obtains the above SADense dense block output feature map F vifinal and F irfinal .

[0118] Step 204: Infrared image enhancement module Haar Image Enhancement, decomposes the image through Haar transform, uses the low-frequency signal LL to enhance the infrared background contour information, and uses the high-frequency signals HL, LH, HH to enhance the detail information of the visible light, and then fits it. The enhancement method is the module described in step 205, and outputs the infrared feature enhanced image.

[0119] Step 205: Construct a Haar Cross Attention module. Infrared and visible light are first transformed through Haar transform to obtain low-frequency and high-frequency subbands. Low-frequency signals emphasize correlation, and the low-frequency subband feature maps are added. High-frequency signals emphasize irrelevance, and the high-frequency subband feature maps are subtracted. The resulting feature maps are input into the cross attention module. The input feature maps are processed in parallel, focusing on the spatial and channel dimensions respectively. The channel dimension is first average pooled, as shown in the following formula:

[0120]

[0121] The resulting feature map is 1*1*C in size. Each channel contains the global channel information of its own space. After 1*1 convolution, the linear transformation of each channel is completed, and the channel dimension is reduced and redundant channel dimensions are removed, as shown in the following formula:

[0122]

[0123] Introduce nonlinear transformation and implement it through activation function Relu:

[0124]

[0125] Then, through a layer of 1*1 convolution operation and the introduction of nonlinear activation function, the dimension of the feature map is restored while the channels are reintegrated and reconstructed for subsequent Sigmoid normalization processing, compressing the weights to the range of [0,1].

[0126] In the spatial dimension, the channel dimension of the feature map is first average pooled, as shown in the following formula:

[0127]

[0128] A feature map of size w*h*1 is obtained. Each pixel in the space contains global channel information. The spatial dimension requires a larger receptive field to enhance the connection between features in different spaces. Therefore, a large kernel convolution of 7*7 is used, as shown in the following formula:

[0129]

[0130] Among them, in order to prevent the convolution process from crossing the boundary and still outputting the feature map of size w*h, padding is required. Here, p is set to 3. The feature map processed in the spatial dimension is input into the Sigmoid activation function, the weight is normalized, and multiplied with the tensor in the channel dimension to generate a feature map that focuses on both the channel and spatial dimensions, as shown in the following formula:

[0131] Y(w,h,c)=A(1,1,c)×B(w,h,1)

[0132] Finally, we get the feature map F processed in both spatial and channel dimensions. SC , starting from its own global information, and then processing in parallel with the transposed attention module, by transposing the image itself, then splicing, and then convolving it with a weight ratio of 0.1 and F SC Splicing, completing weighting, and then performing inverse Haar transform to restore the image.

[0133] S3: Set network parameters: set the initial learning rate Ir0 = 0.001, training rounds epochs = 50, and batch size batch-size = 1.

[0134] S4: Based on the neural network model reconstructed in steps 2 and 3, the training set in step 1 is used for training to obtain the optimal weight file of the network model.

[0135] S5:,The obtained weight file is applied to image fusion, and finally the obtained fused image is input into the downstream object detection task.

[0136] Reference Figure 8 , an infrared and visible light feature-level fusion system, comprising:

[0137] Data processing module: uses a camera to collect images and obtain the processed data.

[0138] Network construction module: Build a network model based on feature-level infrared and visible light image fusion, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haarimage enhancement, and the cross-attention module Haar CrossAttention.

[0139] Network training module: used to train on the training set according to the set network parameters, output the training weight file, and obtain the required optimal image fusion weight file.

[0140] Output module: Deploy the obtained image fusion model on the device, build a post-processing unit, and output the fused image to the downstream target detection model.

[0141] Reference Figure 9 , an infrared and visible light feature-level fusion device, comprising:

[0142] Machine executable instructions: a computer program storing the infrared and visible light feature-level fusion method, which is a computer-readable executable instruction;

[0143] Processor: A processor used to execute the computer program to implement the infrared and visible light feature-level fusion method.

[0144] A computer-readable storage medium: The computer-readable storage medium stores a computer program, and when the computer program is processed by a processor, the storage medium can store the infrared and visible light feature-level fusion method. The storage medium can be any physical storage medium that can include or store information, such as RAM, EMMC, ROM, SD, DVD, etc.

[0145] The present invention conducts experiments on the MSRS dataset and compares it with existing image fusion algorithms. The comparison is made on six indicators, including SSIM (structural similarity between the fused image and the original image), AG (activity of the fused image), EN (the amount of information contained in the original image in the fused image), SD (contrast of the fused image), SCD (the degree of preservation of detail and texture information in the fused image), and Qabf (comprehensive performance), as shown in the following table:

[0146]

[0147] The present invention achieved the best results in the four indicators of AG, EN, SCD, and Qabf, indicating that the fused image of the present invention has achieved more effective integration of information such as detail texture, more comprehensive utilization of information of visible light and infrared images, and the fused image has a higher visual contrast.

[0148] In this specification, the schematic descriptions of the present invention do not necessarily refer to the same embodiment or example. Those skilled in the art may combine and combine different embodiments or examples described in this specification. In addition, the contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be considered as limited to the specific forms described in the implementation cases. The scope of protection of the present invention also includes equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for fusion of infrared and visible light feature levels, characterized in that: The following steps are involved: S1: Obtain infrared and visible light image datasets, preprocess the datasets, and obtain training sets; S2: Build a feature-level infrared and visible light image fusion network model, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haar Image Enhancement, and the cross attention module WT CrossAttention; The shallow feature extraction module Haar Feature Extraction uses Haar transform to decompose the image and divide the visible light and infrared images into high-frequency and low-frequency sub-bands; the shallow feature extraction module includes an infrared shallow feature extraction module and a visible light shallow feature extraction module, wherein the infrared feature extraction module includes two 3*3 convolutional layers and one 1*1 convolutional layer, and the visible light feature extraction module includes two 3*3 convolutional layers and one 1*1 convolutional layer; The dense connection module SADense achieves the reuse of shallow features by densely connecting shallow features. It includes a Sobel operator, a 1*1 convolutional layer, two 3*3 convolutional layers and a convolutional block attention mechanism. The infrared image enhancement module, Haar Image Enhancement, decomposes the image through Haar transform, enhances the infrared background contour information, fits it with the visible light detail information, and outputs an infrared feature-enhanced image; The cross attention module Haar CrossAttention includes a channel dimension attention module, which consists of a spatial average pooling layer, two 1*1 convolution layers, two relu activation functions and a sigmoid activation function; a spatial dimension attention module, which consists of a channel average pooling layer, a 7*7 convolution layer, and a sigmoid activation function; a transposition attention module, which consists of an image transposition layer and two 1*1 convolution layers; and uses wavelet transform to process high and low frequency information separately; S3: Set network parameters, including: initial learning rate Ir0, training rounds epochs, batch size batch-size; S4: Based on the feature-level infrared and visible light image fusion network model constructed in S2, the training set in S1 is used for training to obtain the optimal weight file of the network model; S5: Apply the obtained weight file to image fusion, and finally input the obtained fused image into the downstream object detection task.

2. The method for fusion of infrared and visible light feature levels according to claim 1, characterized in that: The specific contents of S1 are as follows: Step 101: Obtain a sensor data set; use visible light and infrared cameras to shoot and create a data set D.

3. The method for fusion of infrared and visible light feature levels according to claim 1, characterized in that: The specific contents of S2 are as follows: Step 201: Construct a visible light shallow feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency subband LL and three high-frequency subbands HL, LH, and HH. Perform a second-order Haar transform on the HH high-frequency subband. Perform a 3*3 convolution operation on the four subbands obtained from the second-order transform. Perform an inverse Haar transform on the obtained feature map to restore the image. A 1*1 convolution is connected in series. The number of channels is adjusted to obtain a visible light shallow feature extraction feature map F that focuses on high-frequency signals. vi1 , so that the next layer of network can process it; Step 202: Construct an infrared,feature extraction module. Use Haar transform on the visible light image to obtain a low-frequency sub-band LL and three high-frequency sub-bands HL, LH, and HH. Perform a second-order Haar transform on the low-frequency sub-band LL. Perform a 3*3 convolution operation on the four sub-bands obtained by the second-order transform. Perform an inverse Haar transform on the obtained feature map to restore the image. Then, perform a 1*1 convolution in series and adjust the number of channels to obtain an infrared shallow feature extraction feature map F that focuses on low-frequency signals. ir1 , and then the next layer of network processing; Step 203: Construct a dense connection module SADense, and combine the feature maps F obtained in steps 201 and 202 above vi1 and F ir1 Feature reuse on each branch, on each single branch, connected in each subsequent layer, the first layer feature map F of size W*H*C vi1 and F ir1 The weight information is added through 3*3 convolution respectively, as follows: Where m, n represent the spatial dimensions of the convolution kernel, K(m,n,c,k) represents the learned weight information, after the convolution operation, i, j represent the corresponding pixel coordinates on the feature map, 0≤i≤W, 0≤j≤H, and the weighted learned feature map F is obtained. vi2 and F ir2 , and then compared with the feature map F obtained in step 201 and step 202 above vi1 and F ir1 Splicing: As shown above, we get a new feature map F of size W*H*(C+C1) viconcat1 and F irconcat1 , convolution weighted to obtain feature map F vi3 and F ir3 , and then reuse the feature map F obtained in step 201 and step 202 vi1 and F ir1 , and the feature map F vi3 and F ir3 Splice as follows: The size of the visible light and infrared branches is W*H*(C+C3) feature map F viconcat2 and F irconcat2 Input them into the convolution block attention mechanism respectively to get the output feature map M vis and M irs , as follows: M vis =σ(Conv(F vic )) M irs =σ(Conv(F irc )) Among them, F vic It's F viconcat2 After the middle layer output feature map of the convolution attention mechanism, F irc It's F irconcat2 After the middle layer output feature map of the convolution attention mechanism, it is added with nonlinear features through the activation function to obtain M vis and M irs ;σ represents the activation function, M vis and the feature map F obtained in step 201 vi1 The feature map F obtained by Sobel operator and convolution operation viSout Multiplication, M irs and the feature map F obtained in step 202 ir1 The feature map F obtained by Sobel operator and convolution operation irSout Multiply, as shown below: F visout =Conv(Concat(Sobel x (F vi1 ),Sobel y (F vi1 )),K conv ) F vifinal =F SCout ·F sout F irsout =Conv(Concat(Sobel x (F ir1 ),Sobel y (F ir1 )),K conv ) F irfinal =F SCout ·F sout Among them Sobel x Represents the Sobel operator in the horizontal direction, Sobel y Represents the Sobel operator in the vertical direction, concatenated convolution, and finally obtains the above SADense dense block output feature map F vifinal and F irfinal ; Step 204: Infrared image enhancement module Haar Image Enhancement, which decomposes the image through Haar transform, uses the low-frequency signal LL to enhance the infrared background contour information, and the high-frequency signals HL, LH, HH to enhance the visible light detail information, and then fits the image. The enhancement method is the same as the module in step 205, and the infrared feature enhanced image is output; Step 205: Construct a Haar Cross Attention module. The infrared and visible light feature maps are first subjected to Haar transformation to obtain low-frequency sub-bands and high-frequency sub-bands. The low-frequency signal emphasizes correlation, and the low-frequency sub-band feature maps are added. The high-frequency signal emphasizes irrelevance, and the high-frequency sub-band feature maps are subtracted. The resulting feature maps are input into the cross attention module. The input feature maps are processed in parallel, focusing on the spatial and channel dimensions respectively. The channel dimension is first average pooled, as shown in the following formula: A feature map of size 1*1*C can be obtained. Each channel contains the global channel information of its own space. After 1*1 convolution, the linear transformation of each channel is completed, and the channel dimension is reduced and redundant channel dimensions are removed, as shown in the following formula: Introduce nonlinear transformation and implement it through activation function Relu: Then, through a layer of 1*1 convolution operation and the introduction of nonlinear activation function, the feature map dimension is restored while the channels are re-fused and reconstructed for subsequent Sigmoid normalization processing, compressing the weights to the range of [0,1]. In the spatial dimension, the channel dimension of the feature map is first average pooled, as shown in the following formula: A feature map of size w*h*1 is obtained. Each pixel in the space contains global channel information. The spatial dimension requires a larger receptive field to enhance the connection between features in different spaces. Therefore, a large kernel convolution of 7*7 is used, as shown in the following formula: Among them, in order to prevent the convolution process from crossing the boundary and still outputting the feature map of size w*h, padding is required. p represents the extra pixels added to the edge of the data. Here p is set to 3. The feature map processed in the spatial dimension is input into the Sigmoid activation function, the weight is normalized, and multiplied with the tensor in the channel dimension to generate a feature map that focuses on both the channel and spatial dimensions, as shown in the following formula: Y(w,h,c)=A(1,1,c)×B(w,h,1) Finally, we get the feature map F processed in both spatial and channel dimensions. SC , starting from its own global information, and then processing in parallel with the transposed attention module, by transposing the image itself, then splicing, and then convolving it in the form of appropriate weight ratio with F SC Splicing, completing weighting, and then performing inverse Haar transform to restore the image.

4. An infrared and visible light feature-level fusion system according to any one of claims 1 to 3, characterized in that: include: Data processing module: using a camera to collect images and obtain the processed data; Network construction module: Builds a network model based on feature-level infrared and visible light image fusion, including the shallow feature extraction module Haar Feature Extraction, the dense connection module SADense, the infrared image enhancement module Haar image enhancement, and the cross attention module Haar Cross Attention. Network training module: Used to train on the training set according to the set network parameters, output the training weight file, and obtain the required optimal image fusion weight file; Output module: Deploy the feature-level infrared and visible light image fusion network model on the device, build a post-processing unit, and output the fused image to the downstream target detection model.

5. An infrared and visible light feature-level fusion device, characterized in that: include: Machine executable instructions: a computer program storing a method for fusion of infrared and visible light feature levels as described in any one of claims 1 to 3, which is a computer-readable executable instruction; Processor: a processor configured to execute the computer program to implement the infrared and visible light feature-level fusion method according to any one of claims 1 to 3; A computer-readable storage medium: the computer-readable storage medium stores a computer program, and when the computer program is processed by a processor, the storage medium can store the infrared and visible light feature level fusion method as described in any one of claims 1-3.

Images