Low-illumination visible light and infrared image fusion method for automatic driving

Abstract

The invention discloses a low-illumination visible light and infrared image fusion method for automatic driving. A1, infrared image edge enhancement; a2, enhancing the brightness of the visible light image; a3, preliminarily fusing the infrared image and the visible light image; a4, carrying out secondary fusion on the edge-enhanced infrared image, the brightness-enhanced feature map and the preliminarily fused feature map; according to the method, fusion and brightness improvement are carried out in parallel, and information loss caused by incompatibility of an enhancement algorithm and a fusion algorithm is reduced. The effectiveness of the method is verified on an LLVIP data set and a TNO data set, and the result proves that compared with a classic fusion algorithm, the method can reserve more source image detail information in a low-light environment, can improve the contrast and brightness of the image, and obtains better results in subjective and objective evaluation.

Description

Technical Field

[0001] The present invention relates to image fusion technology in the field of autonomous driving, and in particular to a low-illumination visible light and infrared image fusion method for autonomous driving. Background Art

[0002] Autonomous driving is a major trend in future development. In the process of autonomous driving, a safe and comfortable driving experience is one of the important goals that people pursue. To achieve this goal, autonomous driving vehicles need to fully perceive the surrounding environment, road signs, pedestrians, etc. Good target detection technology can improve the perception of the surrounding environment of autonomous driving vehicles, and accurate detection of pedestrians can ensure the road safety of driving vehicles and pedestrians, and improve the driving and riding experience. At present, the detection of pedestrians on the road is based on the visible light vision system. In the daytime environment with good light, there will be better detection results, but in the low light environment with insufficient light, there will be problems such as missed detection or false detection. In order to improve the detection accuracy of autonomous driving vehicles for pedestrians in different environments and adapt to the detection task requirements in low light environments, it is possible to consider adding infrared sensors as a supplement. Infrared sensor imaging is less affected by ambient light. Even in low light environments such as night or foggy weather, the contour position and size of the target can be detected based on the thermal distribution of the object. However, single infrared imaging is also affected by ambient temperature. In summer, the temperature of pedestrians and the road temperature are very close. Using a single infrared sensor for detection will also reduce the accuracy of detection. Therefore, in order to improve the safety of autonomous driving and enhance the sensitivity of autonomous driving vehicles to pedestrians, detection can be considered based on the fusion of infrared and visible light images. To this end, the present invention proposes a new visible light and infrared image fusion method in a low-light environment.

[0003] Target detection based on multimodal fusion can be divided into three categories according to the different stages of fusion in detection: detection before classification - fusing multimodal detection results. This algorithm does not take advantage of the image fusion technology; fusion during the detection process - using the detection results to promote the adjustment of fusion network parameters. The seAFusion network uses high-level tasks to promote the adjustment of the fusion network, and Yang also fuses images during the detection process; fusion before detection - advanced multimodal image fusion, and then detection of the fused image. This algorithm allows the detection network to simultaneously use the information in the multimodal image to improve the detection accuracy. Compared with the algorithm that performs fusion and detection simultaneously, this algorithm has a simple algorithm structure and a faster calculation speed.

[0004] Structural reparameterization refers to a more complex model structure designed to extract richer or more accurate features during training. Its parameters can be equivalently converted to another set of parameters during the inference stage, thereby simplifying the resource and time overhead of the inference process while ensuring the same accuracy. In computer vision tasks, structural reparameterization has been applied in many classic models. RepVGG has modified the VGG model without a branch structure, converting the single-branch structure into a multi-branch structure during training, and converting it into a single-channel model during inference, so that the model has the advantages of high performance after training of the multi-branch model and fast speed during inference of the single-channel model. RepMLP adds parallel convolution and BN branches on the basis of MLP, which speeds up the operation and enhances the global modeling ability. These reparameterization methods have achieved obvious results in the inference stage, but the complex structure will incur huge additional training costs during training. After analyzing the key role of BN, online reparameterization (OREPA) introduced a linear scaling layer, which saved training costs and increased training speed while ensuring training results. The workflow of OREPA is as follows Figure 1 As shown, the internal structure is as Figure 2 shown.

[0005] RetinexFormer is an algorithm based on the combination of Retinex and Transformer. It simplifies the training and fine-tuning process and completes image enhancement in one stage in an end-to-end manner. Compared with the multi-stage low-light image enhancement method, RetinexFormer can better capture the long-term and better handle uneven lighting and shadow problems. In the process of infrared and visible light image fusion, visible light images usually provide texture details and background content. However, visible light images are easily affected by the shooting environment. In low-light conditions, more details and textures will be hidden, resulting in a lot of information cannot be fully utilized.

[0006] Adaptive Sparse Self-Attention (ASSA) is adaptively calculated using a dual-branch paradigm, which handles the attention relationship between pixels more intelligently. The sparse branch (SSA) is introduced to eliminate the negative impact of low query key matching scores on aggregate features, while the dense branch (DSA) ensures sufficient information flow. It can significantly improve computational efficiency while maintaining high model performance, accelerate the model training and inference process, and reduce computational costs.

[0007] Methods for infrared images and visible light images can be divided into two categories: traditional methods and deep learning-based methods.

[0008] Traditional fusion methods generally include three stages: first, extract infrared and visible light image features; then, design fusion strategies according to different fusion levels, including pixel-level fusion, feature-level fusion and decision-level fusion, and apply these strategies to the extracted features; finally, reconstruct the fusion features using the corresponding inverse transform. According to the different operation spaces, traditional methods can be divided into spatial domain-based methods and transform domain-based methods. Spatial domain-based methods are usually pixel-level fusion, which directly operates on pixels. The most common ones are the maximum method and the weighted average method, which are easy to implement and have excellent results. However, these algorithms have obvious limitations. The maximum method is prone to lose image information, which may lead to reduced saturation and resolution. The weighted average method is difficult to calculate the weight value and often cannot preserve complete texture details. In addition, the spatial domain method also uses the matrix similarity principle to fuse images, but it ignores that some images cannot be similarly diagonalized. The transform domain-based method converts the source image from the spatial domain to the frequency domain through a multi-scale pyramid transform (including Gaussian pyramid, Laplace pyramid, etc.) or a wavelet transform (including discrete wavelet transform, continuous wavelet transform, etc.), performs a fusion operation in the frequency domain, and finally applies the corresponding inverse transform to generate a fused image. Shi et al. used sparse representation to complete feature extraction and reconstruction, Simrandeep used discrete wavelet transform and bilateral filtering to fuse infrared and visible light images, and Madheswari et al. proposed a fusion method based on dual tree filtering. Although the loss of details is reduced compared with the spatial domain-based method, the boundary information is still difficult to fully preserve. In summary, the traditional methods have the following disadvantages: ① Due to the difficulty in weight selection, traditional fusion methods often find it difficult to fully retain details; ② The quality of multimodal image processing results is low and difficult to process; ③ The fusion strategy needs to be manually designed.

[0009] According to different learning strategies, deep learning-based fusion methods can be divided into methods based on convolutional neural networks (CNNs), autoencoders (AEs), and generative adversarial networks (GANs). Liu et al. introduced CNN to predict the coefficients of the feature fusion stage, which simplified the calculation. However, in the feature extraction stage, they still used traditional methods instead of convolution, did not directly operate the source image, and the source image data was not fully utilized. The coefficients calculated by CNN may not meet the actual needs. Li et al. proposed a fusion framework based on the pre-trained model VGG-19 to extract the high-frequency information contained in the source image, but did not operate the extracted features. The performance improvement of the fusion result was limited. In order to make full use of the features of the source image, Resnet replaced VGG-19 to extract features. Zhang et al. proposed a general structure IFCNN based on full convolution to achieve end-to-end training. Li et al. used AE to fuse infrared and visible light images, and improved the encoder with a dense network, called DenseFuse. In order to fully acquire and utilize multi-scale features, Li et al. designed a multi-scale encoder and an improved decoder structure, named NestFuse. On this basis, the authors proposed a new fusion strategy, using residual blocks to better retain the salient parts, called RFN-Nest, but the complex network leads to a long time consumption. Ma et al. first introduced the generative adversarial network into the fusion of infrared and visible light images to form the FusionGAN network, and established an adversarial relationship between the generator and the discriminator, aiming to make the generator produce higher quality images. The adversarial mechanism avoids the manual design of the fusion strategy. In addition, DDcGAN adds a discriminator on the basis of FusionGAN, and allows both visible light images and infrared images to participate in the adversarial training process. Rao et al. proposed TGFuse based on lightweight transformer modules and adversarial learning. Although GAN-based fusion methods have made great achievements, the network architecture often leads to unstable results and is relatively difficult to train. The above methods all take into account the network structure and training methods, but ignore the upgrade loss function. SeAFusion proposed by Tang et al. innovatively combines content loss with semantic loss, increasing the consideration of subsequent tasks. However, different tasks require loss functions with different focuses, and the design of loss functions is more difficult. The deep learning-based methods have achieved excellent results, but there are still limitations. On the one hand, the above algorithms only focus on normal scenes, ignoring extreme environments such as extremely dark environments and overexposed extreme environments. On the other hand, the convolution operation focuses on local relationships, ignores long-distance dependencies, and may not be able to retain global features.

[0010] In order to make the hidden details in visible light images visible and improve the utilization of details, methods used include histogram equalization and histogram equalization improvement, Gaussian filter, bilateral filter, gamma correction and other methods, but these traditional methods are not universal and cannot adapt to complex situations. Another type of enhancement algorithm is based on Retinex theory, including single-scale Retinex (SSR), multi-scale Retinex (MSR), and multi-scale Retinex with color restoration factor (MSRCR). Although these methods can produce excellent results, they still have limitations in decomposing reflections and lighting. Deep learning has made great progress in the past decade and has made great achievements in the field of image processing. Wei et al. combined Retinex with deep learning for the first time through Decom-net and Enhance-net. Li et al. also used the conclusion of Retinex to decompose and adjust the V channel of the image to obtain an enhanced image. Zhang et al. combined Retinex and denoising algorithm BM3D to propose "Kindling the Darkness" (KinD). Compared with traditional methods, deep learning-based methods improve the efficiency of decomposition. Through these enhancement methods, the enhanced image contains richer details in advance and obtains better results. Wei Caixia used the dual-domain enhancement method to enhance the infrared and visible light images first, and then fuse them, which significantly improved the fusion effect. In 2022, Tang proposed PIAFusion to use perceptual factors to estimate the illumination distribution of low-light images and obtain illumination probability during the fusion process, and to construct perceptual loss to guide the training of the fusion network, but it was not possible to obtain good fusion effects in complex environments. In 2023, Tang proposed the DIVFusion method to design a scene illumination disentanglement network (SIDNet), which was trained through the Retinex theory to obtain the enhanced feature maps of the Y channel visible light image and infrared image, and did not directly use the enhanced image obtained by the decoder as the subsequent input, which reduced the loss of effective information in the decoding process to a certain extent. Using the feature map as the input of the fusion network solves the incompatibility problem, and using the texture contrast enhancement network to reduce the texture and contrast loss of the fused image. It can be seen that using the Retinex theory to enhance the illumination channel and then fuse it can improve the image fusion effect. In addition, different network structures will also have an important impact on the fusion process. Yin et al. proposed a Double-Stream asymmetric network, which is to build a more targeted network to extract features based on the characteristics of visible light images and infrared images themselves, which can give full play to the advantages of the network and extract richer features.CCFusion proposed by Li Xiaoling et al. built a new coordinate attention mechanism, and in order to reduce the loss of effective information in the encoder when extracting features from images, a new network structure of multi-scale encoder-decoder was adopted, which effectively retained more information and improved the fusion effect. Min Li et al. used convolution kernels of different sizes based on the structure of the Inception module to obtain multi-scale features and combined them with the self-attention mechanism to extract richer texture detail information. It can be seen that a reasonable network architecture helps to better extract image features based on the characteristics of the image itself, and multi-scale information also plays a vital role in the fusion process. Summary of the invention

[0011] In view of the problems of serious detail loss, low brightness and contrast in low-light environment of existing fusion algorithms, the present invention proposes an improved low-light visible light and infrared image fusion method, which retains the source image detail information to the greatest extent and improves the brightness and contrast. This method realizes the parallel fusion and brightness enhancement, reduces the information loss caused by the incompatibility of the enhancement algorithm and the fusion algorithm, realizes the edge enhancement of the infrared image through visual saliency mapping, and adopts an adaptive sparse attention module to extract important information in the fusion process. At the same time, ordinary convolution is replaced by deep convolution and OREPA is used for reparameterization to reduce the network training and reasoning time overhead. The effectiveness of the method is verified on the LLVIP dataset and TNO dataset. The results show that compared with the classical fusion algorithm, this method can retain more source image detail information in low-light environment, improve the contrast and brightness of the image, and achieve good or similar results in both subjective and objective evaluation.

[0012] The technical solution of the present invention is as follows:

[0013] A low-light visible light and infrared image fusion method for autonomous driving comprises the following steps:

[0014] A1: Infrared image edge enhancement: For infrared images, we first use convolutions of different sizes to obtain features of different scales. The convolution sizes are 3*3, 5*5, and 7*7. To reduce the number of parameters, we use two and three 3*3 convolutions to replace 5*5 and 7*7 convolutions, respectively. On this basis, we use visual saliency mapping (VSM) to extract salient features from the extracted features and obtain the target feature map in the infrared image.

[0015] A2: Brightness enhancement of visible light images: The visible light image is first converted into the i channel through color space conversion, and then the brightness is enhanced using RetinexFormer, which is an enhancement method based on Retinex theory and Transformer. Retinex theory believes that visible light images can be decomposed into two parts: reflection component (R) and illumination component (I) (Formula 1). The image brightness can be improved by adjusting the illumination component. After being combined with Transformer, two-layer normalization, self-attention mechanism and feedforward network are used to better capture long-term features and deal with uneven illumination and shadow problems, thereby obtaining visible light images with richer detail information.

[0016]

[0017] A3: Preliminary fusion of infrared image and visible light image: Dense blocks are used for preliminary fusion to obtain preliminary fusion feature maps. The purpose of preliminary fusion is to retain more detailed texture information. Dense connections make each layer connected to all previous layers, reducing the loss of information during transmission.

[0018] A4: Edge-enhanced infrared image, feature map after brightness enhancement, and preliminary fusion features Figure 2 Secondary fusion: The three feature maps are merged through channels and fused twice through depthwise separable convolution, which reduces the number of parameters and calculation time. An online reparameterization structure is added, and the BN layer is replaced with a linear scaling layer to improve the model's ability to extract features and improve inference efficiency.

[0019] A5: Texture preservation: The features of different scales obtained from the above structure are first subjected to three convolutions of the same size, and then the small-sized features are upsampled to the same size as the adjacent large-sized features. The two are passed through the adaptive sparse attention module, and the sparse matrix is ​​introduced so that each input element only calculates the attention weight with part of the input elements, thereby reducing noise interaction and adaptively increasing the weight of sparse attention to better capture and retain texture feature information. After two upsampling and attention modules, three feature maps are obtained, which are merged into one feature map through the channel;

[0020] A6: Feature reconstruction: After four 3*3 convolutions, feature reconstruction is performed to obtain the fused image, which is then converted back to RGB color space to obtain the final fused image.

[0021] A7: Training and loss function

[0022] In order to obtain good training results, Loss1 is designed, as shown in Formula 3, including the structural similarity loss L ssim , perceptual loss Lperceptual loss, gradient loss Lgradient loss and pixel loss Lpixel loss, where m=10,n=0.5.

[0023] Loss1 = L ssim + L perceptual loss + m×L gradient loss + n×L pixel loss (3).

[0024] The low-illumination visible light and infrared image fusion method for autonomous driving, the step A1, when processing the infrared image, we use visual saliency mapping (VSM) to increase the contrast of the infrared image, VSM can detect the prominent part of the image according to the distance between the image pixels, reflecting the significant image features, when the infrared image is fused with the visible light image, the infrared image mainly provides the significant part of the image, so when extracting the infrared image features, VSM is used to extract the significant area, the present invention uses the principle that the saliency value of each pixel is the sum of the distances of the pixel from all other pixels in the picture to calculate the saliency map, this distance uses the Euclidean distance of the pixel feature value, as shown in Formula 1:

[0025] V(q)=||I q -I1||+||I q -I2||+…+||I q -I n || (1)

[0026] In the formula, I q Represents the pixel with pixel value q, I1…I n Represents the 1st to nth pixel values ​​in the image.

[0027] In the low-illumination visible light and infrared image fusion method for autonomous driving, in step A7, the final fused image must contain both the information of the infrared image and the information of the visible light image, and the result image is constrained by similarity so that it does not deviate from the original image. First, a structural similarity index is used to ensure the similarity between the fused image and the source image. The specific expression is shown in formula 5, which includes two parts. The structural similarity index of the result image and the infrared and visible light images is calculated respectively, where μ and σ in (4) represent the mean and standard deviation of the image respectively, and C1 and C2 represent two parameters to avoid the denominator being 0 and the numerator being 0.

[0028]

[0029] According to the low-illumination visible light and infrared image fusion method for autonomous driving, it is characterized in that, in step A7, a pre-trained VGG-19 network is applied to extract deep features and limit differences, which is called perceptual loss, as shown in Formula 6, using features 3, 8, 13, 22, and 31 of the VGG-19 network to calculate, where N represents the number of layers for extracting features, and the value is 5, φ j(x) represents the features of image x extracted from the VGG-19 network;

[0030]

[0031] In the low-light visible light and infrared image fusion method for autonomous driving, in step A7, the gradient loss and pixel loss constraints result in image boundary information and intensity information, as shown in equations 7 and 8, where The gradient is calculated using Sobel, and the pixel loss is calculated using the L2 loss of the result image and the original image.

[0032]

[0033] The present invention has the following beneficial effects:

[0034] 1. A parallel enhancement fusion network is used to carry out the enhancement process and the fusion process simultaneously, and the enhancement process is used to supplement the fusion, effectively avoiding the loss of information in the process of enhancing the image first and then fusing it.

[0035] 2. Use the adaptive sparse attention mechanism to fuse multi-scale features, eliminate the negative impact of low query key matching scores on aggregate features through sparse branches, accelerate the training and inference process of the model, and reduce the computational cost.

[0036] 3. When extracting features of visible light and infrared images, the same network is not used. Instead, different methods are used to extract them according to their respective characteristics, and a more targeted network is designed to better extract their respective features, better retain texture information and edge information, and obtain clearer fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 : OREPA workflow;

[0038] Figure 2 :OREPA structure;

[0039] Figure 3 : ASSA structure diagram;

[0040] Figure 4 : Flow chart of the improved low-illumination visible light and infrared image fusion method of the present invention;

[0041] Figure 5 : Flowchart of infrared image edge enhancement;

[0042] Figure 6 :DW-OREPA;

[0043] Figure 7 : Overall network architecture diagram of the present invention;

[0044] Figure 8 : Test results on the LLVIP dataset;

[0045] Fig. 9 : Test results on the TNO dataset; DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with specific embodiments.

[0047] The present invention proposes a parallel network to supplement the initial fusion result by enhancing the visible light image features and extracting the infrared features, thereby reducing the loss of source image information in the result image. The three feature maps are subjected to deep convolution with different convolution kernel sizes to obtain features of different scales, and then the texture details are retained by an adaptive sparse attention module. Finally, the features are reconstructed to obtain the final fused image. The overall network architecture is as follows: Figure 7 shown.

[0048] like Figure 4 As shown, a low-light visible light and infrared image fusion method for autonomous driving includes the following steps:

[0049] A1: Infrared image edge enhancement ( Figure 5 ): For infrared images, the main purpose is to retain target information. First, convolutions of different sizes are used to obtain features of different scales. The convolution sizes are 3*3, 5*5, and 7*7, respectively. In order to reduce the number of parameters, two and three 3*3 convolutions are used to replace 5*5 and 7*7 convolutions, respectively. On this basis, the visual saliency map (VSM) is used to extract the significant features of the extracted features to obtain the target feature map in the infrared image.

[0050] When processing infrared images, we use visual saliency mapping (VSM) to increase the contrast of infrared images. VSM can detect the prominent parts of the image according to the distance between image pixels, reflecting the significant image features. When the infrared image is fused with the visible light image, the infrared image mainly provides the significant parts of the image. Therefore, VSM is used to extract the significant area when extracting the infrared image features. The present invention calculates the saliency map based on the principle that the saliency value of each pixel is the sum of the distances from the pixel to all other pixels in the image. This distance uses the Euclidean distance of the pixel feature value, as shown in Formula 1:

[0051] V(q)=||I q -I1||+||I q -I2||+…+||I q -I n || (1)

[0052] In the formula, I qRepresents the pixel with pixel value q, I1…I n Represents the 1st to nth pixel values ​​in the image.

[0053] A2: Brightness enhancement of visible light images: The visible light image is first converted into the i channel (hsi color space) by color space conversion, and then the brightness is enhanced by RetinexFormer, which is an enhancement method based on Retinex theory and Transformer. Retinex theory believes that visible light images can be decomposed into two parts: reflection component (R) and illumination component (I) (Formula 1). The image brightness can be improved by adjusting the illumination component. After being combined with Transformer, two-layer normalization, self-attention mechanism and feedforward network are used to better capture long-term features and deal with uneven illumination and shadow problems, so as to obtain visible light images with richer detail information.

[0054]

[0055] A3: Preliminary fusion of infrared image and visible light image: Dense blocks are used for preliminary fusion to obtain a preliminary fusion feature map; the purpose of preliminary fusion is to retain more detailed texture information, and dense connections make each layer connected to all previous layers, reducing the loss of information during transmission.

[0056] A4: Edge-enhanced infrared image, feature map after brightness enhancement, and preliminary fusion features Figure 2 Secondary fusion ( Figure 6 ): The three feature maps are merged through channels and fused twice through depthwise separable convolution, which reduces the number of parameters and calculation time. An online reparameterization structure is added, and the BN layer is replaced by a linear scaling layer to improve the model's ability to extract features and improve inference efficiency.

[0057] A5: Texture preservation: The features of different scales obtained from the above structure are first subjected to three convolutions of the same size, and then the small-sized features are upsampled to the same size as the adjacent large-sized features. The two are passed through an adaptive sparse attention module and a sparse matrix is ​​introduced so that each input element only calculates the attention weight with part of the input elements, thereby reducing noise interaction and adaptively increasing the weight of sparse attention to better capture and retain texture feature information. After two upsampling and attention modules, three feature maps are obtained, which are merged into one feature map through channels, such as Figure 7 shown.

[0058] A6: Feature reconstruction: After four 3*3 convolutions, feature reconstruction is performed to obtain the fused image, which is then converted back to RGB color space to obtain the final fused image.

[0059] A7: Training and loss function

[0060] In order to make the model have a better effect, it is necessary to train the model. The dataset used in the training process is the LLVIP dataset, which is suitable for visual tasks under low-light conditions and contains 15,488 pairs of infrared and visible light images. Before entering the network, we resize all images to 256×256 and normalize them. The learning rate is set to 0.0001, and the Adam optimizer is used.

[0061] In order to obtain good training results, Loss1 is designed, as shown in Formula 3, including the structural similarity loss L ssim , perceptual loss Lperceptual loss, gradient loss Lgradient loss and pixel loss Lpixel loss, where m=10,n=0.5.

[0062] Loss1 = L ssim + L perceptual loss + m×L gradient loss + n×L pixel loss (3)

[0063] The final fused image should contain both the information of the infrared image and the information of the visible light image, so it is necessary to use similarity constraints on the resulting image so that it does not deviate from the original image. First, the structural similarity index is used to ensure the similarity between the fused image and the source image. The specific expression is shown in formula (5), which includes two parts. The structural similarity index of the resulting image and the infrared and visible light images should be calculated respectively. In (4), μ and σ represent the mean and standard deviation of the image respectively, and C1 and C2 represent two parameters to avoid the denominator and numerator being 0.

[0064]

[0065] When we use neural networks to extract features, low-frequency information (such as color and brightness) is obtained by shallow networks, while high-frequency information, such as texture details, is usually obtained by deep networks. Preserving more details is our expectation, so the pre-trained VGG-19 network is used to extract deep features and limit the difference, which is called perceptual loss. As shown in Formula 6, the present invention uses features 3, 8, 13, 22, and 31 of the VGG-19 network to calculate. N in the formula represents the number of layers for extracting features, which is 5, and φ j (x) represents the features of image x extracted from the VGG-19 network.

[0066]

[0067] The gradient loss and pixel loss constrain the boundary information and intensity information of the resulting image, as shown in Equations 7 and 8, where The gradient is calculated using Sobel, and the pixel loss is calculated using the L2 loss of the result image and the original image.

[0068]

[0069] Subjective evaluation of experimental results

[0070] exist Figure 8 In the comparison, we selected three groups of images from the LLVIP dataset and found that the output images of the rfn model appear to be fuzzy in details and lose a lot of delicate information. For example, in the first group of images, the texture of the floor tiles becomes unclear; while in the second and third groups of images, the outlines and features of pedestrians appear blurred and artifacts appear. Although the nestfuse model successfully retains some features of infrared images, it lacks in color restoration. For example, the red light in the first group of images and the color of the express truck in the second group of images are not accurately preserved. The ifcnn model does a good job in retaining the details of visible light images, but the overall brightness of the image is insufficient, and the loss of details in dark light environments is not fully considered. In particular, the express truck in the second group of images and the trash can in the third group of images become difficult to recognize in the fused image. The res2fusion model makes the image look clearer by increasing the contrast of the image. However, this also leads to a large loss of image edge information. In the second and third groups of images, it can be clearly seen that the information in the lower left corner is seriously lost. The densefuse model performs well in detail preservation and noise control, but the overall image is dark, which affects the visual effect. In contrast, the model proposed by the present invention performs better in improving image brightness and contrast, and the information of edges and corners is also better preserved. Although some detail information is still not fully preserved, overall, the visual experience of the model is better.

[0071] Fig. 9In the comparison of three groups of images in the TNO dataset, the following points can be observed: when the rfn model processes the infrared image information, there is a relatively obvious loss, the boundary is blurred, and the white area in the first group of images and the details of the characters in the second group of images lose a lot of information, which affects the clarity of the image. Although the nestfuse model has a higher contrast, the processing of the white area appears too sharp and the exposure is not natural enough. For example, the white flames in the first group of images and the vehicle tires in the third group of images have such problems. The ifcnn model has a good effect in fusing visible light and infrared image information, but the image surface is not smooth enough and there is a small amount of noise, which affects the overall visual effect. The problem is particularly obvious in the result image of the first group. During the processing, the res2fusion model loses part of the information of the visible light image, resulting in a low contrast. In the third group of images, the shape of the clouds in the sky and the outline of the tree on the left are not fully preserved, which affects the detail performance of the image. When the densefuse model processes the white target information in the infrared image, there is a lot of loss, which makes this part of the information not prominent enough in the fused image. In contrast, the model proposed in the present invention performs better in contrast enhancement and information retention. It not only effectively fuses the information in infrared and visible light images, but also obtains more natural and clear fusion results, with better overall visual effects.

[0072] Objective evaluation of experimental results

[0073] In order to quantitatively evaluate this method and the comparison algorithm, information entropy (EN), standard deviation (SD), mean square error (MSE), peak signal-to-noise ratio (PSNR) and average gradient (AG) are used for quantitative evaluation, and the calculation expressions are shown in Figures 9 to 15. The smaller the value of MSE, the better the fusion quality, and the larger the values ​​of the other five indicators, the better the fusion effect.

[0074]

[0075] MSE(A,B,F)=MSE(A,F)+MSE(B,F) (13)

[0076]

[0077] The test results of different models on the LLVIP dataset and TNO dataset are shown in Table 1-2. The bold data are the best values, and the bold and underlined values ​​are the suboptimal values.

[0078] It can be seen from Table 1 that the mean square error of the model of the present invention can reach the best value of the listed models on the three groups of images in the LLVIP data set, and the standard improvement of the second and third groups is particularly obvious, which shows that the model of the present invention performs better than other models in retaining the source image information; the standard deviation can achieve the best value on the three groups of images, and the information entropy also has the best value for two groups, and the information entropy of the second group is also slightly different from the best value, which shows that the model of the present invention can retain a relatively clear image; the peak signal-to-noise ratio of the nest model is the last one or two digits in all models, indicating that its image distortion is more serious, and the peak signal-to-noise ratio of the first and second groups of images of the model of the present invention can achieve the highest results, indicating that the image distortion obtained by the model of the present invention is small, and a higher quality fusion image can be obtained; the average gradient is not the best on these three groups of images, indicating that the model of the present invention needs to be improved in edge information processing. From the overall effect, the model of the present invention can obtain better or similar results than other models. As can be seen from Table 2, the information entropy, standard deviation and average gradient of the model of the present invention in the TNO data set all achieve the best values ​​among all models, and except for the average gradient of the third group, they are all significantly improved relative to the second best value, which shows that the model of the present invention has a great advantage in improving image clarity and contrast, and can obtain images with better visual perception; the mean square error achieves the second best value in the first and third groups, second only to the densityfuse model; the peak signal-to-noise ratio is not the best in these images, but the difference is not big, but from the overall analysis, the model of the present invention has at least 3 of the 5 indicators that are the best, and the performance is better than the other 5 models. It can be seen that the fusion result obtained by the model of the present invention better retains the information and structure of the source image, and also performs better in visual effect, which fully proves the effectiveness of the model of the present invention.

[0079] Table 1 Objective evaluation criteria of LLVIP dataset images in different model result graphs

[0080]

[0081]

[0082] Table 2 Objective evaluation criteria values ​​of TNO dataset in different model result graphs

[0083]

[0084] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A low-illumination visible light and infrared image fusion method for autonomous driving, characterized in that: The following steps are involved: A1: infrared image edge enhancement; A2: Visible light image brightness enhancement; A3: Preliminary fusion of infrared image and visible light image; A4: secondary fusion of edge-enhanced infrared image, brightness-enhanced feature map and preliminary fusion feature map; A5: Texture preservation: The features of different scales obtained from the above structure are first subjected to three convolutions of the same size, and then the small-sized features are upsampled to the same size as the adjacent large-sized features. The two are passed through the adaptive sparse attention module, and the sparse matrix is ​​introduced so that each input element only calculates the attention weight with part of the input elements, thereby reducing noise interaction and adaptively increasing the weight of sparse attention to better capture and retain texture feature information. After two upsampling and attention modules, three feature maps are obtained, which are merged into one feature map through the channel; A6: Feature reconstruction: After four 3*3 convolutions, feature reconstruction is performed to obtain the fused image, which is then converted back to RGB color space to obtain the final fused image; A7: Training and loss function In order to obtain good training results, Loss1 is designed, as shown in Formula 3, including the structural similarity loss L ssim , perceptual loss Lperceptual loss, gradient loss Lgradient loss and pixel loss Lpixel loss, where m=10, n=0.5; Loss1 = L ssim + L perceptual loss + m×L gradient loss + n×L pixel loss (3).

2. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: In step A1, when processing the infrared image, we use visual saliency mapping (VSM) to increase the contrast of the infrared image. VSM can detect the prominent part of the image according to the distance between the image pixels, reflecting the significant image features. When the infrared image is fused with the visible light image, the infrared image mainly provides the significant part of the image. Therefore, when extracting the infrared image features, VSM is used to extract the significant area. The saliency map is calculated based on the principle that the saliency value of each pixel is the sum of the distances from the pixel to all other pixels in the image. This distance uses the Euclidean distance of the pixel feature value, as shown in Formula 1: V(q)=||I q -I1||+||I q -I2||+…+||I q -I n || (1) In the formula, I q Indicates the pixel with pixel value q, I1…I n Represents the 1st to nth pixel values ​​in the image.

3. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: In step A7, the final fused image should contain both the information of the infrared image and the information of the visible light image. The result image is constrained by similarity so that it does not deviate from the original image. First, the structural similarity index is used to ensure the similarity between the fused image and the source image. The specific expression is shown in formula 5, which includes two parts. The structural similarity index of the result image and the infrared and visible light images should be calculated respectively. In (4), μ and σ represent the mean and standard deviation of the image respectively, and C1 and C2 represent two parameters to avoid the denominator being 0 and the numerator being 0.

4. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: In step A7, the pre-trained VGG-19 network is applied to extract deep features and limit the difference, which is called perceptual loss. As shown in Formula 6, it is calculated using features 3, 8, 13, 22, and 31 of the VGG-19 network. N in the formula represents the number of layers for extracting features, which is 5. j (x) represents the features of image x extracted from the VGG-19 network; 5. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: In step A7, the gradient loss and pixel loss constraints result in image boundary information and intensity information, as shown in equations 7 and 8, where The gradient is calculated using Sobel, and the pixel loss is calculated using the L2 loss of the result image and the original image; 6. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: The step A1: for infrared images, first use convolutions of different sizes to obtain features of different scales, and the convolution sizes are 3*3, 5*5, and 7*7 respectively. In order to reduce the number of parameters, two and three 3*3 convolutions are used to replace 5*5 and 7*7 convolutions respectively. On this basis, the visual saliency map (VSM) is used to extract significant features from the extracted features to obtain a target feature map in the infrared image.

7. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: Step A2: The visible light image is first converted into a color space to obtain an i channel, and brightness is enhanced using RetinexFormer, which is an enhancement method based on Retinex theory and Transformer. Retinex theory believes that visible light images can be decomposed into two parts: a reflection component (R) and an illumination component (I) (Formula 1). The image brightness can be improved by adjusting the illumination component. After being combined with Transformer, two-layer normalization, self-attention mechanism and feedforward network are used to better capture long-term features and handle uneven illumination and shadow problems, thereby obtaining a visible light image with richer detail information; 8. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: The step A3: using dense blocks to perform preliminary fusion to obtain a preliminary fusion feature map; the purpose of preliminary fusion is to retain more detailed texture information, and dense connection enables each layer to be connected to all previous layers, reducing the loss of information during transmission.

9. The low-illumination visible light and infrared image fusion method for autonomous driving according to claim 1, characterized in that: The step A4: merging the channels of the above three feature maps, and performing secondary fusion through depth-wise separable convolution, thereby reducing the number of parameters and calculation time. An online reparameterization structure is added, and the BN layer is replaced by a linear scaling layer, thereby improving the model's ability to extract features and improving the inference efficiency.

Images