A method and apparatus for image enhancement and fusion of low-light images and infrared images

Abstract

This invention discloses a method for fusing low-light and infrared images based on image enhancement. The method includes: performing image enhancement processing on the low-light image data to obtain enhanced low-light image data; performing a pre-training process on the enhanced low-light image data and infrared image data to obtain intermediate low-light image data and intermediate infrared image data; and performing a secondary fusion based on the intermediate low-light image data and intermediate infrared image data to obtain a fused image. This invention solves the problem of how to achieve infrared and visible light image fusion in low-light environments at night.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and apparatus for image enhancement and fusion of low-light images and infrared images. Background Technology

[0002] Image information acquired from a single band or single detector cannot represent all the information in a scene. Image fusion technology can achieve complementary information between different images, reduce data redundancy, and has great application prospects. Infrared and visible light fusion algorithms are currently a popular research direction and have wide applications in military monitoring, traffic security, and other fields. Infrared images receive thermal radiation information from a scene, are not affected by illumination, and can highlight the characteristics of thermal targets in the scene, but the image resolution is low and the texture information is limited; while visible light images contain rich texture and background information, have higher resolution, but target features are not prominent enough, and in low ambient illumination, the image brightness is dark and the contrast is low.

[0003] Currently, numerous algorithms for infrared and visible light image fusion have emerged, broadly categorized into two types: traditional algorithms and deep learning-based algorithms. The core idea of ​​traditional methods is to map the source image to a feature dimension using predetermined image decomposition representation methods, then employ feature fusion strategies, and finally inversely map the fused features back to the image. Depending on the image decomposition method, these algorithms can be further divided into multi-scale transformation methods, sparse representation methods, subspace clustering methods, optimization-based methods, and hybrid methods. However, these methods have significant drawbacks. Their fusion effect is highly dependent on manually designed feature extraction methods, making it difficult to handle increasingly complex application scenarios. Furthermore, the feature extraction methods are too simplistic, and the algorithm structure is complex and inefficient.

[0004] In recent years, the emergence of deep learning technology has also promoted the development of image fusion technology. Deep learning methods use data-driven approaches to extract original features from source images and achieve better fusion results. Currently, mainstream infrared and visible light image fusion algorithms can be categorized into: AE-based methods, CNN-based methods, GAN-based methods, and Transformer-based methods. AE-based methods use encoders and decoders to extract and reconstruct image features, while feature fusion employs specific strategies. CNN-based methods achieve fusion effects by designing specific network structures and loss functions. GAN-based methods use generators to generate fused images and discriminators to constrain the fusion effect. Transformer-based methods replace CNNs with the Transformer architecture in the previous model.

[0005] While current deep learning-based methods can achieve relatively ideal image fusion results, several issues remain to be addressed. First, infrared and visible light fusion is primarily used in low-light scenarios such as nighttime. Due to the limitations of scene illumination, visible light images suffer from low brightness, low contrast, and a lack of texture detail. Current algorithms that directly fuse low-quality visible light and infrared images lead to the loss of numerous potential features, resulting in infrared features dominating the fused image – not the desired outcome of image fusion. Second, existing deep learning methods neglect the feature requirements of different source images during the fusion process, extracting features equally from both. This results in the fused image resembling a weighted average of the images, severely impacting the quality of the fused image. Summary of the Invention

[0006] This invention provides a method and apparatus for fusing low-light images and infrared images based on image enhancement, solving the problem of how to achieve the fusion of infrared and visible light images in low-light environments at night.

[0007] A method for image enhancement fusion of low-light and infrared images includes:

[0008] The low-light image data is subjected to image enhancement processing to obtain enhanced low-light image data;

[0009] The enhanced low-light image data and infrared image data are pre-trained to obtain intermediate low-light image data and intermediate infrared image data.

[0010] A fused image is obtained by performing a secondary fusion of the intermediate data from the low-light image and the intermediate data from the infrared image.

[0011] In one embodiment of the present invention, the step of performing image enhancement processing on low-light image data to obtain enhanced low-light image data specifically includes: a low-light image enhancement algorithm based on a global tone mapping operator stretches the original dynamic range of the low-light image to restore the texture details of the low-light image, so that the original image with low brightness is restored into a normal image with high contrast and high brightness.

[0012] In one embodiment of the present invention, the step of pre-training the enhanced low-light image data and infrared image data specifically includes: inputting the enhanced low-light image data and infrared image data into two autoencoders with the same structure for pre-training; and obtaining the optimal autoencoder parameters with the mean absolute error L1 loss as a constraint.

[0013] In one embodiment of the present invention, the step of performing secondary fusion based on the intermediate data of the low-light image and the intermediate data of the infrared image to obtain a fused image specifically includes: obtaining the encoder parameters in the autoencoder after a first pre-training; copying the encoder parameters into the feature extraction unit of the secondary training model; inputting the intermediate data of the low-light image and the intermediate data of the infrared image into the feature extraction unit for feature extraction to obtain two corresponding high-dimensional feature vectors; integrating the two high-dimensional feature vectors to obtain a single high-dimensional feature vector; and reconstructing the image using the high-dimensional feature vector to generate a single-channel fused image.

[0014] In one embodiment of the present invention, two high-dimensional feature vectors are integrated to obtain a high-dimensional feature vector. Specifically, the two high-dimensional feature vectors are concatenated along the channel dimension, and then channel compression is performed through a convolutional layer to output a high-dimensional feature vector.

[0015] An apparatus for image enhancement fusion of low-light images and infrared images, comprising:

[0016] The low-light image enhancement module is used to perform image enhancement processing on low-light image data to obtain enhanced low-light image data;

[0017] The image fusion module is used to perform a pre-training process on the enhanced low-light image data and infrared image data to obtain intermediate low-light image data and intermediate infrared image data; and to perform a secondary fusion process on the intermediate low-light image data and intermediate infrared image data to obtain a fused image.

[0018] In one embodiment of the present invention, the image fusion module further includes a pre-training unit; the pre-training unit includes two autoencoders based on Swing Transformer, one autoencoder including a low-light encoder and a low-light decoder, and the other autoencoder including an infrared encoder and an infrared decoder.

[0019] In one embodiment of the present invention, the image fusion module further includes: a feature extraction unit, used to input the low-light image intermediate data and the infrared image intermediate data into the feature extraction unit for feature extraction to obtain two corresponding high-dimensional feature vectors; a feature fusion unit, used to integrate the two high-dimensional feature vectors to obtain a high-dimensional feature vector; and an image reconstruction unit, used to reconstruct the image from the high-dimensional feature vector to generate a single-channel fused image.

[0020] In one embodiment of the present invention, the image fusion module further includes a feature extraction unit and an image reconstruction unit; the low-light encoder and infrared encoder in the feature extraction unit have the same network structure and parameters as the low-light encoder and infrared encoder in the first pre-training unit, and are used to extract features from the input three-channel source image and output a high-dimensional feature vector; the image reconstruction unit adopts a Swing Transformer-based model and has the same network structure as the decoder in the first pre-training unit.

[0021] In one embodiment of the present invention, the parameters of the feature fusion unit and the image reconstruction unit are obtained through secondary training; L1 loss and gradient loss are used to constrain the convergence process of the network of the feature fusion unit and the image reconstruction unit.

[0022] This invention provides a method and apparatus for fusing low-light images and infrared images based on image enhancement, which includes at least the following beneficial effects: Starting from low-light environments such as nighttime, this invention addresses the defects of low-light images, restores the texture details of the low-light images themselves through global tone mapping, effectively improves the brightness and contrast of the low-light images, and enhances the visual effect of the fused image; through a dual-training mode, feature extractors with different parameters are used for different source images, meeting the feature requirements of different images in the fusion process, and effectively improving the performance of image fusion. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0024] Figure 1 This is a schematic diagram illustrating the steps of a method for fusing low-light images and infrared images based on image enhancement, as provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of a low-light image enhancement module provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a pre-training of the image fusion module provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the secondary training of the image fusion module provided in an embodiment of the present invention;

[0028] Figure 5 A comparison chart of the effects of the algorithm of the present invention and mainstream fusion algorithms provided in this embodiment of the invention;

[0029] Figure 6A comparison chart of the effects of the algorithm of the present invention and mainstream fusion algorithms provided in this embodiment of the invention;

[0030] Figure 7 This is a schematic diagram of a device based on image enhancement and fusion of low-light images and infrared images, provided for an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described clearly and completely below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] It should be noted that those skilled in the art will understand, explicitly and implicitly, that the embodiments described in this invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical or scientific terms used in this invention should be understood in their ordinary sense by those skilled in the art. The terms "a," "an," "an," "the," etc., used in this invention do not indicate quantity limitation and can represent singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this invention are intended to cover non-exclusive inclusion; the terms "first," "second," "third," etc., used in this invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0033] This invention proposes a method and apparatus for fusing low-light and infrared images based on image enhancement. The method can be implemented using an apparatus including a low-light image enhancement module and an image fusion module. First, this invention employs a low-light image enhancement algorithm based on global tone mapping, which can restore texture details of visible light images in low-light environments at night. Second, the enhanced low-light and infrared images are respectively encoded using their respective encoder structures to extract latent features from the source images, and after feature fusion, these features are input into an image reconstruction unit to generate a fused image. This invention adopts a dual-training mode, using different feature extraction models for different source images to meet the fusion requirements of different image features. Combining low-light image enhancement and low-light infrared image fusion algorithms improves the overall visual effect of the fused image, providing an effective means for low-light infrared dual-channel monitoring in low-light environments. Detailed explanation follows.

[0034] Figure 1 A schematic diagram illustrating the steps of a method for fusing low-light images and infrared images based on image enhancement, as provided in an embodiment of the present invention, may include the following steps:

[0035] S110: Perform image enhancement processing on the low-light image data to obtain enhanced low-light image data.

[0036] In one embodiment of the present invention, low-light image data is subjected to image enhancement processing to obtain enhanced low-light image data. Specifically, the low-light image enhancement algorithm based on global tone mapping operator stretches the original dynamic range of the low-light image and restores the texture details of the low-light image so that the original image with low brightness is restored into a normal image with high contrast and high brightness.

[0037] Specifically, such as Figure 2 As shown, a low-light image enhancement algorithm based on a global tone mapping operator is used to restore the texture details of low-light images. By stretching the overall dynamic range of the image, the originally dark image is restored into a normal image with high contrast and high brightness, greatly restoring the hidden image details in the low-light image.

[0038] The specific implementation formula is as follows:

[0039]

[0040]

[0041]

[0042] In the formula, This represents the input low-light image. This represents the enhanced low-light image. Represents the logarithmic mean of the input low-light image, Represents the maximum pixel value of a low-light image. Represents the number of pixels in a low-light image. The coordinate axes of the low-light image are respectively: It is a very small non-zero bias.

[0043] S120: Perform a pre-training operation on the enhanced low-light image data and infrared image data to obtain intermediate low-light image data and intermediate infrared image data.

[0044] In one embodiment of the present invention, the enhanced low-light image data and infrared image data are pre-trained, specifically including: inputting the enhanced low-light image data and infrared image data into two autoencoders with the same structure for pre-training; and obtaining the optimal autoencoder parameters with the mean absolute error L1 loss as a constraint.

[0045] Specifically, the enhanced low-light image data and infrared image data are fused together using an image fusion module. The image fusion module employs a dual-training approach. First, the enhanced low-light image and infrared image datasets are pre-trained separately using a Swing Transformer autoencoder with the same structure. Using L1 loss as a constraint, the optimal autoencoder parameters are obtained. Only the encoder portion of this autoencoder is used as the feature extraction unit during the second training of the image fusion module.

[0046] like Figure 3 As shown, in one pre-training process of the image fusion module, low-light image data and infrared image data are input into two different autoencoders, respectively, and the convergence process of the autoencoders is constrained by the L1 loss function. The autoencoder consists of an encoder and a decoder, both of which are composed of four Swing Transformer Blocks. After pre-training, only the encoder is used as the low-light encoder and infrared encoder as feature extraction units.

[0047] The L1 loss function in a single pre-training iteration is as follows:

[0048]

[0049] In the formula, This represents the input of the autoencoder. This represents the output of the autoencoder. This represents the L1 norm.

[0050] Through a single pre-training process, it is possible to denoise low-light images and infrared images, assign weights to low-light images and infrared images in the model, and correct the non-uniformity of infrared images.

[0051] S130: Perform secondary fusion based on the intermediate data of the low-light image and the intermediate data of the infrared image to obtain the fused image.

[0052] In one embodiment of the present invention, a fused image is obtained by secondary fusion of intermediate data from low-light images and intermediate data from infrared images. Specifically, this includes: obtaining encoder parameters from the autoencoder after a first pre-training; copying the encoder parameters into the feature extraction unit of the secondary training model; inputting the intermediate data from low-light images and intermediate data from infrared images into the feature extraction unit for feature extraction to obtain two corresponding high-dimensional feature vectors; integrating the two high-dimensional feature vectors to obtain a single high-dimensional feature vector; and reconstructing the image using the high-dimensional feature vector to generate a single-channel fused image.

[0053] In one embodiment of the present invention, two high-dimensional feature vectors are integrated to obtain a high-dimensional feature vector. Specifically, the two high-dimensional feature vectors are concatenated along the channel dimension, and then channel compression is performed through a convolutional layer to output a high-dimensional feature vector.

[0054] Specifically, after one pre-training iteration, the parameters of the low-light encoder and infrared encoder in the feature extraction unit remain fixed. Then, the feature fusion unit in the image fusion module integrates the two high-dimensional feature vectors output by the feature extraction unit, concatenates them along the channel dimension, and performs channel compression through a 1*1 convolutional layer to output a single high-dimensional feature vector. Finally, this high-dimensional feature vector is processed by the image reconstruction unit to generate a single-channel fused image. The image reconstruction unit has the same structure as the decoder in the Swin Transformer autoencoder. The parameters of the feature fusion unit and the image reconstruction unit are obtained through a second training iteration. At this point, the network input consists of pairs of low-light and infrared images. L1 loss and gradient loss are used to constrain the network's convergence process. The L1 loss measures the pixel difference between the fused image and the two source images, while the gradient loss measures the difference between the gradient of the fused image and the maximum gradient of the two source images.

[0055] like Figure 4 As shown, the secondary training process in the image fusion module first constructs the network models for the feature fusion unit and the image reconstruction unit. The feature extraction unit uses the encoder parameters pre-trained in the first training iteration. The secondary training uses two functions, L1 loss and gradient loss, to constrain the convergence process.

[0056] The loss function formula for L1 in the second training is as follows:

[0057]

[0058] In the formula, This represents the output fused image. This represents the enhanced low-light image. This represents the input infrared image.

[0059] The gradient loss function formula for secondary training is as follows:

[0060]

[0061] In the formula, This represents the gradient of the output fused image. This represents the gradient of the enhanced low-light image. This represents the gradient of the input infrared image.

[0062] To better demonstrate the advantages of the algorithm of this invention, it was compared with the current mainstream visible light and infrared image fusion algorithms. For example... Figure 5 and Figure 6 As shown in the figure, the algorithm of the present invention can effectively restore the texture details of low-light images and highlight the current characteristics in infrared images, while the fusion results of other algorithms include more infrared information and ignore the features of visible light images.

[0063] In summary, the low-light and infrared image fusion method combining low-light image enhancement can achieve high-brightness and high-contrast image fusion effects in low-light environments. While highlighting significant target features in infrared images, it recovers a large amount of background detail information in low-light images, providing conditions for target detection, surveillance, and other applications in low-light environments such as at night.

[0064] The above describes a method for fusing low-light images and infrared images based on image enhancement according to an embodiment of the present invention. Based on the same inventive concept, the present invention also provides a corresponding apparatus for fusing low-light images and infrared images based on image enhancement, such as… Figure 7 As shown, it includes:

[0065] The low-light image enhancement module 702 is used to perform image enhancement processing on the low-light image data to obtain enhanced low-light image data; the image fusion module 704 is used to perform a pre-training on the enhanced low-light image data and infrared image data to obtain intermediate low-light image data and intermediate infrared image data; and to perform a secondary fusion based on the intermediate low-light image data and intermediate infrared image data to obtain a fused image.

[0066] In one embodiment of the present invention, the image fusion module 704 further includes a pre-training unit 7041; the pre-training unit 7041 includes two autoencoders based on Swing Transformer, one of which includes a low-light encoder and a low-light decoder, and the other of which includes an infrared encoder and an infrared decoder.

[0067] In one embodiment of the present invention, the image fusion module 704 further includes: a feature extraction unit 7042, used to input the low-light image intermediate data and the infrared image intermediate data into the feature extraction unit for feature extraction to obtain two corresponding high-dimensional feature vectors; a feature fusion unit 7043, used to integrate the two high-dimensional feature vectors to obtain a high-dimensional feature vector; and an image reconstruction unit 7044, used to reconstruct the image from the high-dimensional feature vector to generate a single-channel fused image.

[0068] In one embodiment of the present invention, the image fusion module 704 further includes a feature extraction unit 7042 and an image reconstruction unit 7044; the low-light encoder and infrared encoder in the feature extraction unit 7042 have the same network structure and parameters as the low-light encoder and infrared encoder in the first pre-training unit 7041, and are used to extract features from the input three-channel source image and output a high-dimensional feature vector; the image reconstruction unit 7044 adopts a Swing Transformer-based model and has the same network structure as the decoder in the first pre-training unit 7041.

[0069] In one embodiment of the present invention, the parameters of the feature fusion unit 7043 and the image reconstruction unit 7044 are obtained through secondary training; L1 loss and gradient loss are used to constrain the convergence process of the network of the feature fusion unit 7043 and the image reconstruction unit 7044.

[0070] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process method or product that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process method or product. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process method or product that includes that element.

[0071] The above are merely embodiments of the present invention and are not intended to limit the invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for fusing low-light images and infrared images based on image enhancement, characterized in that, include: The low-light image data is subjected to image enhancement processing to obtain enhanced low-light image data; The enhanced low-light image data and infrared image data are pre-trained to obtain intermediate low-light image data and intermediate infrared image data. The fused image is obtained by secondary fusion of the intermediate data of the low-light image and the intermediate data of the infrared image. Specifically, this includes: obtaining the encoder parameters in the autoencoder after one pre-training; copying the encoder parameters into the feature extraction unit of the secondary training model; inputting the intermediate data of the low-light image and the intermediate data of the infrared image into the feature extraction unit for feature extraction to obtain two corresponding high-dimensional feature vectors; integrating the two high-dimensional feature vectors to obtain a single high-dimensional feature vector; and reconstructing the image using the high-dimensional feature vector to generate a single-channel fused image.

2. The method according to claim 1, characterized in that, The step of performing image enhancement processing on the low-light image data to obtain enhanced low-light image data specifically includes: Low-light image enhancement algorithms based on global tone mapping operators stretch the dynamic range of low-light images and restore the texture details of low-light images, so that the original dark images can be restored into normal images with high contrast and high brightness.

3. The method according to claim 1, characterized in that, The step of pre-training the enhanced low-light image data and infrared image data specifically includes: The enhanced low-light image data and infrared image data are respectively input into two autoencoders with the same structure for pre-training. The optimal autoencoder parameters are obtained by using the mean absolute error L1 loss as a constraint.

4. The method according to claim 1, characterized in that, The integration of two high-dimensional feature vectors yields a single high-dimensional feature vector, specifically including: Two high-dimensional feature vectors are concatenated along the channel dimension, and then channel compression is performed through a convolutional layer to output a single high-dimensional feature vector.

5. A device for image enhancement and fusion of low-light images and infrared images, characterized in that, include: The low-light image enhancement module is used to perform image enhancement processing on low-light image data to obtain enhanced low-light image data; The image fusion module is used to perform a pre-training process on the enhanced low-light image data and infrared image data to obtain intermediate low-light image data and intermediate infrared image data. A secondary fusion is performed based on the intermediate data of the low-light image and the intermediate data of the infrared image to obtain a fused image; The image fusion module further includes a feature extraction unit, which is used to input the low-light image intermediate data and the infrared image intermediate data into the feature extraction unit to extract features and obtain two corresponding high-dimensional feature vectors. The feature fusion unit is used to integrate two high-dimensional feature vectors to obtain a single high-dimensional feature vector; the image reconstruction unit is used to reconstruct the image from the high-dimensional feature vector to generate a single-channel fused image.

6. The apparatus according to claim 5, characterized in that, The image fusion module also includes a pre-training unit; The first pre-training unit includes two Swing Transformer-based autoencoders, one of which includes a low-light encoder and a low-light decoder, and the other includes an infrared encoder and an infrared decoder.

7. The apparatus according to claim 6, characterized in that, The image fusion module further includes a feature extraction unit and an image reconstruction unit; The light encoder and infrared encoder in the feature extraction unit have the same network structure and parameters as the light encoder and infrared encoder in the first pre-training unit. They are used to extract features from the input three-channel source image and output a high-dimensional feature vector. The image reconstruction unit adopts a Swing Transformer-based model, and has the same network structure as the decoder in the first pre-training unit.

8. The apparatus according to claim 5, characterized in that, The parameters of the feature fusion unit and the image reconstruction unit are obtained through secondary training; L1 loss and gradient loss are used to constrain the convergence process of the network of the feature fusion unit and the image reconstruction unit.

Images