Low-light image enhancement method and device based on infrared visible light information integration

Abstract

The invention provides a low-light image enhancement method and device based on infrared visible light information integration, and belongs to the technical field of image enhancement, and the method comprises the steps: carrying out the processing of a normal illumination image through a series of image degradation methods based on image augmentation, and obtaining a pseudo low-light image needed by training; enabling the pseudo low-light image and the corresponding infrared image to pass through a submerged space and a multi-layer encoder to obtain a deep multi-scale feature, and building a multi-modal information interaction module based on sparse cross attention to process a dual-modal variable-scale feature and enhance feature interaction fusion; and a dual-mode residual feature enhancement diffusion model is established, and the feature enhancement diffusion model comprises a feature splicing module and a series of multi-mode information interaction modules, so that complementary features are learned and fused in the denoising process, the features are mapped to a normal exposure domain, and deep integration of image information is realized. According to the method, low-light images under various exposure conditions can be effectively enhanced, and the robustness is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and in particular relates to a low-light image enhancement method and device based on the integration of infrared and visible light information. Background Art

[0002] Image enhancement is a key technology in the field of computer vision. Image enhancement plays a vital role in many fields, including medical imaging, remote sensing, video surveillance, autonomous driving, and artistic creation. Traditional image enhancement methods use various image transformation and filtering algorithms to improve image visibility and eliminate image noise. However, such methods have poor robustness and are difficult to adapt to complex shooting scenes. In order to solve the defects of traditional image enhancement methods, deep learning-based methods have emerged. However, existing deep learning enhancement methods usually rely on information input from a single visible light modality and cannot handle the problem of visible light image information loss under extreme conditions. Summary of the invention

[0003] The purpose of the present invention is to propose a low-light image enhancement method and device based on the integration of infrared and visible light information, which can effectively repair scenes under any exposure conditions and has strong robustness.

[0004] The present invention is achieved through the following technical solutions:

[0005] A low-light image enhancement method based on infrared and visible light information integration comprises the following steps:

[0006] Step S1, processing the normal light image through a series of image degradation methods based on image augmentation to obtain a pseudo low light image required for training to guide subsequent model training;

[0007] Step S2, the pseudo low-light image and its corresponding infrared image are passed through the latent space and multi-layer encoder to obtain deep multi-scale features, and the above dual-modal variable-scale features are processed through a multi-modal information interaction module based on sparse cross-attention, so as to realize feature interactive fusion adaptively by a deformable cross-attention mechanism;

[0008] Step S3: construct a dual-mode residual feature enhancement diffusion model. The feature enhancement diffusion model includes a series of multimodal information interaction modules to learn to fuse complementary features during the denoising process and map the features to the normal exposure domain to achieve deep integration of image information.

[0009] Furthermore, the step S1 specifically includes the following steps:

[0010] Step S11: Given a normal exposure image, according to the formula The gamma transform is used to reduce the image brightness and visibility, and damage the original information in some relatively low-light areas. orepresents the original normally exposed image, γ represents the gamma parameter of the gamma image conversion algorithm, I g Represents the image after gamma transformation.

[0011] Step S12: According to formula I c =αI g The image I is transformed by contrast algorithm g Processing is performed to reduce the image contrast and further reduce the overall visibility, where α is the contrast coefficient. When α < 1, the transformation reduces the image contrast. c represents the image after contrast transformation;

[0012] Step S12: According to formula I v =I c +N(0,σ)+P(λ) adds Gaussian and Poisson noise to the image to further degrade the image quality and simulate electronic and photographic noise at night;

[0013] Furthermore, in step S2, the visible light image I v and infrared image I i The representation L is obtained by pre-training the latent space encoder ε(·) v =ε(I v ) and L i =ε(I i ), L v and L i Then, the multi-scale features are obtained by encoding through a multi-scale encoder based on a convolutional structure and a Transformer. and

[0014] Furthermore, in step S2, the feature Intermediate features in the denoising process of the diffusion model According to the formula Projected into the multi-head space to obtain a more comprehensive feature representation, where W Q ,W V is the linear mapping operator, m is the attention head index, and M is the total number of heads.

[0015] Furthermore, in step S2, the feature The fusion formula is expressed as Where k is the index of feature points involved in deformable attention, K is the total number of feature points, and r i is the relative position of feature point i anchored on Q, W m is a linear mapping operator, A k (·) and O k (·) are the linear operators for calculating the weight and offset of the kth sampling point respectively.

[0016] Furthermore, in step S3, the network architecture of the diffusion model is used to achieve deep feature understanding fusion. In the diffusion process, the original image I is converted into o The latent space representation z is obtained through the latent space encoder 0 =ε(I o ), then, noise is added to the latent space representation of the original image through the forward diffusion process, which is expressed as Where T is the total number of diffusion steps, z t For z 0 The representation after adding noise, α t ∈(0,1) is used to adjust the noise intensity added during the diffusion process, and I is related to z t Identity matrices of the same dimensions.

[0017] Furthermore, in step S3, during the reverse diffusion process, the noise signal must first be compared with the visible light and infrared features L encoded by the latent space encoder. v =ε(I v ), L i =ε(I i ) to obtain the initial intermediate feature F m 0 , achieving the preliminary fusion of noise signals with infrared and visible light information.

[0018] Furthermore, in step S3, during the reverse diffusion process, the model is encoded by a U-shaped coding network ∈ θ (z t ,t,C) realizes noise prediction, where ∈ θ For the U-shaped encoding network, C contains infrared and visible light image information as conditional input, t is the current step number, and the model is passed through the loss function Optimize the network.

[0019] In the further step S3, the intermediate feature Mapped to multi-scale feature representation in U-shaped encoding network The infrared and visible light features are then iterated multiple times through a deformable cross-attention mechanism.

[0020] The present invention is also achieved through the following technical solutions:

[0021] A low-light image enhancement device based on infrared and visible light information integration includes a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions in the memory to implement the following modules:

[0022] Feature acquisition module: obtains multi-level features of infrared images and visible light images respectively. The multi-level features have low-level features with rich color and texture information and structural features with rich contextual semantic information;

[0023] Feature fusion module: constructs a feature memory pool, stores the multi-level features corresponding to infrared and visible light samples in the normal feature memory pool, and concatenates the stored features with the noise features to obtain the initial fusion features. The deformable cross-attention mechanism is used to achieve deep fusion of infrared and visible light features with noise features.

[0024] Original image reconstruction module: Build an iterative inverse diffusion network composed of superimposed U-shaped encoders. Each U-shaped encoder can independently realize feature splicing and feature deep fusion, understand the infrared and visible light image information in a multi-step iterative process, and thus predict the noise signal added in the diffusion process to obtain an accurate original image representation.

[0025] The present invention has the following beneficial effects:

[0026] 1. The present invention first processes the visible light image under normal illumination through a series of image degradation methods based on image augmentation to obtain the pseudo low-light visible light image required for training, guide the subsequent model training, and realize the enhancement and fusion of multimodal information. Then, the pseudo low-light image and its corresponding infrared image are passed through a latent space encoder to obtain the latent space representation of the two-modal image. The latent space representation is spliced ​​with the noise signal obtained in the diffusion process to complete the preliminary fusion of the two modalities. On the other hand, the latent space representation of the two-modal image is passed through a multi-scale encoder to obtain multi-scale features, and is deeply fused with the features after preliminary fusion through a deformable cross-attention mechanism to realize the effective interaction of the two-modal information. The noise in the diffusion process is iteratively predicted through a U-shaped coding network to realize the restoration of the original image, thereby enhancing the robustness and providing a new method for solving the problem of low-light image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention is further described in detail below with reference to the accompanying drawings.

[0028] Figure 1 Flow chart of the method of the present invention.

[0029] Figure 2 It is a principle block diagram of the device of the present invention. DETAILED DESCRIPTION

[0030] like Figure 1 and Figure 2 As shown, a low-light image enhancement method and device based on infrared and visible light information integration includes the following steps:

[0031] Step S1, processing the normal light image through a series of image degradation methods based on image augmentation to obtain a pseudo low light image required for training to guide subsequent model training;

[0032] Specifically, given a normal exposure image I o , according to formula I g =I o γ The gamma transform is used to reduce the image brightness and visibility, and damage the original information in some relatively low-light areas. o represents the original normally exposed image, γ represents the gamma parameter of the gamma image conversion algorithm, I g Represents the image after gamma transformation. According to formula I c =αI g The image I is transformed by contrast algorithm g Processing is performed to reduce the image contrast and further reduce the overall visibility, where α is the contrast coefficient. When α < 1, the transformation reduces the image contrast. c Represents the image after contrast transformation. According to formula I v =I c +N(0,σ)+P(λ) adds Gaussian and Poisson noise to the image to further degrade the image quality and simulate electronic and camera noise at night.

[0033] The pseudo low-light images generated by this method have the characteristics of low overall brightness and contrast, and missing information in some areas. By taking the pseudo low-light images and their corresponding infrared images as the input of the model, the model is trained in a way to restore the normal exposure images. This process can effectively guide the model to restore the normal exposure images by fusing the infrared modal information. At the same time, in order to further enhance the brightness and contrast of the image and predict the color information of the infrared modality to improve the perceptual quality of the restored image, the model also enhances the internal information of the modality.

[0034] Step S2, the pseudo low-light image and its corresponding infrared image are passed through the latent space and multi-layer encoder to obtain deep multi-scale features, and the above dual-modal variable-scale features are processed through a multi-modal information interaction module based on sparse cross-attention, so as to realize feature interactive fusion adaptively by a deformable cross-attention mechanism;

[0035] Specifically, for the visible light image I v and infrared image I i The representation L is obtained by pre-training the latent space encoder ε(·) v =ε(I v ) and L i =ε(I i ), L v and L iThen, the multi-scale features are obtained by encoding through a multi-scale encoder based on a convolutional structure and a Transformer. and The characteristics Intermediate features in the denoising process of the diffusion model According to the formula Projected into the multi-head space to obtain a more comprehensive feature representation, where W Q ,W V is a linear mapping operator, m is the attention head index, and M is the total number of heads. The feature The fusion formula is expressed as Where k is the index of feature points involved in deformable attention, K is the total number of feature points, and r i is the relative position of feature point i anchored on Q, W m is a linear mapping operator, A k (·) and O k (·) are the linear operators for calculating the weight and offset of the kth sampling point respectively.

[0036] Through this deformable cross attention mechanism, the intermediate features Iteratively analyze and process infrared and visible light fusion. In each iteration, the intermediate features containing the prior information of visible light and infrared selectively perform weighted fusion operations on some feature points on the visible light or infrared feature map to achieve effective interaction of the two modal information.

[0037] Step S3: construct a dual-mode residual feature enhancement diffusion model. The feature enhancement diffusion model includes a series of multimodal information interaction modules to learn to fuse complementary features during the denoising process and map the features to the normal exposure domain to achieve deep integration of image information.

[0038] Specifically, we first build the network architecture of the diffusion model. In the diffusion process, in order to reduce the computational complexity, the original image I o The latent space representation z is obtained through the latent space encoder 0 =ε(I o ), and then, through the forward diffusion process, the latent space representation z of the original image is 0 Add noise, the formula is expressed as Where T is the total number of diffusion steps, z t For z 0 The representation after adding noise, α t ∈(0,1) is used to adjust the noise intensity added during the diffusion process, and I is related to z t The unit matrix of the same dimension, after T steps of noise addition, the latent space representation z of the original image 0 is degenerated into a Gaussian-like distribution.

[0039] In the reverse diffusion process, the noise signal must first be compared with the visible light and infrared features L encoded by the latent space encoder. v =ε(I v ), L i =ε(I i ) to obtain the initial intermediate features The initial fusion of noise signal and infrared and visible light information is realized. In the process of inverse diffusion and noise prediction, the model uses a U-shaped coding network ∈ θ (z t ,t,C) realizes noise prediction, where ∈ θ For the U-shaped encoding network, C contains the infrared and visible light image information as conditional input, and t is the current step number. The model is implemented through the loss function Realize the optimization of the network. Intermediate features Mapping to multi-scale feature representation in U-shaped encoding network The infrared and visible light features are then iterated multiple times through a deformable cross-attention mechanism.

[0040] Correspondingly, the low-light image enhancement device based on the integration of infrared and visible light information includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:

[0041] Feature acquisition module: obtains multi-level features of infrared images and visible light images respectively. The multi-level features have low-level features with rich color and texture information and structural features with rich contextual semantic information;

[0042] Feature fusion module: constructs a feature memory pool, stores the multi-level features corresponding to infrared and visible light samples in the normal feature memory pool, and concatenates the stored features with the noise features to obtain the initial fusion features. The deformable cross-attention mechanism is used to achieve deep fusion of infrared and visible light features with noise features.

[0043] Original image reconstruction module: Build an iterative inverse diffusion network composed of superimposed U-shaped encoders. Each U-shaped encoder can independently realize feature splicing and feature deep fusion, understand the infrared and visible light image information in a multi-step iterative process, and thus predict the noise signal added in the diffusion process to obtain an accurate original image representation.

[0044] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A low-light image enhancement method based on the integration of infrared and visible light information, characterized in that: The steps include: Step S1, processing the normal light image through a series of image degradation methods based on image augmentation to obtain a pseudo low light image required for training to guide subsequent model training; Step S2, the pseudo low-light image and its corresponding infrared image are passed through the latent space and multi-layer encoder to obtain deep multi-scale features, and the above dual-modal variable-scale features are processed through a multi-modal information interaction module based on sparse cross-attention, so as to realize feature interactive fusion adaptively by a deformable cross-attention mechanism; Step S3: construct a dual-mode residual feature enhancement diffusion model. The feature enhancement diffusion model includes a series of multimodal information interaction modules to learn to fuse complementary features during the denoising process and map the features to the normal exposure domain to achieve deep integration of image information.

2. The low-light image enhancement method based on infrared and visible light information integration according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: Given a normal exposure image, according to formula I g =I o γ The gamma transform is used to reduce the image brightness and visibility, and damage the original information in some relatively low-light areas. o represents the original normally exposed image, γ represents the gamma parameter of the gamma image conversion algorithm, I g represents the image after gamma transformation; Step S12: According to formula I c =αI g The image I is transformed by contrast algorithm g Processing is performed to reduce the image contrast and further reduce the overall visibility, where α is the contrast coefficient. When α < 1, the transformation reduces the image contrast. c represents the image after contrast transformation; Step S12: According to formula I v =I c +N(0,σ)+P(λ) adds Gaussian and Poisson noise to the image to further degrade the image quality and simulate electronic and camera noise at night.

3. The low-light image enhancement method based on infrared and visible light information integration according to claim 2, characterized in that: In step S2, the visible light image I v and infrared image I i The representation L is obtained by pre-training the latent space encoder ε(·) v =ε(I v ) and L i =ε(I i ), L v and L i Then, the multi-scale features are obtained by encoding through a multi-scale encoder based on a convolutional structure and a Transformer. and 4. The low-light image enhancement method based on infrared and visible light information integration according to claim 3, characterized in that: In step S2, the feature Intermediate features in the denoising process of the diffusion model According to the formula Projected into the multi-head space to obtain a more comprehensive feature representation, where W Q ,W V is the linear mapping operator, m is the attention head index, and M is the total number of heads.

5. The low-light image enhancement method based on infrared and visible light information integration according to claim 4, characterized in that: In step S2, the feature is centered by a deformable cross attention mechanism. The fusion formula is expressed as Where k is the index of feature points involved in deformable attention, K is the total number of feature points, and r i is the relative position of feature point i anchored on Q, W m is a linear mapping operator, A k (·) and O k (·) are the linear operators for calculating the weight and offset of the kth sampling point respectively.

6. The low-light image enhancement method based on infrared and visible light information integration according to claim 5, characterized in that: In step S3, the network architecture of the diffusion model is used to achieve deep feature understanding fusion, and the original image I is converted into o Through the latent space encoder, we get its latent space representation z0=ε(I o ), then, noise is added to the latent space representation of the original image through the forward diffusion process, which is expressed as Where T is the total number of diffusion steps, z t is the representation of z0 after adding noise, α t ∈(0,1) is used to adjust the noise intensity added during the diffusion process, and I is related to z t Identity matrices of the same dimensions.

7. The low-light image enhancement method based on infrared and visible light information integration according to claim 6, characterized in that: In step S3, during the reverse diffusion process, the noise signal must first be compared with the visible light and infrared features L encoded by the latent space encoder. v =ε(I v ), L i =ε(I i ) to obtain the initial intermediate features Realize the preliminary fusion of noise signal, infrared and visible light information.

8. The low-light image enhancement method based on infrared and visible light information integration according to claim 7, characterized in that: In step S3, during the reverse diffusion process, the model passes through the U-shaped encoding network ∈ θ (z t ,t,C) realizes noise prediction, where ∈ θ For the U-shaped encoding network, C contains infrared and visible light image information as conditional input, t is the current step number, and the model is passed through the loss function Optimize the network.

9. The low-light image enhancement method based on infrared and visible light information integration according to claim 8, characterized in that: In step S3, the intermediate feature Mapped to multi-scale feature representation in U-shaped encoding network The infrared and visible light features are then iterated multiple times through a deformable cross-attention mechanism.

10. A low-light image enhancement device based on the integration of infrared and visible light information, characterized in that: It includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules: Feature acquisition module: obtains multi-level features of infrared images and visible light images respectively. The multi-level features have low-level features with rich color and texture information and structural features with rich contextual semantic information; Feature fusion module: constructs a feature memory pool, stores the multi-level features corresponding to infrared and visible light samples in the normal feature memory pool, and concatenates the stored features with the noise features to obtain the initial fusion features. The deformable cross-attention mechanism is used to achieve deep fusion of infrared and visible light features with noise features. Original image reconstruction module: Build an iterative inverse diffusion network composed of superimposed U-shaped encoders. Each U-shaped encoder can independently realize feature splicing and feature deep fusion, understand the infrared and visible light image information in a multi-step iterative process, and thus predict the noise signal added in the diffusion process to obtain an accurate original image representation.

Images