Image processing method based on infrared and visible light fusion for dark scene

Through the dark scene image processing method based on the Restormer model, combined with low light enhancement and infrared recovery network, anti-noise processing and multimodal feature fusion are performed, and the problem of poor fusion effect of infrared and visible light images in dark scenes is solved, achieving high contrast Fusion of image generation with color fidelity.

CN118887101AActive Publication Date: 2024-11-01CENT SOUTH UNIV

Patent Information

Application Number
CN202410929508.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-01
Estimated Expiration
2044-07-11

AI Technical Summary

Technical Problem

In dark scenes, the existing infrared and visible image fusion method is blurred and color distorted under low light conditions, making information unable to be effectively extracted, resulting in poor fusion effect.

Method used

The dark scene image processing method based on the Restormer model is adopted, and image preprocessing is performed through the low light enhancement network submodule and the infrared recovery network submodule, and multimodal feature fusion is performed by combining the frozen infrared and visible light encoder with the channel attention space chimeric module. , trained using anti-noise processing algorithms and loss function constraint models to generate high-contrast and color fidelity fusion images.

Benefits of technology

Generating fusion images with high contrast and color fidelity under a variety of degradation conditions improves visual quality and ensures detailed texture and color fidelity of the fusion results, in line with human perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887101A_ABST
    Figure CN118887101A_ABST
Patent Text Reader

Abstract

The invention discloses an infrared and visible light fusion-based image processing method for a dark scene, and the method comprises the following steps: obtaining existing image data, and obtaining a low-light data set and a dual-light fusion data set; carrying out anti-noise processing on the low-light data set to obtain a low-light training data set; constructing a dark scene image processing initial model based on infrared and visible light fusion; training the dark scene image processing initial model by using the dual-light fusion data set and the low-light training data set to obtain a dark scene image processing model; and carrying out actual image processing by using a dark scene image processing model. According to the method, a fused image with high contrast and color fidelity can be generated under various degradation conditions. According to the method, the maximum selection and the visual fidelity strategy are combined in the loss function of model training, so that the fusion result has good visual quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an image processing method based on infrared and visible light fusion for dark scenes. Background Art

[0002] Due to the inherent technical limitations of different devices and the complexity of real-world scenes, images taken by a single sensor often lack the ability to fully represent the scene. Therefore, in order to obtain images with high-quality visual effects in extreme environments, researchers fuse original images of different modalities to obtain fused images containing more scene information, among which infrared and visible light fusion is the most widely used image fusion method.

[0003] Infrared images can retain the thermal radiation information of the scene in extreme environments, but they lack texture details and have low spatial resolution. In contrast, visible light images contain rich texture information and are consistent with human eye perception, but their image quality is greatly affected by the environment. Combining the high-contrast pixel information of infrared images with the texture information of visible light can improve the visual effect of the image and enhance the scene representation ability.

[0004] Infrared visible light fusion technology can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods map the source image to the transform domain through an artificially designed mathematical model, and finally use the inverse transform to obtain the final fusion result. Representative traditional methods include multi-scale transform-based methods, sparse representation-based methods, subspace-based methods, saliency-based methods, and hybrid methods. Although the development of traditional methods has become complete, the following problems still exist: artificially designed models cannot meet the needs of complex scenes; fusion algorithms with high time complexity cannot achieve real-time fusion.

[0005] In recent years, infrared and visible light fusion methods based on deep learning have become the mainstream method. By learning the potential connections between images, efficient feature extraction is achieved, and high visual effects are obtained that exceed traditional methods. In the field of infrared and visible light fusion technology, deep learning-based methods include methods based on convolutional neural networks, methods based on autoencoders, and methods based on generative adversarial networks. Although existing deep learning algorithms have surpassed traditional methods in the field of image fusion, the fusion effect is often poor when the scene is in dark conditions. Insufficient illumination and a large amount of sensor noise under low-light conditions destroy the texture of visible light images, resulting in the algorithm being able to extract information only from existing degraded scenes, causing texture blur and color distortion. Summary of the invention

[0006] The object of the present invention is to provide an image processing method based on infrared and visible light fusion for dark scenes.

[0007] The present invention provides an image processing method based on infrared and visible light fusion for dark scenes, comprising the following steps:

[0008] S1. Acquire existing image data to obtain a low-light dataset and a dual-light fusion dataset;

[0009] S2. Perform anti-noise processing on the low-light data set obtained in step S1 to obtain a low-light training data set;

[0010] S3. Based on the fusion of infrared and visible light, an initial model for dark scene image processing is constructed;

[0011] S4. Using the dual-light fusion data set obtained in step S1 and the low-light training data set obtained in step S2, the initial model for dark scene image processing obtained in step S3 is trained to obtain a dark scene image processing model;

[0012] S5. Use the dark scene image processing model obtained in step S4 to perform actual image processing.

[0013] The anti-noise processing described in step S2 is expressed using the following formula:

[0014]

[0015] Among them, V is the low-light data set; n is additive Gaussian white noise; It is a low-light training dataset.

[0016] The dark scene image processing initial model described in step S3 includes a first stage model and a second stage model;

[0017] The initial model for dark scene image processing uses the first-stage model to perform low-light enhancement and infrared contrast improvement on the input image data, and then inputs the obtained results into the second-stage model for multimodal feature fusion and reconstruction, and finally obtains the output result of the initial model for dark scene image processing.

[0018] The first-stage model includes a low-light enhancement network submodule and an infrared restoration network submodule;

[0019] Based on the Restormer model, a low-light enhancement network submodule and an infrared restoration network submodule are constructed; the low-light enhancement network submodule has the same structure as the infrared restoration network submodule; the submodule includes an encoder part and a decoder part; the encoder part includes a first convolutional layer module, a first Restormer module, a first downsampling module, a second Restormer module, a second downsampling module, a third Restormer module, and a third downsampling module connected in series in sequence; the output of the third downsampling module is used as the input of the decoder part; the decoder part includes a fourth Restormer module, a first upsampling module, a fifth Restormer module, a second upsampling module, and a sixth Restormer module connected in series in sequence , the third upsampling module, the seventh Restormer module, the second convolutional layer module; the fifth Restormer module uses the output of the first upsampling module and the output of the third Restormer module as input; the sixth Restormer module uses the output of the second upsampling module and the output of the second Restormer module as input; the seventh Restormer module uses the output of the third upsampling module and the output of the first Restormer module as input; the outputs of the first Restormer module, the second Restormer module, the third Restormer module and the third downsampling module of the encoder part of the two submodules are extracted to obtain infrared multi-scale features respectively. Low-light multi-scale features The output of the decoder part in the low-light enhancement network submodule is used as the output of the low-light enhancement network submodule; the output of the decoder part in the infrared restoration network submodule is used as the output of the infrared restoration network submodule;

[0020] The first downsampling module to the third downsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series; the first upsampling module to the third upsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series;

[0021] The first convolutional layer module has the same structure as the second convolutional layer module, including convolutional blocks, instance normalization, and GRLU activation functions connected in series. The convolutional block consists of a convolution kernel, which moves on the input image in a sliding window manner to calculate the dot product at each position to generate a feature map.

[0022] The structures of the first to seventh Restormer modules are the same;

[0023] The input of the low-light enhancement network submodule is the low-light image data; the input of the infrared restoration network submodule is the infrared image data;

[0024] The second stage model includes an image fusion module and a fused image decoding module;

[0025] The image fusion module includes frozen infrared and visible light encoders and a channel attention spatial mosaic module; the image fusion module uses low-light multi-scale features Infrared multi-scale features As input;

[0026] The frozen infrared and visible light encoders are the same as the encoder parts of the low-light enhancement network submodule and the infrared restoration network submodule in the first stage model after training; the frozen infrared and visible light encoders are used to input low-light multi-scale features. Infrared multi-scale features Perform feature extraction to obtain deep features The frozen infrared and visible light encoders only participate in the model training process;

[0027] The channel attention spatial embedding module includes an average pooling branch, a maximum pooling branch, and a spatial embedding submodule; the input low-light multi-scale features Infrared multi-scale features Extended features obtained by channel concatenation Extended Features The results are processed by the maximum pooling branch and the average pooling branch respectively, and then added pixel by pixel, and then combined with the expanded feature Perform pixel-by-pixel multiplication to obtain channel attention features

[0028] The spatial embedding submodule transforms the input low-light multi-scale features Infrared multi-scale features Processing to obtain spatial mosaic features Spatial embedding is achieved by calculating the difference matrix between features, which can be expressed as follows:

[0029]

[0030] Where i is the different scales of the feature vector and i = 1, 2, 3, 4; ⊙ is the pixel-by-pixel multiplication; A i is the difference matrix, expressed using the following formula:

[0031] A i =tanh(M i )⊙sigmoid(C i )

[0032] Among them, tanh(·) is the tanh activation function; sigmoid(·) is the sigmoid activation function; ⊙ is the pixel-by-pixel multiplication; M i is the Manhattan distance matrix, Ci is the cosine similarity matrix, and the elements in the matrix are calculated using the following formula:

[0033]

[0034] in, is the infrared feature vector of position (x, y); is the twilight feature vector at position (x, y); is the pixel value of the position (x, y) in the cth dimension in the infrared multi-scale feature; The pixel value at position (x, y) in the cth dimension in the low-light multi-scale feature; M i (x,y) is and Manhattan distance of C i (x,y) is and cosine similarity; N is the number of feature channels; ||·||1 is the absolute value constraint;

[0035] Finally, the channel attention spatial embedding module integrates the channel attention features With spatial mosaic features By comparing the differences between infrared and visible light features at different distances, the network can better aggregate the complementary information of different modalities and obtain fusion features. Use the following formula to express it:

[0036]

[0037] Where i is the different scales of the feature vector and i = 1, 2, 3, 4;

[0038] The structure of the fusion image decoding module is the same as the decoder part of the low-light enhancement network submodule or the infrared restoration network submodule in the first stage model; the input of the fusion image decoding module is the fusion feature The decoder reconstructs the fused image with normal illumination as the output of the model.

[0039] The training described in step S4 includes pre-training of the first-stage model, fine-tuning of the pre-trained low-light enhancement network sub-module, and training of the second-stage model; the initial model for dark scene image processing first completes the pre-training of the first-stage model, then fine-tunes the pre-trained low-light enhancement network sub-module, and finally completes the training of the second-stage model to obtain a dark scene image processing model.

[0040] The pre-training of the first stage model includes the pre-training of the low-light enhancement network sub-module and the pre-training of the infrared restoration network sub-module;

[0041] The low-light enhancement network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula:

[0042]

[0043] Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used:

[0044]

[0045] in, is the output of the low-light enhancement network submodule; V normal is the corresponding normal illumination image;

[0046] The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula:

[0047]

[0048] Among them, SSIM(·) is the structural similarity loss, which is calculated using the following formula:

[0049]

[0050] Among them, (x, y) is the image pair to be calculated; μ k , k∈{x,y} is the mean of the image pair (x,y); σ k , k∈{x,y} is the variance of the image pair (x,y); σ xy is the covariance between images; c1 and c2 are constants;

[0051] It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula:

[0052]

[0053] Among them, Y is the output of the low-light enhancement network submodule; ▽ x and y is a first-order gradient operator; H is the feature height; W is the feature width; c is the color channel, c∈{R,G,B}, R is the red channel, G is the green channel, and B is the blue channel;

[0054] The infrared restoration network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula:

[0055]

[0056] Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used:

[0057]

[0058] in, is the output of the infrared restoration network submodule; I stretch To stretch the contrast of the infrared image, the following formula is used:

[0059] I stretch =αI+(1-α)I mean

[0060] Among them, I mean is the average value of the infrared image; α is the intensity adjustment parameter; I is the input of the infrared restoration network submodule;

[0061] The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula:

[0062]

[0063] It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula:

[0064]

[0065] Where Y is the output of the infrared restoration network submodule; x and y is a first-order gradient operator;

[0066] Fine-tune the pre-trained low-light enhancement network submodule, and the loss function is expressed using the following formula:

[0067]

[0068] Among them, λ4 and λ5 are weight coefficients; To fine-tune the strength loss, is the structural loss, expressed using the following formula:

[0069]

[0070] in, It is the output of the low-light enhancement network submodule; is the adjusted pseudo-true value;

[0071] The second stage model is trained, and its total loss function lfuse As shown below:

[0072]

[0073] Among them, λ6, λ7, λ8 and λ9 are weight coefficients; is the second stage strength loss, For visual fidelity loss, is the gradient loss, is the color loss, expressed using the following formula:

[0074]

[0075] in, is the output of the low-light enhancement network submodule; P is the fused image output by the model; Cb p Refers to the Cb channel in the YCbCr color space; Cr p Refer to the Cr channel in the YCbCr color space; Output the Cb channel of the network in the YCbCr color space; is the Cr channel of the network output in the YCbCr color space; max(·) is the maximum value operation; ▽P is the gradient of the output image; ▽I is the gradient of the infrared image; ▽V is the gradient of the low-light image; It is the gradient output by the low-light enhancement network submodule;

[0076] and It ensures that the fusion result contains information from both visible light and infrared modalities; Make sure the network retains all the detailed textures in the source image; The constrained model learns the color fidelity capability; by constraining the loss of the second-stage model, the complementary information from the infrared and visible light modalities is effectively fused, making the fusion result visually consistent with human perception;

[0077] After training is completed, a dark scene image processing model is obtained.

[0078] The present invention discloses an image processing method based on infrared and visible light fusion for dark scenes, which can produce fused images with high contrast and color fidelity under various degradation conditions. The method of the present invention combines maximum selection and visual fidelity strategies in the loss function of model training, so that the fusion result has good visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the process of the present invention;

[0080] Figure 2The comparison of the results of the method of the present invention and the existing direct fusion method in the LLVIP scenario in the embodiment;

[0081] Figure 3 The comparison of the results of the method of the present invention and the existing direct fusion method in the MSRS scenario in the embodiment;

[0082] Figure 4 The comparison of the results of the method of the present invention and the existing enhanced fusion method in the LLVIP scenario in the embodiment;

[0083] Figure 5 The results of the method of the present invention and the existing enhanced fusion method in the MSRS scenario are compared in the embodiment. DETAILED DESCRIPTION

[0084] The present invention provides an image processing method based on infrared and visible light fusion for dark scenes, and its flow chart is as follows: Figure 1 As shown, the following steps are included:

[0085] S1. Acquire existing image data to obtain a low-light dataset and a dual-light fusion dataset;

[0086] S2. Perform anti-noise processing on the low-light data set obtained in step S1 to obtain a low-light training data set;

[0087] The anti-noise processing is expressed using the following formula:

[0088]

[0089] Among them, V is the low-light data set; n is additive Gaussian white noise; This is a low-light training dataset. (Please confirm whether this part is accurate)

[0090] S3. Based on the fusion of infrared and visible light, an initial model for dark scene image processing is constructed;

[0091] The dark scene image processing initial model includes a first stage model and a second stage model;

[0092] The initial model for dark scene image processing uses the first-stage model to perform low-light enhancement and infrared contrast improvement on the input image data, and then inputs the obtained results into the second-stage model for multimodal feature fusion and reconstruction, and finally obtains the output result of the initial model for dark scene image processing.

[0093] The first-stage model includes a low-light enhancement network submodule and an infrared restoration network submodule;

[0094] Based on the Restormer model, a low-light enhancement network submodule and an infrared restoration network submodule are constructed; the low-light enhancement network submodule has the same structure as the infrared restoration network submodule; the submodule includes an encoder part and a decoder part; the encoder part includes a first convolutional layer module, a first Restormer module, a first downsampling module, a second Restormer module, a second downsampling module, a third Restormer module, and a third downsampling module connected in series in sequence; the output of the third downsampling module is used as the input of the decoder part; the decoder part includes a fourth Restormer module, a first upsampling module, a fifth Restormer module, a second upsampling module, and a sixth Restormer module connected in series in sequence , the third upsampling module, the seventh Restormer module, the second convolutional layer module; the fifth Restormer module uses the output of the first upsampling module and the output of the third Restormer module as input; the sixth Restormer module uses the output of the second upsampling module and the output of the second Restormer module as input; the seventh Restormer module uses the output of the third upsampling module and the output of the first Restormer module as input; the outputs of the first Restormer module, the second Restormer module, the third Restormer module and the third downsampling module of the encoder part of the two submodules are extracted to obtain infrared multi-scale features respectively. Low-light multi-scale features The output of the decoder part in the low-light enhancement network submodule is used as the output of the low-light enhancement network submodule; the output of the decoder part in the infrared restoration network submodule is used as the output of the infrared restoration network submodule;

[0095] The first downsampling module to the third downsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series; the first upsampling module to the third upsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series;

[0096] The first convolutional layer module has the same structure as the second convolutional layer module, including convolutional blocks, instance normalization, and GRLU activation functions connected in series. The convolutional block consists of a convolution kernel, which moves on the input image in a sliding window manner to calculate the dot product at each position to generate a feature map.

[0097] The structures of the first to seventh Restormer modules are the same; the Restormer modules are prior art;

[0098] The input of the low-light enhancement network submodule is the low-light image data; the input of the infrared restoration network submodule is the infrared image data;

[0099] The second stage model includes an image fusion module and a fused image decoding module;

[0100] The image fusion module includes frozen infrared and visible light encoders and a channel attention spatial mosaic module; the image fusion module uses low-light multi-scale features Infrared multi-scale features As input;

[0101] The frozen infrared and visible light encoders are the same as the encoder parts of the low-light enhancement network submodule and the infrared restoration network submodule in the first stage model after training; the frozen infrared and visible light encoders are used to input low-light multi-scale features. Infrared multi-scale features Perform feature extraction to obtain deep features

[0102] The channel attention spatial embedding module includes an average pooling branch, a maximum pooling branch, and a spatial embedding submodule; the input low-light multi-scale features Infrared multi-scale features Extended features obtained by channel concatenation Extended Features The results are processed by the maximum pooling branch and the average pooling branch respectively, and then added pixel by pixel, and then combined with the expanded feature Perform pixel-by-pixel multiplication to obtain channel attention features

[0103] The spatial embedding submodule transforms the input low-light multi-scale features Infrared multi-scale features Processing to obtain spatial mosaic features Spatial embedding is achieved by calculating the difference matrix between features, which can be expressed as follows:

[0104]

[0105] Where i is the different scales of the feature vector and i = 1, 2, 3, 4; ⊙ is the pixel-by-pixel multiplication; A i is the difference matrix, expressed using the following formula:

[0106] A i =tanh(M i )⊙sigmoid(C i )

[0107] Among them, tanh(·) is the tanh activation function; sigmoid(·) is the sigmoid activation function; ⊙ is the pixel-by-pixel multiplication; M i is the Manhattan distance matrix, C iis the cosine similarity matrix, and the elements in the matrix are calculated using the following formula:

[0108]

[0109] in, is the infrared feature vector of position (x, y); is the twilight feature vector at position (x, y); is the pixel value of the position (x, y) in the cth dimension in the infrared multi-scale feature; The pixel value at position (x, y) in the cth dimension in the low-light multi-scale feature; M i (x,y) is and Manhattan distance of C i (x,y) is and cosine similarity; N is the number of feature channels; ||·||1 is the absolute value constraint;

[0110] Finally, the channel attention spatial embedding module integrates the channel attention features With spatial mosaic features By comparing the differences between infrared and visible light features at different distances, the network can better aggregate the complementary information of different modalities and obtain fusion features. Use the following formula to express it:

[0111]

[0112] Where i is the different scales of the feature vector and i = 1, 2, 3, 4;

[0113] The structure of the fusion image decoding module is the same as the decoder part of the low-light enhancement network submodule or the infrared restoration network submodule in the first stage model; the input of the fusion image decoding module is the fusion feature The decoder reconstructs the fused image with normal illumination as the output of the model.

[0114] S4. Use the dual-light fusion dataset obtained in step S1 and the low-light training dataset obtained in step S2 to train the initial model for dark scene image processing obtained in step S3 to obtain a dark scene image processing model; the initial model for dark scene image processing first completes the pre-training of the first-stage model, then fine-tunes the pre-trained low-light enhancement network sub-module, and finally completes the training of the second-stage model to obtain a dark scene image processing model.

[0115] The training includes pre-training of the first-stage model, fine-tuning of the pre-trained low-light enhancement network submodule, and training of the second-stage model;

[0116] The pre-training of the first stage model includes the pre-training of the low-light enhancement network sub-module and the pre-training of the infrared restoration network sub-module;

[0117] The low-light enhancement network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula:

[0118]

[0119] Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used:

[0120]

[0121] in, is the output of the low-light enhancement network submodule; V normal is the corresponding normal illumination image;

[0122] The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula:

[0123]

[0124] Among them, SSIM(·) is the structural similarity loss, which is calculated using the following formula:

[0125]

[0126] Among them, (x, y) is the image pair to be calculated; μ k , k∈{x,y} is the mean of the image pair (x,y); σ k , k∈{x,y} is the variance of the image pair (x,y); σ xy is the covariance between images; c1 and c2 are constants;

[0127] It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula:

[0128]

[0129] Among them, Y is the output of the low-light enhancement network submodule; ▽ x and y is a first-order gradient operator; H is the feature height; W is the feature width; c is the color channel, c∈{R,G,B}, R is the red channel, G is the green channel, and B is the blue channel;

[0130] The infrared restoration network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula:

[0131]

[0132] Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used:

[0133]

[0134] in, is the output of the infrared restoration network submodule; I stretch To stretch the contrast of the infrared image, the following formula is used:

[0135] I stretch =αI+(1-α)I mean

[0136] Among them, I mean is the average value of the infrared image; α is the intensity adjustment parameter; I is the input of the infrared restoration network submodule;

[0137] The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula:

[0138]

[0139] It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula:

[0140]

[0141] Where Y is the output of the infrared restoration network submodule; x and y is a first-order gradient operator;

[0142] Fine-tune the pre-trained low-light enhancement network submodule, and the loss function is expressed using the following formula:

[0143]

[0144] Among them, λ4 and λ5 are weight coefficients; To fine-tune the strength loss, is the structural loss, expressed using the following formula:

[0145]

[0146] in, It is the output of the low-light enhancement network submodule; is the adjusted pseudo-true value;

[0147] The second stage model is trained, and its total loss function l fuse As shown below:

[0148]

[0149] Among them, λ6, λ7, λ8 and λ9 are weight coefficients; is the second stage strength loss, For visual fidelity loss, is the gradient loss, is the color loss, expressed using the following formula:

[0150]

[0151]

[0152] in, is the output of the low-light enhancement network submodule; P is the fused image output by the model; Cb p Refers to the Cb channel in the YCbCr color space; Cr p Refer to the Cr channel in the YCbCr color space; Output the Cb channel of the network in the YCbCr color space; is the Cr channel of the network output in the YCbCr color space; max(·) is the maximum value operation; ▽P is the gradient of the output image; ▽I is the gradient of the infrared image; ▽V is the gradient of the low-light image; It is the gradient output by the low-light enhancement network submodule;

[0153] and It ensures that the fusion result contains information from both visible light and infrared modalities; Make sure the network retains all the detailed textures in the source image; The constrained model learns the color fidelity capability; by constraining the loss of the second-stage model, the complementary information from the infrared and visible light modalities is effectively fused, making the fusion result visually consistent with human perception;

[0154] After training is completed, a dark scene image processing model is obtained.

[0155] S5. Use the dark scene image processing model obtained in step S4 to perform actual image processing.

[0156] The method of the present invention is further described below in conjunction with an embodiment:

[0157] In order to verify the fusion performance of the method of the present invention in a low-light environment, the existing direct fusion method and enhanced fusion method are used to process the images in the LLVIP scene and the MSRS scene, and the resulting images are compared. Among them, the direct fusion method does not consider the degradation phenomenon of the source image, but only considers the ability of the algorithm to fuse complementary information. The comparison in this embodiment uses CDDFuse, U2Fusion, PIAFusion, MetaFusion, DDFM, Dif-Fusion and SwinFusion as comparison benchmarks. The comparison results are shown in the figure below. Figure 2 , Figure 3 The enhanced fusion method obtains a fused image with high-quality visual effects through image enhancement operations. The comparison in this embodiment uses MHLP, DIVFusion, DDBF, Text-IF, and three latest low-light enhancement networks URetinex-Net, PairLIE, and MIRNet-v2 as the comparison benchmarks. The comparison results are shown in the figure below. Figure 4 , Figure 5 shown.

Claims

1. An image processing method based on infrared and visible light fusion for dark scenes, characterized in that: The following steps are involved: S1. Acquire existing image data to obtain a low-light dataset and a dual-light fusion dataset; S2. Perform anti-noise processing on the low-light dataset obtained in step S1 to obtain a low-light training dataset; S3. Based on the fusion of infrared and visible light, an initial model for dark scene image processing is constructed; S4. Using the dual-light fusion data set obtained in step S1 and the low-light training data set obtained in step S2, the initial model for dark scene image processing obtained in step S3 is trained to obtain a dark scene image processing model; S5. Use the dark scene image processing model obtained in step S4 to perform actual image processing.

2. The image processing method based on infrared and visible light fusion for dark scenes according to claim 1, characterized in that: The anti-noise processing described in step S2 is expressed using the following formula: Among them, V is the low-light data set; n is additive Gaussian white noise; It is a low-light training dataset.

3. The image processing method based on infrared and visible light fusion for dark scenes according to claim 2, characterized in that: The dark scene image processing initial model described in step S3 includes a first stage model and a second stage model; The initial model for dark scene image processing uses the first-stage model to perform low-light enhancement and infrared contrast improvement on the input image data, and then inputs the obtained results into the second-stage model for multimodal feature fusion and reconstruction, and finally obtains the output result of the initial model for dark scene image processing.

4. The image processing method based on infrared and visible light fusion for dark scenes according to claim 3, characterized in that: The first-stage model includes a low-light enhancement network submodule and an infrared restoration network submodule; Based on the Restormer model, the low-light enhancement network submodule and the infrared restoration network submodule are constructed; The structure of the low-light enhancement network submodule is the same as that of the infrared restoration network submodule; the submodule includes an encoder part and a decoder part; the encoder part includes a first convolutional layer module, a first Restormer module, a first downsampling module, a second Restormer module, a second downsampling module, a third Restormer module, and a third downsampling module connected in series in sequence; the output of the third downsampling module is used as the input of the decoder part; the decoder part includes a fourth Restormer module, a first upsampling module, a fifth Restormer module, a second upsampling module, a sixth Restormer module, a third upsampling module, a seventh Restormer module, and a second convolutional layer module connected in series in sequence; the fifth Restormer module uses the output of the first upsampling module and the output of the third Restormer module as input; The sixth Restormer module uses the output of the second upsampling module and the output of the second Restormer module as input; The seventh Restormer module uses the output of the third upsampling module and the output of the first Restormer module as input; the outputs of the first Restormer module, the second Restormer module, the third Restormer module and the third downsampling module of the encoder part of the two submodules are extracted to obtain infrared multi-scale features respectively. Low-light multi-scale features The output of the decoder part in the low-light enhancement network submodule is used as the output of the low-light enhancement network submodule; The output of the decoder part in the infrared restoration network submodule is used as the output of the infrared restoration network submodule; The first downsampling module to the third downsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series; the first upsampling module to the third upsampling module have the same structure, including a convolutional layer, instance normalization, and a GRLU activation function connected in series; The first convolutional layer module has the same structure as the second convolutional layer module, including convolutional blocks, instance normalization, and GRLU activation functions connected in series. The convolutional block consists of a convolution kernel, which moves on the input image in a sliding window manner to calculate the dot product at each position to generate a feature map. The structures of the first to seventh Restormer modules are the same; The input of the low-light enhancement network submodule is low-light image data; The input of the infrared restoration network submodule is infrared image data.

5. The image processing method based on infrared and visible light fusion for dark scenes according to claim 4, characterized in that: The second stage model includes an image fusion module and a fused image decoding module; The image fusion module includes frozen infrared and visible light encoders and a channel-attention spatial embedding module; The image fusion module uses low-light multi-scale features Infrared multi-scale features As input; The frozen infrared and visible light encoders are the same as the encoder parts of the low-light enhancement network submodule and the infrared restoration network submodule in the first stage model after training; the frozen infrared and visible light encoders are used to input low-light multi-scale features. Infrared multi-scale features Perform feature extraction to obtain deep features The frozen infrared and visible light encoders only participate in the model training process; The channel attention spatial embedding module includes an average pooling branch, a maximum pooling branch, and a spatial embedding submodule; the input low-light multi-scale features Infrared multi-scale features Extended features obtained by channel concatenation Extended Features The results are processed by the maximum pooling branch and the average pooling branch respectively, and then added pixel by pixel, and then combined with the expanded feature Perform pixel-by-pixel multiplication to obtain channel attention features The spatial embedding submodule transforms the input low-light multi-scale features Infrared multi-scale features Processing to obtain spatial mosaic features Spatial embedding is achieved by calculating the difference matrix between features, which can be expressed as follows: Where i is the different scales of the feature vector and i = 1, 2, 3, 4; ⊙ is the pixel-by-pixel multiplication; A i is the difference matrix, expressed using the following formula: A i =tanh(M i )⊙sigmoid(C i ) Among them, tanh(·) is the tanh activation function; sigmoid(·) is the sigmoid activation function; ⊙ is the pixel-by-pixel multiplication; M i is the Manhattan distance matrix, C i is the cosine similarity matrix, and the elements in the matrix are calculated using the following formula: in, is the infrared feature vector of position (x, y); is the twilight feature vector at position (x, y); is the pixel value of the position (x, y) in the cth dimension in the infrared multi-scale feature; The pixel value at position (x, y) in the cth dimension in the low-light multi-scale feature; M i (x,y) is and Manhattan distance of C i (x,y) is and cosine similarity; N is the number of feature channels; ||·||1 is the absolute value constraint; Finally, the channel attention spatial embedding module integrates the channel attention features With spatial mosaic features By comparing the differences between infrared and visible light features at different distances, the network can better aggregate the complementary information of different modalities and obtain fusion features. Use the following formula to express it: Where i is the different scales of the feature vector and i = 1, 2, 3, 4; The structure of the fusion image decoding module is the same as the decoder part of the low-light enhancement network submodule or the infrared restoration network submodule in the first stage model; the input of the fusion image decoding module is the fusion feature The decoder reconstructs the fused image with normal illumination as the output of the model.

6. The image processing method based on infrared and visible light fusion for dark scenes according to claim 5, characterized in that: The training includes pre-training of the first-stage model, fine-tuning of the pre-trained low-light enhancement network submodule, and training of the second-stage model; the initial model for dark scene image processing first completes the pre-training of the first-stage model, then fine-tunes the pre-trained low-light enhancement network submodule, and finally completes the training of the second-stage model to obtain a dark scene image processing model.

7. The image processing method based on infrared and visible light fusion for dark scenes according to claim 6, characterized in that: The pre-training of the first stage model includes the pre-training of the low-light enhancement network sub-module and the pre-training of the infrared restoration network sub-module; The low-light enhancement network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula: Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used: in, is the output of the low-light enhancement network submodule; V normal is the corresponding normal illumination image; The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula: Among them, SSIM(·) is the structural similarity loss, which is calculated using the following formula: Among them, (x, y) is the image pair to be calculated; μ k , k∈{x,y} is the mean of the image pair (x,y); σ k , k∈{x,y} is the variance of the image pair (x,y); σ xy is the covariance between images; c1 and c2 are constants; It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula: Among them, Y is the output of the low-light enhancement network submodule; ▽ x and y is a first-order gradient operator; H is the feature height; W is the feature width; c is the color channel, c∈{R,G,B}, R is the red channel, G is the green channel, and B is the blue channel; The infrared restoration network submodule is pre-trained in a supervised manner, with the normal illumination image as the reference truth value, and the loss function is expressed using the following formula: Among them, λ1, λ2 and λ3 are weight coefficients; To guide the learning of the intensity distribution of the reference image, the following formula is used: in, is the output of the infrared restoration network submodule; I stretch To stretch the contrast of the infrared image, the following formula is used: I stretch =αI+(1-α)I mean Among them, I mean is the average value of the infrared image; α is the intensity adjustment parameter; I is the input of the infrared restoration network submodule; The pre-training of the low-light enhancement network submodule is constrained at the structural level and expressed using the following formula: It is used to constrain the gradient smoothness of the output image and suppress noise, and is expressed using the following formula: Where Y is the output of the infrared restoration network submodule; x and y is a first-order gradient operator; H is the feature height; W is the feature width; c is the color channel, c∈{R,G,B}, R is the red channel, G is the green channel, and B is the blue channel.

8. The image processing method based on infrared and visible light fusion for dark scenes according to claim 7, characterized in that: Fine-tune the pre-trained low-light enhancement network submodule, and the loss function is expressed using the following formula: Among them, λ4 and λ5 are weight coefficients; To fine-tune the strength loss, is the structural loss, expressed using the following formula: in, It is the output of the low-light enhancement network submodule; is the adjusted false true value.

9. The image processing method based on infrared and visible light fusion for dark scenes according to claim 8, characterized in that: The second stage model is trained, and its total loss function l fuse As shown below: Among them, λ6, λ7, λ8 and λ9 are weight coefficients; is the second stage strength loss, For visual fidelity loss, is the gradient loss, is the color loss, expressed using the following formula: in, is the output of the low-light enhancement network submodule; P is the fused image output by the model; Cb p Refers to the Cb channel in the YCbCr color space; Cr p Refer to the Cr channel in the YCbCr color space; Output the Cb channel of the network in the YCbCr color space; is the Cr channel of the network output in the YCbCr color space; max(·) is the maximum value operation; ▽P is the gradient of the output image; ▽I is the gradient of the infrared image; ▽V is the gradient of the low-light image; It is the gradient output by the low-light enhancement network submodule; and It ensures that the fusion result contains information from both visible light and infrared modalities; Make sure the network retains all the detailed textures in the source image; The constrained model learns color fidelity capabilities; by constraining the loss of the second-stage model, the complementary information from the infrared and visible light modalities is effectively fused, making the fusion result visually consistent with human perception; after completing the training, a dark scene image processing model is obtained.

Citation Information

Patent Citations

  • Road intelligent monitoring system based on multi-view multispectral fusion

    CN113609942A

  • Multi-scene dual-band image fusion method and device and storable medium

    CN118037561A

  • Feature decomposition-based infrared image and visible light image fusion method

    CN118134780A

  • Medical image segmentation method based on u-net

    US20220309674A1

Cited By

  • Intelligent camera dual-light image fusion method and dual-light integrated intelligent camera

    CN121010510A

  • Intelligent camera dual-light image fusion method and dual-light integrated intelligent camera

    CN121010510B