A method for constructing an image enhancement large model for harsh environments

Abstract

The application relates to a construction method of an image enhancement large model for a harsh environment, and belongs to the technical field of image enhancement. The application provides an image enhancement model for a harsh environment in a night dark light condition under an automatic driving scene, which can simultaneously realize the functions of dark light enhancement and rain, snow and fog denoising, improves the visibility of a single night rain and fog image by inhibiting the glow and enhancing the low light area, and improves the perception ability and driving safety of an automatic driving system in a harsh environment.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and more specifically to a method for constructing a large-scale image enhancement model for harsh environments. Background Technology

[0002] With the rapid development of autonomous driving technology, the perception capabilities of autonomous vehicles in various environments have become particularly crucial. Among these, the visual perception system, especially cameras, is one of the main ways for autonomous vehicles to acquire external information, and its performance directly affects the safety and reliability of autonomous driving. However, in harsh environments, such as low light conditions at night or rainy / foggy weather, the image quality captured by cameras will significantly decrease, seriously affecting the judgment and decision-making of the autonomous driving system.

[0003] When driving at night, insufficient light makes it difficult for cameras to capture enough information, resulting in blurry images and low contrast. This not only makes it difficult for autonomous vehicles to recognize key information such as road signs and obstacles, but may also lead to traffic accidents. Therefore, improving the brightness and contrast of nighttime images and restoring image details is of great significance for improving the nighttime driving capabilities of autonomous driving systems.

[0004] Similarly, in rainy or foggy weather, images captured by cameras are affected by raindrops and fog, resulting in blurriness and reduced contrast. These interferences not only affect the autonomous vehicle's ability to perceive its surroundings but may also mislead the system into making incorrect judgments and decisions. Therefore, removing raindrops and fog from images and restoring image clarity and contrast is crucial for improving the driving capabilities of autonomous driving systems in rainy and foggy weather.

[0005] Current image enhancement methods for harsh environments mainly fall into two categories: traditional methods and deep learning methods. Traditional methods include histogram equalization, color thresholding, and perspective transformation, but they have low accuracy and weak generalization ability. Deep learning-based image enhancement methods for harsh environments are usually only applicable to a single scene, have poor algorithm applicability, or require large-scale computing power, resulting in low computational efficiency. This makes them unsuitable for subsequent deployment and application, and they cannot be used for various sudden weather conditions in the large-scale application of autonomous vehicles, which can easily lead to vehicle accidents.

[0006] Chinese patent application CN113191977A primarily provides a method for light enhancement and rain / fog denoising based on traditional image processing. First, it obtains the dark light region and rain / fog region of interest through object detection. Then, it uses histogram equalization, a traditional image processing technique, to adjust pixel values ​​and reduce the impact of noise from dark light or rain / fog on the image. However, this method is limited to specific scenarios and cannot meet the image enhancement needs of complex environments in multiple scenarios. Summary of the Invention

[0007] In view of the above problems, this invention provides a method for constructing a large-scale image enhancement model for harsh environments. It proposes an image enhancement model for harsh environments such as low light conditions at night in autonomous driving scenarios, which can simultaneously achieve low light enhancement and rain, snow and fog denoising. It improves the visibility of a single nighttime rain and fog image by suppressing glow and enhancing low light areas; thereby enhancing the perception capability and driving safety of autonomous driving systems in harsh environments.

[0008] This invention provides a method for constructing a large-scale image enhancement model for harsh environments, comprising: Step 1: Collect video data to create a dataset of images of harsh environments; Preferably, the specific steps for creating a harsh environment image dataset from the video data collected in step 1 include: Cameras were installed on the data collection vehicle to collect dynamic video data of various roads during rainy and foggy periods; dynamic video data of the same roads were also collected during sunny weather. A dataset of harsh environment images was created by selecting image data from various scenes and different lighting conditions in the dynamic video data of the various roads during rain and fog. Furthermore, step 1 also includes ground truth values ​​for the harsh environment image dataset; Furthermore, the specific steps for obtaining the ground truth of the severe environment image dataset include: obtaining a clear weather image dataset based on the dynamic video data under clear weather conditions; and using the clear weather image dataset as the ground truth of the severe environment image dataset.

[0009] Step 2: Create an image data training set from the harsh environment image dataset; Preferably, step 2, which involves creating an image data training set from the harsh environment image dataset, includes the following specific steps: Multiple harsh environment image data in the harsh environment image dataset are classified to obtain various types of image datasets; the classified image datasets include five categories: dim night scenes, night glare and sudden light change scenes, rainy night scenes, foggy night scenes, and snowy night scenes. In each type of image dataset, 10,000 images were selected to create an image data training set; Furthermore, step 2 also includes a test set; a portion of the image training set is selected as the test set; Furthermore, 5,000 images of different states and scenes were randomly selected from the image training set as the test set.

[0010] Step 3: Establish a bright / dark image pair dataset based on the aforementioned harsh environment image dataset; Preferably, step 3, which involves establishing the bright / dark image pair dataset, includes the following specific steps: Multiple nighttime images were obtained based on the aforementioned harsh environment image dataset; The brightness of adverse light sources in each nighttime image is detected based on a light source perception network; the brightness of adverse light sources in the nighttime image specifically refers to light source glow and / or haze brightness. A light source map is obtained based on the light source brightness of multiple nighttime images; The atmospheric point spread function is converted to a 2D format to obtain the updated atmospheric point spread function; Each nighttime image is convolved on the light source map using the updated atmospheric point spread function and the corresponding light source consistency loss to enhance the image brightness, resulting in a brightness-enhanced image for each nighttime image. Each nighttime image is fused pixel-by-pixel with its corresponding brightness enhancement image to obtain a light enhancement image for each nighttime image. A dataset of bright and dark image pairs was created based on multiple nighttime images and their corresponding light-enhanced images.

[0011] It is understood that the brightness-enhanced image is a low-light or glare image; For example, the light source is an active light source, which can produce strong light in a hazy night scene, such as streetlights, car headlights and building lights; Preferably, the expression for the brightness of the adverse light source in the night image is:

[0012] in, Representing nighttime images x The expression for the brightness of the light source, Representing nighttime images x Overall brightness of the mid-scene (excluding the effects of severe weather conditions). Representing nighttime images x Atmospheric algorithms for light source mapping detection Representing nighttime images x Light transmission time, ,in It is the extinction coefficient. Representing nighttime images x The distance of light transmission; Representing nighttime images x The light source brightness is represented by APSF, which stands for Atmospheric Point Diffusion Function.

[0013] Preferably, the expression for the light source consistency loss is:

[0014] in, Representing nighttime images x Loss of light source consistency Nighttime images x The corresponding clean image, Nighttime images x The mask image, where ⊙ represents the element-wise multiplication operation. This is a map of light sources.

[0015] Furthermore, step 3 also includes obtaining the light source consistency loss for each nighttime image, specifically including: Nighttime images Thresholding is performed to generate an initial light source mask. ´ ; Based on the initial light source mask, the night image Distinguish the light sources to obtain the corresponding light source regions; The initial light source mask is processed using alpha matting technology. ´ After fine-tuning, a mask image is obtained. ; Nighttime images The light source area and the corresponding mask image Element-wise multiplication is used to obtain the light source uniformity loss. .

[0016] Step 4: Design the non-diffusion translation branch (NTB) in the initial diffusion model to obtain the diffusion model; The diffusion model is trained based on the bright and dark image dataset described in step 3 to obtain an updated diffusion model; It is understood that the diffusion model is used to obtain similarity scores between unpaired rain / fog images and clear images; Preferably, step 4, which involves obtaining the updated diffusion model, includes the following specific steps: The multiple bright and dark image pairs in the dataset described in step 3 are sequentially input into the diffusion model for training, thereby obtaining an updated diffusion model. Preferably, the initial diffusion model is a denoised diffusion probability model (DDPM). Preferably, the non-diffusion translation branch includes network structure one and network structure two; network structure one and network structure two have the same structure and parameters; network structure one and network structure two are used to input unpaired rain and fog images respectively. y and sunny images x ; The non-diffusion translation branch is used to translate unpaired rain and fog images. y and sunny images x Perform translation estimation to obtain unpaired rain and fog images. y and sunny images x Similarity score; Furthermore, the specific steps of the translation estimation include: The unpaired rain and fog image is calculated pixel-by-pixel based on network structure one and network structure two. y and sunny images x The pixel interpolation is used to obtain a similarity score between the unpaired rain / fog image and the clear image.

[0017] Preferably, the diffusion generator of the diffusion model is a U-Net network structure.

[0018] Step 5: Design a gradient adaptive convolution structure and embed the gradient adaptive convolution structure into the update diffusion model described in Step 4 to obtain the initial image enhancement model; It is understood that the initial image enhancement model is based on the unpaired rain and fog images. y and sunny images x Similarity scores and gradient-adaptive convolutional structures are used to extract unpaired rain and fog images. y and sunny images x Similar features.

[0019] Preferably, the gradient adaptive convolution structure is expressed as follows:

[0020] in, Indicates the input image i feature, i=1,2,3…I,I Represents the total number of images, input image i Includes nighttime and non-nighttime images; Indicates the input image j , j∈I , For the input image in × The weights in the convolution window; Indicates the input image i In the spatial dimension of two-dimensional space; Indicates the input image j In the spatial dimension of two-dimensional space Indicates the index of the spatial dimension of an array with a two-dimensional spatial offset.

[0021] Step 6: Input multiple image data from the image data training set described in Step 2 into the initial image enhancement model for training to obtain the image enhancement model; Preferably, step 6 further includes evaluating the image enhancement model based on a test set.

[0022] For example, the present invention further includes step 7: selecting nighttime images from the adverse environment image dataset and inputting them into the image enhancement model to perform rain and fog removal reconstruction to obtain an updated nighttime image; it can be understood that the updated nighttime image is an image after rain and fog removal processing; The inner contour edges of the updated night image are optimized and enhanced using a self-supervised gradient loss method, and the restored image corresponding to the night image is output.

[0023] Preferably, the self-supervised gradient loss expression is:

[0024] in, Nighttime images x The self-supervised gradient loss value; This represents the nighttime image output by the network. x Images after removing rain, snow, and fog. Represents nighttime images input from the network. x Images of harsh environmental noise, Representing nighttime images x Image after removing rain, snow, and fog; Representing nighttime images x Images of harsh environmental noise.

[0025] The expression for updating the nighttime image is:

[0026] in, To update the image x， Reconstructing rain and fog images; This represents the translation estimation network function based on the parameter φB of the nighttime defogging image; This represents a nighttime image without rain or fog (image without rain or fog). Furthermore, the nighttime defogging image expression is as follows:

[0027] in, x This is a nighttime image (with rain and fog). This represents the translation estimation network function based on the nighttime image parameter φA.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects: (1) This invention creates a dataset of bright and dark image pairs based on a dataset of harsh environment images and the APSF function, eliminating the need to collect the corresponding ground truth values, which greatly reduces the difficulty of data collection and thus improves the efficiency of data collection. (2) This invention proposes a denoising network for harsh environment images based on a diffusion model. For different scenes and different degrees of rain and fog images, a rain and fog denoising diffusion model is designed, and a non-diffusion translation branch is designed. Based on semi-supervised learning, the problem of no ground truth in training is solved, which greatly improves the final denoising effect for harsh environments. (3) The present invention designs a gradient adaptive convolution structure, extracts the inner contour and edge features of the image based on the adaptive convolution kernel, and performs secondary optimization and image enhancement based on these features, thus solving the problem of blurred edges in previous image enhancement results. Attached Figure Description

[0029] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] Picture 1 This is a schematic diagram of an image enhancement method for harsh environments based on an image upscaling model in an embodiment of the present invention; Picture 2 This is a schematic diagram illustrating an example of a harsh environment dataset in an embodiment of the present invention; Picture 3 This is a schematic diagram of the process for generating bright and dark image matching pairs in an embodiment of the present invention; Picture 4 This is a schematic diagram of the gradient adaptive convolution structure in an embodiment of the present invention. Detailed Implementation

[0031] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0032] A specific embodiment of the present invention, such as Picture 1-4 This invention discloses a method for constructing a large-scale image enhancement model for harsh environments. To illustrate the effectiveness of the proposed method, a specific embodiment is provided below for detailed explanation of the above-mentioned technical solution. The specific implementation steps are as follows: Step 1: Collect video data to create a dataset of images of harsh environments; Preferably, the specific steps for creating a harsh environment image dataset from the video data collected in step 1 include: Cameras were installed on the data collection vehicle to collect dynamic video data of various roads during rainy and foggy periods; dynamic video data of the same roads were also collected during sunny weather. A dataset of harsh environment images was created by selecting image data from various scenes and different lighting conditions in the dynamic video data of the various roads during rain and fog. Furthermore, step 1 also includes ground truth values ​​for the harsh environment image dataset; Furthermore, the specific steps for obtaining the ground truth of the severe environment image dataset include: obtaining a clear weather image dataset based on the dynamic video data under clear weather conditions; and using the clear weather image dataset as the ground truth of the severe environment image dataset.

[0033] Step 2: Create an image data training set from the harsh environment image dataset; Preferably, step 2, which involves creating an image data training set from the harsh environment image dataset, includes the following specific steps: Multiple harsh environment image data in the harsh environment image dataset are classified to obtain various types of image datasets; the classified image datasets include five categories: dim night scenes, night glare and sudden light change scenes, rainy night scenes, foggy night scenes, and snowy night scenes. In each type of image dataset, 10,000 images were selected to create an image data training set; Furthermore, step 2 also includes a test set; a portion of the image training set is selected as the test set; Furthermore, 5,000 images of different states and scenes were randomly selected from the image training set as the test set.

[0034] Step 3: Establish a bright / dark image pair dataset based on the aforementioned harsh environment image dataset; Preferably, step 3, which involves establishing the bright / dark image pair dataset, includes the following specific steps: Multiple nighttime images were obtained based on the aforementioned harsh environment image dataset; The brightness of adverse light sources in each nighttime image is detected based on a light source perception network; the brightness of adverse light sources in the nighttime image specifically refers to light source glow and / or haze brightness. A light source map is obtained based on the light source brightness of multiple nighttime images; The atmospheric point spread function is converted to a 2D format to obtain the updated atmospheric point spread function; Each nighttime image is convolved on the light source map using the updated atmospheric point spread function and the corresponding light source consistency loss to enhance the image brightness, resulting in a brightness-enhanced image for each nighttime image. Each nighttime image is fused pixel-by-pixel with its corresponding brightness enhancement image to obtain a light enhancement image for each nighttime image. A dataset of bright and dark image pairs was created based on multiple nighttime images and their corresponding light-enhanced images.

[0035] It is understood that the brightness-enhanced image is a low-light or glare image; For example, the light source is an active light source, which can produce strong light in a hazy night scene, such as streetlights, car headlights and building lights; Preferably, the expression for the brightness of the adverse light source in the night image is:

[0036] in, Representing nighttime images x The expression for the brightness of the light source, Representing nighttime images x Overall brightness of the mid-scene (excluding the effects of severe weather conditions). Representing nighttime images x Atmospheric algorithms for light source mapping detection Representing nighttime images x Light transmission time, ,in It is the extinction coefficient. Representing nighttime images x The distance of light transmission; Representing nighttime images x The light source brightness is represented by APSF, which stands for Atmospheric Point Diffusion Function.

[0037] Preferably, the expression for the light source consistency loss is:

[0038] in, Representing nighttime images x Loss of light source consistency Nighttime images x The corresponding clean image, Nighttime images x The mask image, where ⊙ represents the element-wise multiplication operation. This is a map of light sources.

[0039] Furthermore, step 3 also includes obtaining the light source consistency loss for each nighttime image, specifically including: Nighttime images Thresholding is performed to generate an initial light source mask. ´ ; Based on the initial light source mask, the night image Distinguish the light sources to obtain the corresponding light source regions; The initial light source mask is processed using alpha matting technology. ´ After fine-tuning, a mask image is obtained. ; Nighttime images The light source area and the corresponding mask image Element-wise multiplication is used to obtain the light source uniformity loss. .

[0040] Step 4: Design the non-diffusion translation branch (NTB) in the initial diffusion model to obtain the diffusion model; The diffusion model is trained based on the bright and dark image dataset described in step 3 to obtain an updated diffusion model; It is understood that the diffusion model is used to obtain similarity scores between unpaired rain / fog images and clear images; Preferably, step 4, which involves obtaining the updated diffusion model, includes the following specific steps: The multiple bright and dark image pairs in the dataset described in step 3 are sequentially input into the diffusion model for training, thereby obtaining an updated diffusion model. Preferably, the initial diffusion model is a denoised diffusion probability model (DDPM). Preferably, the non-diffusion translation branch includes network structure one and network structure two; network structure one and network structure two have the same structure and parameters; network structure one and network structure two are used to input unpaired rain and fog images respectively. y and sunny images x ; The non-diffusion translation branch is used to translate unpaired rain and fog images. y and sunny images x Perform translation estimation to obtain unpaired rain and fog images. y and sunny images x Similarity score; Furthermore, the specific steps of the translation estimation include: The unpaired rain and fog image is calculated pixel-by-pixel based on network structure one and network structure two. y and sunny images x The pixel interpolation is used to obtain a similarity score between the unpaired rain / fog image and the clear image.

[0041] Preferably, the diffusion generator of the diffusion model is a U-Net network structure.

[0042] Step 5: Design a gradient adaptive convolution structure and embed the gradient adaptive convolution structure into the update diffusion model described in Step 4 to obtain the initial image enhancement model; It is understood that the initial image enhancement model is based on the unpaired rain and fog images. y and sunny images x Similarity scores and gradient-adaptive convolutional structures are used to extract unpaired rain and fog images. y and sunny images x Similar features.

[0043] Preferably, the gradient adaptive convolution structure is expressed as follows:

[0044] in, Indicates the input image i feature, i=1,2,3…I,I Represents the total number of images, input image i Includes nighttime and non-nighttime images; Indicates the input image j , j∈I , For the input image in × The weights in the convolution window; Indicates the input image i In the spatial dimension of two-dimensional space; Indicates the input image j In the spatial dimension of two-dimensional space Indicates the index of the spatial dimension of an array with a two-dimensional spatial offset.

[0045] Step 6: Input multiple image data from the image data training set described in Step 2 into the initial image enhancement model for training to obtain the image enhancement model; Preferably, step 6 further includes evaluating the image enhancement model based on a test set.

[0046] For example, the present invention further includes step 7: selecting nighttime images from the adverse environment image dataset and inputting them into the image enhancement model to perform rain and fog removal reconstruction to obtain an updated nighttime image; it can be understood that the updated nighttime image is an image after rain and fog removal processing; The inner contour edges of the updated night image are optimized and enhanced using a self-supervised gradient loss method, and the restored image corresponding to the night image is output.

[0047] Preferably, the self-supervised gradient loss expression is:

[0048] in, Nighttime images x The self-supervised gradient loss value; This represents the nighttime image output by the network. x Images after removing rain, snow, and fog. Represents nighttime images input from the network. x Images of harsh environmental noise, Representing nighttime images x Image after removing rain, snow, and fog; Representing nighttime images x Images of harsh environmental noise.

[0049] The expression for updating the nighttime image is:

[0050] in, To update the image x， Reconstructing rain and fog images; This represents the translation estimation network function based on the parameter φB of the nighttime defogging image; This represents a nighttime image without rain or fog (image without rain or fog). Furthermore, the nighttime defogging image expression is as follows:

[0051] in, x This is a nighttime image (with rain and fog). This represents the translation estimation network function based on the nighttime image parameter φA.

[0052] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a large-scale image enhancement model for harsh environments, characterized in that, include: Step 1: Collect video data to create a dataset of images of harsh environments; Step 2: Create an image data training set from the harsh environment image dataset; Step 3: Establish a bright / dark image pair dataset based on the aforementioned harsh environment image dataset. Specific steps include: Multiple nighttime images were obtained based on the aforementioned harsh environment image dataset; Detect the brightness of severe light sources in each nighttime image; A light source map is obtained based on the brightness of severe light sources from multiple nighttime images; The atmospheric point spread function is format-converted to obtain the updated atmospheric point spread function; the light source consistency loss for each nighttime image is obtained; the expression for the light source consistency loss is: in, Representing nighttime images x Loss of light source consistency Nighttime images x The corresponding clean image, Nighttime images x The mask image, where ⊙ represents the element-wise multiplication operation. Map of light sources; Each nighttime image is convolved on the light source map using the updated atmospheric point spread function and the corresponding light source consistency loss, and the image brightness is enhanced to obtain a brightness-enhanced image corresponding to each nighttime image. Each nighttime image is fused with its corresponding brightness enhancement image to obtain a light enhancement image for each nighttime image; A dataset of bright and dark image pairs was created based on multiple nighttime images and corresponding light-enhanced images; Step 4: Design a non-diffusion translation branch in the initial diffusion model to obtain the diffusion model; The diffusion model is trained on the bright and dark image dataset described in step 3 to obtain an updated diffusion model; the non-diffusion translation branch includes network structure one and network structure two; The non-diffusion translation branch is used to perform translation estimation on unpaired rain / fog images and clear images to obtain similarity scores between unpaired rain / fog images and clear images; Step 5: Design a gradient adaptive convolution structure and embed the gradient adaptive convolution structure into the update diffusion model described in Step 4 to obtain the initial image enhancement model; The expression for the gradient adaptive convolution structure is: in, Indicates the input image i feature, i=1,2,3…I,I Represents the total number of images, input image i Includes nighttime and non-nighttime images; Indicates the input image j , j∈I , For the input image in × The weights in the convolution window; Indicates the input image i In two-dimensional space; Indicates the input image j In the spatial dimension of two-dimensional space Indicates the index of the spatial dimension of an array with a two-dimensional spatial offset; Step 6: Input multiple image data from the image data training set described in Step 2 into the initial image enhancement model for training to obtain the image enhancement model.

2. The method for constructing a large-scale image enhancement model for harsh environments according to claim 1, characterized in that, The brightness-enhanced image is a low-light or glare image.

3. The method for constructing a large-scale image enhancement model for harsh environments according to claim 2, characterized in that, The intensity of adverse light sources in each nighttime image refers to the brightness of the nighttime image's glow and / or haze.

4. The method for constructing a large-scale image enhancement model for harsh environments according to claim 3, characterized in that, Step 3, obtaining the light source consistency loss for each nighttime image, specifically includes the following steps: Thresholding is applied to nighttime images to generate an initial light source mask; Based on the initial light source mask, the nighttime image is distinguished by light sources to obtain the corresponding light source regions; The initial light source mask is refined to obtain a mask image; The light source consistency loss is obtained by performing element-wise multiplication between the light source region of the night image and the corresponding mask image.

5. The method for constructing a large-scale image enhancement model for harsh environments according to claim 4, characterized in that, This includes step 7: selecting nighttime images from the adverse environment image dataset and inputting them into the image enhancement model to perform rain and fog removal and reconstruction, thereby obtaining updated nighttime images; The inner contour edges of the updated night image are optimized and enhanced using a self-supervised gradient loss method, and the restored image corresponding to the night image is output.

6. The method for constructing a large-scale image enhancement model for harsh environments according to claim 5, characterized in that, The expression for the self-supervised gradient loss is: in, Nighttime images x The self-supervised gradient loss value; This represents the nighttime image output by the network. x Images after removing rain, snow, and fog. Represents nighttime images input from the network. x Images of harsh environmental noise, Representing nighttime images x Image after removing rain, snow, and fog; Representing nighttime images x Images of harsh environmental noise.

Images