Image defogging method for improving Retinex and progressive generative network

Abstract

The invention discloses an image defogging method for improving Retinex and a progressive generation network, which comprises the following steps of: decomposing an image into a reflection component and an illumination component based on an image defogging algorithm of a Retinex theory, using the reflection component as a defogged image, and optimizing the reflection component and the illumination component by adopting a joint regularization method, so that the defogged image is better in effect. WGAN-GP loss is used as discrimination loss, global and local consistency of the network is refined, an image restoration effect conforming to context semantics is generated, and the image defogging effect is further improved by using the distance between the generated data distribution and the real data distribution measured by the Wharisstein distance and optimizing an objective function. The defogged image generated by the method is more natural and clearer in visual effect, rich in details and stronger in sense of reality.

Description

Technical Field

[0001] The present invention relates to the technical field of image defogging, and in particular to an image defogging method based on improved Retinex and progressive generative network. Background Art

[0002] In today's digital age, image acquisition, monitoring and display systems play a vital role in many fields, especially in the application of driverless cars and highway monitoring systems. However, severe weather conditions, such as haze, have a serious impact on image quality, resulting in blurred image details and dim colors, thus affecting the effectiveness of the monitoring system and the safety of driverless cars.

[0003] In order to solve this problem, image defogging algorithms have emerged, aiming to remove fog from images through technical means, restore image clarity and color, and enhance visual effects. Defogging algorithms based on Retinex theory have attracted widespread attention due to their effectiveness in processing natural scene images. Retinex theory simulates the color perception mechanism of the human visual system and adjusts the color and brightness of the image to achieve the purpose of defogging.

[0004] Although the algorithm based on Retinex theory has achieved certain success in multiple application scenarios, it still faces challenges such as insufficient contrast and loss of detail information. The existence of these problems limits the practicality and effectiveness of the algorithm. Therefore, there is still a broad space for research and improvement of image dehazing technology.

[0005] As the future development direction of the automotive and transportation industries, driverless cars are of great significance for improving driving safety, reducing traffic accidents, and improving traffic efficiency. In severe weather conditions such as smog, the visual recognition ability of driverless cars will be affected, which poses a threat to driving safety. In addition, the reduced visibility of surveillance cameras in foggy weather also brings difficulties to traffic monitoring.

[0006] Therefore, developing and improving image defogging algorithms to improve the imaging quality of image acquisition equipment under adverse weather conditions is of great practical significance for ensuring the safe driving of driverless cars and improving the efficiency of monitoring systems. This can not only improve the visibility of images, but also provide clearer and more accurate data support for subsequent image analysis and information extraction. Summary of the invention

[0007] One object of the present invention is to propose an image defogging method that improves Retinex and progressive generative networks. The defogging image generated by the present invention is more natural, clear, rich in details, and more realistic in visual effect.

[0008] An image defogging method based on an improved Retinex and progressive generative network according to an embodiment of the present invention includes the following steps:

[0009] The image defogging algorithm based on Retinex theory decomposes the image into reflection component and illumination component, and uses the reflection component as the defogged image to achieve the image defogging function. The algorithm combining the reflection component and the illumination component with regularization can improve the enhanced image. The image defogging algorithm model is as follows:

[0010] L = R * T;

[0011] Among them, R is the reflection component, which represents the defogging image and contains the object information in the image; T represents the illumination component, which represents the illumination information in the image; L is the foggy image, which represents the original image containing fog;

[0012] Using the joint regularization method, the objective function is in the following form:

[0013]

[0014] Among them, Y 1 (T) represents the regularization term of T, which helps optimize the illumination component, Y 2 (T, R) represents the joint regularization term of T and R, which helps improve the image quality;

[0015] For the first dehazed image after Retinex joint estimation, WGAN-GP loss is used as the discriminant loss to refine the global and local consistency of the network, generating an image restoration effect that satisfies the contextual semantics;

[0016] Using the Wasserstein distance W(P 1 , P 2 ) measures the distance between the generated data distribution and the true data distribution and is defined as follows:

[0017]

[0018] Among them, W(P 1 , P 2 ) represents the Wasserstein distance, which measures the difference between the distribution of generated images and the distribution of real images, Π(P 1 ,P 2 ) indicates that P 1 and P 2The set of all possible joint distributions of the combination. For each possible joint distribution γ, the real sample x and the generated sample y are obtained from the sampling (x, y)~γ, and the distance between a pair of samples ||xy|| is calculated. The expected value of the distance between the samples under the joint distribution γ is calculated. WGAN implicitly learns the distribution of the potential real image. After continuous training, the distance between the generated image and the real image is reduced to complete the image defogging. At this time, the objective function is defined as:

[0019]

[0020] in, is the set of 1-Lipschitz functions, P 2 Is The model distribution defined by z~p(z).

[0021] Optionally, in the objective function, Y 1 and 2 Specifically include:

[0022]

[0023]

[0024] Among them, α represents the regularization coefficient, which is used to control the smoothness of the illumination component, β represents the regularization coefficient, which is used to control the joint optimization of the reflection component and the illumination component, N represents the total number of pixels, represents the gradient of the illumination component T, which indicates the rate of illumination change in the image. Represents the gradient of the reflection component R, indicating the change of the edge of the object in the image, Represents the gradient of the illumination component T, ε is a very small positive number.

[0025] Optionally, a gradient penalty is introduced in the objective function:

[0026]

[0027] in, is from the true distribution P 1 and the generated distribution P 2 Uniform sampling on the straight line between the sampled pairs of points. For the in-image drawing task, only the missing area is predicted, and the gradient penalty is only used for the pixels in the missing holes. At this time, its gradient penalty is expressed as;

[0028]

[0029] Where M is the mask. When a pixel is missing, M=0, which means that the gradient penalty is applied to the area. λ is the penalty coefficient. The generative model can effectively utilize the image context and semantic information to continuously restore the content close to the real image.

[0030] Optionally, in order to enhance the interactive effect of the network at each stage, the loss function also plays different constraints at different stages. In the coarse image restoration network, the structural reconstruction loss is introduced to generate a reasonable content structure in the entire restoration process. In the stage of the refined network, in order to keep the overall color and style of the generated image consistent with the real image, the structural reconstruction loss is introduced:

[0031]

[0032] Among them, the structure reconstruction loss is defined as l 1 Pixel reconstruction loss and L prec The weighted sum of perceptual losses. For the perceptual loss, the image of the repaired area is put into the feature extractor of VGG16 pre-trained by ImageNet. The perceptual loss L prec Measuring the similarity between high-level structures of the repaired area, pixel reconstruction loss l 1 It has the effect of constraining the generation of the overall pixels of the repaired image. The combination of the two ensures the structural rationality of the rough repaired image.

[0033] Optionally, the perceptual loss L prec include:

[0034]

[0035] Among them, H, W, C represent the height, weight and number of channels of each feature map, N represents the number of feature maps extracted from the VGG16 features, and style loss is introduced in the refinement network to visually refine the image:

[0036]

[0037] where Gi is a Ci×Ci Gram matrix constructed from feature maps φi.

[0038] The beneficial effects of the present invention are:

[0039] The present invention innovates and optimizes image decomposition, regularization optimization, Wasserstein distance, gradient penalty, multi-stage loss function and detail enhancement through the improved image defogging method by combining Retinex theory and progressive generative network. Finally, the generated defogging image is more natural, clear, detailed and realistic in visual effect, which not only improves the effect of image defogging, but also has a wide range of value in practical applications, such as improving image processing capabilities in the fields of autonomous driving and monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0041] Figure 1 A flowchart of an image defogging method using an improved Retinex and progressive generative network proposed by the present invention;

[0042] Figure 2 This is a comparison chart of the image dehazing method proposed by the present invention using an improved Retinex and a progressive generative network. DETAILED DESCRIPTION

[0043] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0044] refer to Figure 1-2 , an image dehazing method based on improved Retinex and progressive generative network, comprising the following steps:

[0045] The image defogging algorithm based on Retinex theory decomposes the image into reflection component and illumination component, and uses the reflection component as the defogged image to achieve the image defogging function. The algorithm combining the reflection component and the illumination component with regularization can improve the enhanced image. The image defogging algorithm model is as follows:

[0046] L = R * T;

[0047] Among them, R is the reflection component, which represents the defogging image and contains the object information in the image; T represents the illumination component, which represents the illumination information in the image; L is the foggy image, which represents the original image containing fog;

[0048] Using the joint regularization method, the objective function is in the following form:

[0049]

[0050] Among them, Y 1(T) represents the regularization term of T, which helps optimize the illumination component, Y 2 (T, R) represents the joint regularization term of T and R, which helps improve the image quality;

[0051] For the first dehazed image after Retinex joint estimation, WGAN-GP loss is used as the discriminant loss to refine the global and local consistency of the network, generating an image restoration effect that satisfies the contextual semantics;

[0052] Using the Wasserstein distance W(P 1 , P 2 ) measures the distance between the generated data distribution and the true data distribution and is defined as follows:

[0053]

[0054] Among them, W(P 1 , P 2 ) represents the Wasserstein distance, which measures the difference between the distribution of generated images and the distribution of real images, Π(P 1 ,P 2 ) indicates that P 1 and P 2 The set of all possible joint distributions of the combination. For each possible joint distribution γ, the real sample x and the generated sample y are obtained from the sampling (x, y)~γ, and the distance ‖xy|| between a pair of samples is calculated. The expected value of the distance between the samples under the joint distribution γ is calculated. WGAN implicitly learns the distribution of the potential real image. After continuous training, the distance between the generated image and the real image is reduced to complete the image defogging. At this time, the objective function is defined as:

[0055]

[0056] in, is the set of 1-Lipschitz functions, P 2 Is The model distribution defined by z~p(z).

[0057] In this embodiment, Y in the objective function 1 and Y 2 Specifically include:

[0058]

[0059]

[0060] Among them, α represents the regularization coefficient, which is used to control the smoothness of the illumination component, β represents the regularization coefficient, which is used to control the joint optimization of the reflection component and the illumination component, N represents the total number of pixels, represents the gradient of the illumination component T, which indicates the rate of illumination change in the image. Represents the gradient of the reflection component R, indicating the change of the edge of the object in the image, Represents the gradient of the illumination component T, ε is a very small positive number.

[0061] In this implementation, a gradient penalty is introduced into the objective function:

[0062]

[0063] in, is from the true distribution P 1 and the generated distribution P 2 Uniform sampling on the straight line between the sampled pairs of points. For the in-image drawing task, only the missing area is predicted, and the gradient penalty is only used for the pixels in the missing holes. At this time, its gradient penalty is expressed as;

[0064]

[0065] Where M is the mask. When a pixel is missing, M=0, which means that the gradient penalty is applied to the area. λ is the penalty coefficient. The generative model can effectively utilize the image context and semantic information to continuously restore the content close to the real image.

[0066] In this implementation, in order to enhance the interactive effect of the network at each stage, the loss function also plays different constraints at different stages. In the coarse image restoration network, the structural reconstruction loss is introduced to generate a reasonable content structure in the entire restoration process. In the stage of the refined network, in order to keep the overall color and style of the generated image consistent with the real image, the structural reconstruction loss is introduced:

[0067]

[0068] Among them, the structure reconstruction loss is defined as l 1 Pixel reconstruction loss and L prec The weighted sum of perceptual losses. For the perceptual loss, the image of the repaired area is put into the feature extractor of VGG16 pre-trained by ImageNet. The perceptual loss L prec Measuring the similarity between high-level structures of the repaired area, pixel reconstruction loss l 1 It has the effect of constraining the generation of the overall pixels of the repaired image. The combination of the two ensures the structural rationality of the rough repaired image.

[0069] In this embodiment, the perception loss L prec include:

[0070]

[0071] Among them, H, W, C represent the height, weight and number of channels of each feature map, N represents the number of feature maps extracted from the VGG16 features, and style loss is introduced in the refinement network to visually refine the image:

[0072]

[0073] where Gi is a CixCi Gram matrix constructed from feature maps φi.

[0074] Embodiment 1:

[0075] In the traffic system of modern cities, haze weather poses a serious threat to driving safety. Especially during peak hours, haze weather can cause a sharp drop in visibility, increasing the risk of traffic accidents. Traditional image defogging technology often fails to provide a sufficiently clear view in a complex traffic environment, thus failing to meet the needs of autonomous driving and intelligent transportation systems. In order to improve this situation, the present invention proposes an image defogging method based on an improved Retinex and a progressive generative network to improve image clarity in haze weather and ensure driving safety.

[0076] In a typical urban traffic monitoring system, several cameras are installed to monitor traffic conditions in real time. The images taken by these cameras in haze weather have a significant reduction in visibility due to the interference of haze, making it difficult to identify vehicles, pedestrians and traffic signs, which poses a great challenge to traffic management. In this case, the image defogging method of the present invention can significantly improve the image quality.

[0077] In the specific application process, the foggy image taken by the surveillance camera is first input into the improved Retinex and progressive generative network model. Based on the Retinex theory, the model decomposes the input image into reflection component and illumination component. The reflection component represents the image information after defogging, while the illumination component represents the lighting information in the image. By combining regularized optimization and introducing Wasserstein distance, we can continuously adjust the reflection component and illumination component during the optimization process, making the defogging image clearer.

[0078] In the specific process of image processing, we first apply the joint regularization algorithm to optimize the reflection component and the illumination component. The objective function contains regularization terms for the illumination component and the reflection component to ensure that the dehazed image is more natural and smooth visually. Then, the Wasserstein distance is used to measure the difference between the generated image distribution and the real image distribution, and the generation network is further optimized to narrow the distance between the two and ensure that the generated image is closer to the distribution of the real image.

[0079] During the experiment, we selected a busy road section in a city's traffic monitoring system as the test scene. The test time was from December 5, 2023 to December 15, 2023, during which we experienced several severe haze weathers. We collected monitoring images during these time periods, applied traditional image defogging methods and the method of the present invention to process them, and compared and analyzed the processed images.

[0080] The specific data are as follows:

[0081] 1. Improved visibility:

[0082] The visibility of images processed by the traditional method is increased by 30% on average, while the visibility of images processed by the method of the present invention is increased by 70% on average.

[0083] Experimental location: A busy road section in a city’s traffic monitoring system.

[0084] 2. Image clarity:

[0085] The average score of image clarity after processing by the traditional method is 5.5 (out of 10 points), while the average score of image clarity after processing by the method of the present invention is 8.8.

[0086] Testing time: December 5, 2023 to December 15, 2023.

[0087] 3. Traffic accident rate:

[0088] The traffic accident rate on roads where traditional methods were used during haze weather was 8.2%.

[0089] The traffic accident rate of the road section where the method of the present invention is applied under the same conditions is 3.7%.

[0090] Data source: accident statistics provided by a city’s traffic management department.

[0091] 4. Processing speed:

[0092] The average time it takes to process an image using traditional methods is 2.5 seconds.

[0093] The average time for the method of the present invention to process one image is 1.8 seconds.

[0094] Experimental equipment: the same computer, processor model is Intel Core i7, memory 16GB.

[0095] Through the above experimental data, we can see that the method of the present invention is superior to the traditional method in improving image clarity, increasing visibility, reducing traffic accident rate and processing speed. This not only solves the problem of image blur in haze weather, but also significantly improves the reliability and safety of the traffic monitoring system.

[0096] Specifically, the method of the present invention can not only provide clearer images in complex traffic environments, but also perform well in processing speed to meet the needs of real-time monitoring through improved Retinex theory and progressive generation network. Compared with traditional methods, the method of the present invention is more robust and adaptable, and can maintain consistent excellent performance under different haze conditions. The application of this technology is not only of great significance to autonomous driving and intelligent transportation systems, but also has broad application prospects in other fields that require image clarity and reliability, such as security monitoring, drone vision, etc.

[0097] The theoretical basis of the neighborhood-based Retinex algorithm of the present invention is that the intensity of the illumination component generally changes slowly, and is manifested as a low-frequency component in the frequency domain, while the reflectivity of the surface materials of different objects varies greatly, and is manifested as a high-frequency component. Therefore, this type of algorithm uses a low-pass filtering method to estimate the illumination component. For each pixel in the image, the nonlinear ratio of the pixel value to the weighted value of the pixel in the neighborhood is calculated, which is the new pixel value of the pixel. The weighted value of the pixel in the neighborhood is obtained by spatial convolution of the smoothing function, and the weight is given by the coefficient of the smoothing function. The iterative Retinex algorithm uses matrix calculation instead of path calculation. In the path calculation, pixels are added to the path in sequence, the pixel values ​​are compared in a serial manner, and the distance function is determined according to the length of the path. In the matrix calculation, the ratio and product operations can be processed in parallel, and the distance function is determined according to the number of iterations.

[0098] The present invention innovates and optimizes image decomposition, regularization optimization, Wasserstein distance, gradient penalty, multi-stage loss function and detail enhancement through the improved image defogging method by combining Retinex theory and progressive generative network. Finally, the generated defogging image is more natural, clear, detailed and realistic in visual effect, thus achieving significant beneficial effects. This not only improves the effect of image defogging, but also has a wide range of value in practical applications, such as improving image processing capabilities in the fields of autonomous driving and monitoring systems.

[0099] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An image dehazing method based on improved Retinex and progressive generative network, characterized in that: The steps include: The image defogging algorithm based on Retinex theory decomposes the image into reflection component and illumination component, and uses the reflection component as the defogged image to achieve the image defogging function. The algorithm combining the reflection component and the illumination component with regularization can improve the enhanced image. The image defogging algorithm model is as follows: L = R * T; Among them, R is the reflection component, which represents the defogging image and contains the object information in the image; T represents the illumination component, which represents the illumination information in the image; L is the foggy image, which represents the original image containing fog; Using the joint regularization method, the objective function is in the following form: Among them, Υ1(T) represents the regularization term of T, which helps optimize the illumination component, and Υ2(T,R) represents the joint regularization term of T and R, which helps improve the image quality; For the first dehazed image after Retinex joint estimation, WGAN-GP loss is used as the discriminant loss to refine the global and local consistency of the network, generating an image restoration effect that satisfies the contextual semantics; The distance between the generated data distribution and the true data distribution is measured using the Wasserstein distance W(P1,P2), which is defined as follows: Among them, W(P1,P2) represents the Wasserstein distance, which measures the difference between the generated image distribution and the real image distribution. ∏(P1,P2) represents the set of all possible joint distributions composed of P1 and P2. For each possible joint distribution γ, the real sample x and the generated sample y are obtained from the sampling (x,y)~γ, and the distance ‖xy‖ between a pair of samples is calculated. The expected value of the distance between the samples under the joint distribution γ is calculated. WGAN implicitly learns the distribution of the potential real image. After continuous training, the distance between the generated image and the real image is reduced to complete the image dehazing. At this time, the objective function is defined as: in, is a set of 1-Lipschitz functions, P2 is given by The model distribution defined by z~p(z).

2. According to claim 1, the image dehazing method of improving Retinex and progressive generative network is characterized in that: In the objective function, Y1 and Y2 specifically include: Among them, α represents the regularization coefficient, which is used to control the smoothness of the illumination component, β represents the regularization coefficient, which is used to control the joint optimization of the reflection component and the illumination component, N represents the total number of pixels, represents the gradient of the illumination component T, which indicates the rate of illumination change in the image. Represents the gradient of the reflection component R, indicating the change of the edge of the object in the image, Represents the gradient of the illumination component T, ε is a very small positive number.

3. The image dehazing method of improved Retinex and progressive generative network according to claim 2, characterized in that: A gradient penalty is introduced in the objective function: in, It is a uniform sampling on the straight line between the pairs of points sampled from the true distribution P1 and the generated distribution P2. For the in-image drawing task, only the missing area is predicted, and the gradient penalty is only used for the pixels in the missing holes. At this time, its gradient penalty is expressed as; Where M is the mask. When a pixel is missing, M=0, which means that the gradient penalty is applied to the area. λ is the penalty coefficient. The generative model can effectively utilize the image context and semantic information to continuously restore the content close to the real image.

4. The image dehazing method of improved Retinex and progressive generative network according to claim 3, characterized in that: In order to enhance the interactive effect of the network at each stage, the loss function also plays different constraints at different stages. In the rough image restoration network, the structural reconstruction loss is introduced to generate a reasonable content structure in the entire restoration process. In the stage of the refined network, in order to keep the overall color and style of the generated image consistent with the real image, the structural reconstruction loss is introduced: Among them, the structure reconstruction loss is defined as l1 pixel reconstruction loss and L prec The weighted sum of perceptual losses. For the perceptual loss, the image of the repaired area is put into the feature extractor of VGG16 pre-trained by ImageNet. The perceptual loss L prec The similarity between the high-level structures of the repaired area is measured, and the pixel reconstruction loss l1 has the effect of constraining the generation of the overall pixels of the repaired image. The combination of the two ensures the structural rationality of the rough repaired image.

5. The image dehazing method of improved Retinex and progressive generative network according to claim 4, characterized in that: The perceptual loss L prec include: Among them, H, W, C represent the height, weight and number of channels of each feature map, N represents the number of feature maps extracted from the VGG16 features, and style loss is introduced in the refinement network to visually refine the image: where Gi is a Ci×Ci Gram matrix constructed from feature maps φi.

Images