ASM and GANs-based sand and dust image non-pairing learning simulation method

Abstract

The invention provides a sand and dust image non-pairing learning simulation method based on ASM and GANs, a non-pairing learning network based on ASM and GAN is adopted to carry out sand and dust image rendering, a depth estimation network is adopted to establish an ASM model, and a generative adversarial network is adopted to carry out image enhancement respectively. And fusing the images generated by the two networks through a non-reference image fusion strategy.

Description

Technical Field

[0001] The present invention relates to the technical fields of image rendering, metaverse, creative computing, virtual reality (VR) and augmented reality (AR), and in particular to an unpaired learning simulation method for dust images based on ASM and GANs. Background Art

[0002] Image rendering plays a vital role in the fields of metaverse, creative computing, virtual reality (VR) and augmented reality (AR). In the metaverse, realistic image rendering can build a more realistic and detailed virtual world, allowing users to experience immersion; in creative computing, image rendering technology provides artists and designers with powerful tools, enabling them to achieve richer and more delicate visual creations; in the fields of VR and AR, realistic image rendering can enhance the interaction between the virtual and real worlds, bringing users a more natural and intuitive experience. The development of image rendering technology has not only improved the application effects in various fields, but also promoted the cross-integration of science and technology and art, creating new ways of creation and experience.

[0003] In recent years, deep learning has been widely used in various image enhancement tasks, such as image dehazing, image deraining, and underwater image enhancement. However, due to the lack of large-scale dust image datasets, compared with other image enhancement tasks, the research on image rendering of sandstorm weather is relatively scarce and has not attracted widespread attention from researchers. Existing general image enhancement algorithms have difficulty in preserving the original color and detailed information of the object while rendering sandstorms, and still mainly use traditional color correction methods.

[0004] Existing dust rendering algorithms are mainly divided into three categories: methods based on traditional technology, methods based on physical models, and methods based on deep learning.

[0005] Image enhancement methods based on traditional technologies. Traditional sandstorm image enhancement methods mainly solve the two problems of color shift and clarity processing. They usually restore the visual clarity of the image by equalizing colors, improving contrast, and enhancing brightness. For example, Reza et al. introduced contrast-limited adaptive histogram equalization (CLAHE) to improve the noise sensitivity problem in local areas; Cheng et al. used white balance technology to compensate for the blue channel and used guided image filtering algorithms to enhance the contrast and edge information of the image. Although these methods improve the visual clarity of the image to a certain extent, due to the staged processing and simple adjustment of pixels, they often lead to problems such as color distortion and noise amplification, and the visual fidelity of the enhanced image is poor.

[0006] Image enhancement methods based on physical models. Physical model-based methods use prior knowledge in image degradation to reversely process dust images. Such methods often regard dust images as haze images with color shift and use optical models to reconstruct clear images. For example, Shi et al. proposed a strategy that combines color balance and dark channel prior (DCP) to solve the problem of halo artifacts in the enhancement results. Yu et al. added information loss constraints based on physical models and DCP to keep the brightness of the output image constant. Although these methods are effective in dealing with color distortion and detail loss, they rely on prior knowledge and may cause noise accumulation as the calculation process proceeds, eventually generating noise points in the results.

[0007] Image enhancement methods based on deep learning. With the development of deep learning in the field of vision, researchers have begun to explore its application in image enhancement tasks. These methods train models with large-scale image datasets to reduce the human estimation error of physical model parameters and improve image restoration effects. For example, Huang et al. proposed a dust perception representation extraction module, which estimates the dust color and local spatial information to restore the clarity of the image. However, this requires a large amount of paired dust image data to train the model. To solve this problem, Ding et al. proposed an unsupervised dust image enhancement network based on priors, which reconstructs clear images with real colors and natural contrast through adversarial learning. Gao et al. introduced an end-to-end recurrent generative adversarial network and designed a joint optimization guidance module to reduce color distortion and artifact problems in reconstruction. Although unsupervised methods reduce the dependence on paired datasets, due to the variability of dust scenes, it is difficult for real datasets to cover all features, resulting in poor generalization of such algorithms.

[0008] Therefore, there are three main challenges in the existing dust image rendering methods: 1) Although traditional methods enhance the visual clarity of images by equalizing colors and improving contrast, they are prone to color distortion and noise amplification. 2) Physical model-based methods rely on prior knowledge, often causing noise accumulation problems and generating noise in the results. 3) Although deep learning-based methods reduce the dependence on physical model parameters, the generalization ability of the algorithm is poor due to the diversity of dust scenes and the limitations of data set coverage. Summary of the invention

[0009] In order to solve the defects and shortcomings of the prior art, the present invention proposes an unpaired image dust rendering learning network based on generative adversarial networks (GANs) and atmospheric scattering models. The main contributions of the present invention are as follows:

[0010] (1) Through the reference-free image fusion strategy, the image quality assessment does not require a reference indicator. Two images rendered in different ways are fused together to improve the consistency and stability of the rendering results. Experiments have shown that the color distortion and noise amplification problems that are easily caused by traditional methods can be effectively avoided.

[0011] (2) An unpaired learning network based on ASM and GANs, namely DustGAN, is proposed for dust image rendering. First, DustGAN breaks through the limitation that image enhancement based on paired learning requires clear-dust image pairing by training unpaired clear and dusty images; second, DustGAN contains two network branches, the depth estimation network and the generative adversarial network perform image enhancement respectively, and then fuse the rendered dust images. Experimental results show that DustGAN has significant advantages in dust rendering. It not only avoids the common noise accumulation problem of physical model-based methods, but also uses unpaired learning networks and no-reference image fusion strategies to generate the final results, which improves the generalization ability of various dust scenes and overcomes the limitations of deep learning-based methods.

[0012] (3) A dust image enhancement dataset SCID containing 22 different colors was constructed by using DustGAN, and experimental evaluation was performed. The experimental results show that the dataset has a high restoration ability in maintaining image quality and is close to the real dust image, which verifies the superiority of the DustGAN algorithm over existing methods. In addition, the proposed dust image enhancement dataset SCID is processed using a popular image dust removal algorithm, which verifies its effectiveness in restoring image quality.

[0013] The present invention specifically adopts the following technical solutions:

[0014] An unpaired learning simulation method for dust images based on ASM and GANs: an unpaired learning network based on ASM and GAN is used to render dust images, wherein a depth estimation network is used to establish an ASM model and a generative adversarial network is used to perform image enhancement respectively, and then the images generated by the two networks are fused through a no-reference image fusion strategy.

[0015] Furthermore, Monodepth2 is used to build an ASM model to render dust in the image, including:

[0016] Depth estimation process: load the pre-trained Monodepth2 encoder and decoder models, use the encoder to extract image features, and generate a depth map through the decoder;

[0017] Dust rendering process: normalize the predicted depth map, load the original image and dust effect map, and perform weighted fusion through the normalized value of the depth map:

[0018] J dust =0.8*(0.8*Image org *Depth normalized +1.3*Image color *(l-Depth normalized ))

[0019] Among them, J dust is the generated dust rendering image, Image org is the original image, Depth normalized is the depth map, Image color is the dust rendering image; the dust rendering result J is obtained by linearly combining the content of the original image with the dust rendering image dust .

[0020] Furthermore, the unpaired learning network comprises two branches. First, two preliminary rendering images J are generated from the two branches. dust and J refine , and then fuse them through the reference-free image fusion strategy to obtain the final dust removal output J fused .

[0021] Furthermore, in the unpaired learning network, for the first branch, given a clear and dust-free image, represented as I free ; First, the dust-free image I free Input to the depth estimation network G D , a preliminary rendered image J is generated through the Monodepth2 model dust As the final output of the first branch;

[0022] For the second branch, given a set of unpaired clear dust-free and dusty images, denoted as I free and J dust ; First, a discriminator D is used to transform the clean image I free Input to the refinement network G J ; The discriminator D receives the unpaired clean image I free and dusty image J dust Perform weakly supervised adversarial learning and update G J Learning I free and J dust The mapping relationship between them; finally, another preliminary rendering image J is obtained refine As the final output of the second branch;

[0023] After obtaining two preliminary rendering images J dust and J refine Finally, the image is fused into the final rendering output J through the non-reference image fusion strategy.fused .

[0024] Furthermore, the depth estimation network G D The invention comprises: an encoder based on ResNet-18, including 18 convolutional layers and multiple residual blocks, and a multi-scale decoding network as a decoder, including 4-scale deconvolutional layers and upsampling layers; the refinement network G J It is a ResNet with 9 residual blocks; the discriminator D is PatchGAN, which contains 5 convolutional layers.

[0025] Furthermore, the no-reference image fusion strategy is based on a natural image quality evaluator NIQE and a perception-based image quality evaluator PIQE, combining gradient modulus and chromaticity information in image quality assessment to perform perceptual fusion.

[0026] Furthermore, the no-reference image fusion strategy first calculates the NIQE score: score1 NIQE and score2 NIQE , and PIQE score: score1 PIQE and score2 PIQE J dust and J refine Evaluation is performed; the lower the NIQE and PIQE scores, the better the image quality; then the NIQE weight W is calculated NIQE and PIQE weight W PIQE ;

[0027]

[0028] Among them, W NIQE and W PIQE is the weight coefficient;

[0029] Further calculate the final weight W dust and W refine :

[0030]

[0031] Finally, use the corresponding weight fusion J dust and J refine , generate the final dust removal image J fused :

[0032] J fused =J dust ×W dust +J refine ×W refine .

[0033] Furthermore, the total loss function of the unpaired learning network includes the adversarial loss and identification loss The adversarial loss Used to update the subnetwork G in an adversarial manner J and the discriminator D to ensure that J refine The authenticity of is defined as:

[0034]

[0035] The recognition loss Used to ensure color consistency between input and output and to suppress G J Unrealistic artifacts generated; use Loss is defined as:

[0036]

[0037] Final loss function for:

[0038]

[0039] Among them, λ is the weight parameter.

[0040] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the unpaired learning simulation method for dust images based on ASM and GANs are implemented as described above.

[0041] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dust image unpaired learning simulation method based on ASM and GANs as described above.

[0042] Compared with the prior art, the present invention and its preferred solution firstly break through the limitation of clear-dust image pairing in the current image enhancement field by proposing a method of unpaired learning. Then, a deep estimation network (Monodepth2) and a generative adversarial network (GANs) are used for image enhancement respectively, which overcomes the limitations of deep learning methods and improves the accuracy and generalization ability of various dust scenes. Finally, a reference-free image fusion strategy is designed. The model fuses the images generated by the two networks through the reference-free image fusion strategy, and constructs a dust image enhancement dataset SCID containing 22 different colors, which effectively avoids the color distortion and noise amplification problems that are easily caused by traditional methods, and fills the gap of the lack of datasets in the current dust image enhancement field. Quantitative and qualitative experiments show that DustGAN is superior to the existing most advanced algorithms in color restoration and preservation of image details. The present invention can be widely used in virtual reality, game development, metaverse dust weather simulation, environmental visualization and other fields, and is of great significance to improving the fidelity of dust scenes, enriching simulation data resources, and enhancing the operability and accuracy of related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0044] Figure 1 A schematic diagram of the basic structure of a generative adversarial network;

[0045] Figure 2 This is a schematic diagram of atmospheric scattering model imaging;

[0046] Figure 3 This is a color histogram comparison chart of the haze image and the real image;

[0047] Figure 4 This is a structural framework diagram of DustGAN according to an embodiment of the present invention;

[0048] Figure 5 This is a qualitative comparison diagram of a synthetic dust image and a real dust image according to an embodiment of the present invention;

[0049] Figure 6 RGB color histogram comparison diagram of the embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows:

[0051] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] Introduction to the prior art related to this embodiment:

[0054] 1. Generative Adversarial Networks

[0055] Generative Adversarial Network (GAN) is an unsupervised deep learning model, which mainly consists of two parts: Generator and Discriminator. Through adversarial training, they work together to learn and generate realistic data samples. The basic structure is as follows Figure 1 As shown:

[0056] Generator. Its main task is to generate samples similar to real data from random noise. It is usually composed of multiple convolutional layers and fully connected layers. It maps the input random noise vector z to an image of appropriate size through upsampling. The noise z generally follows a normal distribution (that is, each element z i ~N(0,1)), so that each generated image is different. The performance of the generator determines the realism of the generated image. During the training process, the parameters are continuously adjusted to deceive the discriminator so that it cannot distinguish between the generated images and the real images. The generator optimizes the weights through back propagation and gradient descent to minimize the probability that the generated image is identified as fake data by the discriminator.

[0057] Discriminator. It is mainly responsible for distinguishing between real data and generated data. It consists of multiple convolutional layers and fully connected layers, and the output image is the probability value of the real image. As the training progresses, the generator becomes more and more powerful, the generated images become more and more realistic, and the discriminator's task becomes more and more difficult.

[0058] The generator and the discriminator compete with each other through adversarial training and continuously improve their performance. Finally, the generator can generate samples that are very similar to real data. The loss functions of the generator and the discriminator during the training process can be expressed as formula (1) and formula (2):

[0059]

[0060] Among them, p data (x) is the real data distribution, p z (z) is the noise distribution, G(z) is the output of the generator, and D(x) is the discriminator's judgment result on the sample.

[0061] Generative adversarial networks (GANs) generate sample data with similar distribution to real data from random noise through adversarial training of generators and discriminators. They are widely used in image generation, image restoration, style transfer and other fields, providing important tools and theoretical support for the development of deep learning.

[0062] 2. Dust image imaging model

[0063] In dusty weather, a large number of dust aerosol particles float in the air, which can cause Mie scattering of atmospheric light during propagation. Atmospheric scattering is caused by air molecules and aerosol particles, mainly including Rayleigh scattering and Mie scattering. Mie scattering is a phenomenon in which particles in a medium scatter light. Unlike Rayleigh scattering, Mie scattering does not depend on wavelength, so it can scatter light of all wavelengths, resulting in more uniform scattering of light. It is common in media with larger particles such as fog, dust and water droplets in the atmosphere.

[0064] The diameter of dust particles is usually between a few microns and tens of microns. Particles in this range mainly scatter blue light, causing the images captured by imaging equipment to appear orange or red, and visibility is low. McCartney et al. modeled the imaging process of haze images based on Mie scattering and proposed a famous atmospheric scattering model, which is shown in formulas (3) and (4):

[0065] I(x)=J(x)t(x)+A[1-t(x)] (3)

[0066] t(x)=e -βd(x) (4)

[0067] Among them, x represents the pixel in the image, I represents the haze image, and J represents the natural clear image; A represents the global atmospheric light, t represents the medium transmission rate, β is the scattering coefficient, and d is the distance from the scene to the camera. The specific imaging process of the atmospheric scattering model is as follows: Figure 2 As shown:

[0068] The atmospheric scattering model is mainly a physical modeling of the imaging principle of haze images. Considering the common degradation characteristics between dust images and haze images, Figure 2 The color histograms of two images of similar scenes were compared to intuitively distinguish the characteristics of dust images and haze images.

[0069] Specifically, if Figure 3As shown in the figure, in terms of subjective visual effects, although the haze image is blurred in the area with a large depth of field, the type of object can still be identified more clearly; in contrast, under the same depth of field conditions, it is almost impossible to extract valuable information from the sandstorm image. In addition, by comparing the distribution of the three RGB channels, obvious differences can be observed. The RGB channel distribution of the haze image is relatively concentrated and uniform, showing stronger color consistency, while the three channels of the sandstorm image show obvious independent distribution, with less overlapping areas and a narrow distribution range. This phenomenon shows that the main difference between the sandstorm image and the haze image is the color shift, which may be due to the fact that the diameter of the sand and dust particles is usually more than 20 times that of the haze particles, and the sand and dust particles absorb a lot of light energy and scatter the light. During the propagation process, light of different wavelengths is dispersed, resulting in significant changes in the color of the light captured by the observer or imaging device, which provides an important basis for distinguishing the two images.

[0070] In order to solve the problem of lack of benchmark datasets for deep learning in the field of dust image enhancement, researchers have built an algorithm framework with the help of heuristic ideas and used an improved atmospheric scattering model to synthesize dust image datasets, which has become a mainstream solution. Huang et al. analyzed the color distribution of dust and proposed an imaging physical model of dust images, which is shown in formula (5):

[0071] I(x)=J(x)(1-λ(1-t(x))+Cλ(1-t(x)) (5)

[0072] Where I and J represent the dust storm image and clear image, respectively. C and t represent the pre-set dust color and transmittance map, respectively. The model also introduces a parameter λ to simulate the influence of depth of field.

[0073] In summary, the atmospheric scattering model can accurately simulate the scattering characteristics of light in dust weather in dust rendering, improve the realism and fidelity of dust image rendering, and provide important theoretical support for dust image processing and enhancement.

[0074] 3 Image Generation Algorithm Evaluation Indicators

[0075] Image generation algorithm evaluation indicators are standardized tools used to quantify and evaluate the quality, realism, and diversity of generated images. Image quality evaluation is an important criterion for measuring the performance of image generation algorithms. Depending on whether there is an original image as a control, it is divided into full-reference evaluation and no-reference evaluation methods.

[0076] No reference evaluation index

[0077] No-reference evaluation indicators are an evaluation method used to evaluate image quality without the need for the original image as a control. These indicators mainly evaluate the quality of the image based on the characteristics and statistical information of the image itself, without relying on external references. They can quickly and effectively evaluate the overall quality of the image and are practical for many application scenarios. Due to the lack of sandstorm benchmark datasets, most researchers use non-reference metrics to evaluate the performance of sandstorm image enhancement algorithms. Basic evaluation indicators such as information entropy (IE), natural image quality evaluator (NIQE) and perception based image quality evaluator (PIQE) can effectively measure the performance of the algorithm. The following is a detailed introduction to the commonly used no-reference evaluation indicators in the field of image enhancement.

[0078] Natural Image Quality Evaluator. NIQE is a no-reference image quality evaluation index designed to evaluate the visual quality of natural images. It is based on local statistical information and perceptual features in natural images, and calculates the image quality score by analyzing the local structure, contrast, clarity, and color distribution of the image. It uses a variety of feature extractors, such as Gaussian filters, gradient histograms, and local variance, to capture different features in the image. The NIQE score can reflect the overall visual quality of the image. The smaller the value, the higher the image quality; otherwise, the lower the image quality. The specific calculation is shown in formula (6):

[0079]

[0080] Among them, v1, v2, Σ1, Σ2 represent the mean vector and covariance matrix corresponding to the natural MVG model and the distorted MVG model respectively.

[0081] Perceptual Image Quality Evaluator. PIQE is a no-reference image quality evaluation method that aims to evaluate image quality by simulating the perceptual mechanism of the human visual system. The PIQE algorithm first divides the image into multiple small blocks, analyzes each block, and calculates its spatial complexity and noise level. Then, based on the characteristics and statistics of the blocks, PIQE generates an overall quality score that reflects the clarity, contrast, and distortion of the image. The smaller the PIQE value, the higher the image quality; conversely, the lower the image quality. The specific calculation is shown in formula (7):

[0082]

[0083] in, For all N SAThe distortion score D of a small patch with significant activity (SA) sk The sum of the distortion scores of each small block is D sk Reflects the degree of distortion of the small block, N SA Indicates the number of patches with significant activity.

[0084] In summary, no-reference evaluation indicators are used to evaluate image quality without the need for the original image as a control. They are mainly based on the characteristics and statistical information of the image itself, such as information entropy (IE), natural image quality evaluator (NIQE) and perception-based image quality evaluator (PIQE). These indicators can quickly and effectively evaluate the overall quality of the image and are practical in many application scenarios for evaluating image quality.

[0085] Full reference evaluation index

[0086] Full-reference Image Quality Assessment Metrics is a type of image quality assessment method based on the quality comparison between the original image and the image to be evaluated. This type of metric relies on a high-quality reference image and calculates the difference between the image to be tested and the image to evaluate the degree of image distortion. Common full-reference evaluation metrics include peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). PSNR calculates the difference in pixel values ​​between the original image and the distorted image. It focuses on the signal-to-noise ratio of the image. The higher the value, the smaller the distortion and the better the quality. The calculation of PSNR is based on the mean square error (MSE). It has a weaker capture of details and therefore does not fully reflect the perception of the human eye. In contrast, SSIM is more in line with the characteristics of the human visual system. It evaluates the similarity of images from three aspects: brightness, contrast, and structural information. The closer the value is to 1, the higher the structural similarity and the smaller the distortion. Therefore, PSNR mainly reflects pixel differences, while SSIM emphasizes the perceptual similarity of image content. In complex environments, it is almost impossible to capture both the real degraded scene image and the corresponding clear image at the same time, which increases the difficulty for researchers to use full reference indicators to evaluate the algorithm. In the evaluation of the dust image enhancement results of the present invention, PSNR and SSIM are used to combine the performance of the evaluation algorithm on the synthetic data set to more comprehensively evaluate the image quality.

[0087] Peak signal-to-noise ratio. PSNR is a commonly used full-reference image quality evaluation indicator, which measures the degree of distortion of the reconstructed image by calculating the pixel error between the reconstructed image and the reference image. The specific calculation is shown in formula (8):

[0088]

[0089] PSNR is based on mean square error (MSE), which evaluates the noise or distortion in an image by calculating the square difference of pixel values ​​in the image. Its value is usually expressed in decibels (dB). The larger the value, the better the image quality and the smaller the difference with the reference image. The specific calculation is shown in formula (9):

[0090]

[0091] Among them, H and W represent the height and width of the image, that is, the number of rows and columns of the image, and H×W represents the total number of pixels in the image.

[0092] Structural Similarity Index. SSIM is a full-reference image quality evaluation index that measures the similarity between two images. SSIM evaluates the quality of an image by comparing the brightness, contrast, and structural information of the image. It not only focuses on the differences at the pixel level, but also on the local structural information of the image to reflect the human eye's perception of the image content, which is consistent with the perceptual characteristics of the human visual system. The value range of SSIM is -1 to 1. The closer the value is to 1, the higher the structural similarity of the two images and the smaller the distortion. The specific calculation is shown in formula (10):

[0093]

[0094] Among them, μ x and μ y Respectively represent the local means of the two images X and Y, which are used to measure the brightness information of the image; and Respectively represent the local variance of images X and Y, which are used to measure the contrast of the image; δ xy Represents the covariance of images X and Y, reflecting the structural similarity between the two images; c1 and c2 are two constants used for stable calculation, c1 = (k1L) 2 ,c2=(k2L) 2 , where L represents the maximum possible value of the image pixel, k1=0.01, k2=0.03.

[0095] In summary, the full reference evaluation index measures the degree of image distortion by comparing the reference image and the image to be evaluated. PSNR mainly focuses on the difference at the pixel level, reflects the signal-to-noise ratio of the image through the mean square error, and is suitable for simple noise evaluation; while SSIM is more in line with the visual perception of the human eye through a comprehensive evaluation of brightness, contrast and structural information, and is suitable for evaluating the overall structural similarity of the image. In the dust image enhancement task, the present invention combines PSNR and SSIM to more comprehensively evaluate the algorithm performance and image quality, which can make up for the shortcomings of a single indicator.

[0096] The following is a detailed introduction to the embodiments of the present invention: 1. Dust rendering model based on ASM

[0097] In the embodiment of the present invention, first consider using Monodepth2 to establish a dust rendering model. Monodepth2 is a deep learning model for monocular depth estimation. The model uses a self-supervised learning method to predict a depth map from a single RGB image. Using the Monodepth2 model, the depth of the input image is first estimated, and then the image is rendered based on the estimated depth map. In this way, the dust effect is simulated in the image, increasing the visual effect and realism of the image. The design of building a dust rendering network includes:

[0098] (1) Depth estimation process: load the pre-trained Monodepth2 encoder and decoder models, use the encoder to extract image features, and generate a depth map through the decoder.

[0099] (2) Dust rendering process: The predicted depth map is normalized, the original image and the dust effect map are loaded, and weighted fusion is performed using the normalized value of the depth map.

[0100] The present invention uses the Monodepth2 model for depth estimation and performs dust rendering on the image based on the depth map. The proposed expression is as follows:

[0101] J dust =0.8*(0.8*Image org *Depth normalized +1.3*Image color *(l-

[0102] Depth normalized )) (11)

[0103] Among them, J dust is the generated dust rendering image, Image org is the original image, Depth normalized is the depth map, Image color is the dust rendering image. According to formula (11), the dust rendering result J is obtained by linearly combining the content of the original image with the dust rendering image. dust .

[0104] 2 Overall Framework

[0105] The embodiment of the present invention proposes an unpaired learning network (DustGAN) based on ASM and GAN for dust image rendering. Based on the observation of dust image characteristics, DustGAN is designed from the following two aspects.

[0106] On the one hand, it is found that most existing methods for rendering dust in images have poor visual effects, and the distribution of dust in the image is not accurate enough. Therefore, Monodepth2 is used to establish an ASM model to render dust in images, and the distribution of dust effects in the image is accurately controlled by the depth map. Areas with larger depth values ​​represent objects at a distance, and these areas can add heavier dust effects. Areas with smaller depth values ​​represent objects at a close distance, and these areas can remain clearer to achieve a more natural transition effect. In addition, objects at a distance and near will produce different visual effects due to the occlusion and scattering effects caused by particulate matter in the air, and the dust environment is realistically reproduced in the image; on the other hand, supervised methods need to train the model on large-scale clear-damaged image pairs, which are difficult to collect in practice, so they can easily overfit to synthetic training data sets. In other words, they obtain excellent rendering results on synthetic images, but have limited performance in real image rendering. Therefore, the present invention adopts a weakly supervised model trained on unpaired images to alleviate this problem to a certain extent. The DustGAN structure is as follows Figure 4 shown.

[0107] Network structure. DustGAN consists of two branches, each of which consists of three sub-networks. The main ones include the depth estimation network G D , Refined Network G J and the discriminator D, are the backbone networks of CycleGAN. Specifically, G D It is an encoder based on ResNet-18, which contains 18 convolutional layers and multiple residual blocks. The decoder is a multi-scale decoding network, which contains 4 scale deconvolutional layers and upsampling layers. J is a ResNet with 9 residual blocks, while D is a basic PatchGAN with 5 convolutional layers. Figure 4 The structure of DustGAN is shown. DustGAN first generates two preliminary rendering images J from two branches dust and J refine , and then fuse them through the introduced no-reference image fusion strategy to obtain the final dust removal output J fused .

[0108] The first branch. Given a clear and dust-free image, denoted as I free First, the dust-free image I free Input to the depth estimation network G D , a preliminary rendered image J is generated through the Monodepth2 model (corresponding to formula (11)) dust As the final output of the first branch.

[0109] The second branch. Given a set of unpaired clear dust-free and dusty images, denoted as I free and J dust In the second branch, a discriminator D is added to convert the clean image I free Input to the refinement network G J D accepts unpaired clean images I free and dusty image J dust Perform weakly supervised adversarial learning and update G J Come and learn I free and J dust Finally, another reasonable preliminary rendering image J is obtained. refine As the final output of the second branch.

[0110] The fusion of the two branches. After obtaining two preliminary rendering images J dust and J refine Finally, the no-reference image fusion strategy fuses them into the final rendering output J of DustGAN fused .

[0111] 3 No-reference Image Fusion Strategy

[0112] The no-reference image fusion strategy usually uses the local and global features of the image to adaptively adjust the fusion parameters, so that the fused image can show higher visual quality in different scenes, has stronger adaptability, and is suitable for simulating dust effects in a variety of complex environments.

[0113] The embodiment of the present invention introduces a newly designed reference-free image fusion strategy into the model, which is based on the natural image quality evaluator (NIQE) and the perception-based image quality evaluator (PIQE). Specifically, the gradient modulus (GM) and chromaticity information in image quality assessment are combined for perceptual fusion. The gradient modulus helps to preserve the edge and detail features in the image, while the chromaticity information ensures that the fused image is more realistic and natural in color expression. In this way, the advantages of different network-generated images can be combined to generate a more realistic and delicate dust rendering effect.

[0114] First, the NIQE score (called score1) is calculated NIQE and score2 NIQE ) and PIQE score (referred to as score1 PIQE and score2 PIQE ) respectively for J dust and J refine The lower the NIQE and PIQE scores, the better the image quality. Therefore, the weight values ​​of NIQE and PIQE (called W NIQE and W PIQEIt should be inversely proportional to its score. However, it is found in the experiment that score1 NIQE and score2 NIQE The values ​​of J are very close and it is difficult to distinguish dust and J refine According to formulas (12) and (13), only W PIQE Reverse and balance the final weight to obtain the NIQE weight W NIQE and PIQE weight W PIQE :

[0115]

[0116] Among them, W NIQE and W PIQE To keep J dust and J refine The weight coefficient is designed for the integrity of the system. Next, we can further obtain J by formula (14): dust The final weight W dust . And, through 1-W dust , you can get J refine The final weight W refine , as shown in formula (15):

[0117]

[0118] Finally, use the corresponding weight fusion J dust and J refine , generate the final dust removal image J fused , the specific formula is as follows:

[0119] J fused＝ J dust ×W dust +J refine ×W refine (16).

[0120] 4 Loss Function

[0121] The total loss function of DustGAN consists of two parts, namely adversarial loss and identification loss The first loss Used to update the subnetwork G in an adversarial manner J and the discriminator D to ensure that J refine The authenticity of is defined as:

[0122]

[0123] Considering the traditional The loss can better ensure pixel consistency and generate clearer images.

[0124] Higher. Second loss Ensure color consistency between input and output and suppress G J Unrealistic artifacts are generated. Loss is defined as:

[0125]

[0126] In summary, the final loss function defined is for:

[0127]

[0128] In the present invention, the weight parameter λ is set to 0.2.

[0129] In order for those skilled in the art to better understand the scheme of the present invention, the scheme of the present invention is further compared, verified and tested through specific test examples below.

[0130] The following first introduces the training details and evaluation metrics of the DustGAN algorithm. Then, to demonstrate the superiority of the proposed DustGAN, a comprehensive qualitative and quantitative comparison with the state-of-the-art methods is conducted on synthetic and real dust images. Finally, a dust image enhancement dataset SCID with 22 different colors is proposed and evaluated using popular image dehazing algorithms.

[0131] 1 Dataset

[0132] This example uses the RESIDE outdoor dataset (Realistic Single Image Dehazing Dataset) to train and evaluate the model. Specifically, the RESIDE outdoor clear image subset OTS was selected to evaluate the effect and generalization of all methods in the experiment. It contains 2061 clear outdoor images covering various scenes and environments, which can well represent images in the real world. In addition, the 22 dust colors used in this invention depend on the global color deviation value used in SIRB proposed by Si et al.

[0133] 2 Experimental Environment

[0134] The following model was built and trained using the PyTorch framework on an NVIDIA GeForce RTX 3080Ti GPU. The momentum parameters β1 and β2 were set to 0.5 and 0.999, respectively, and the initial learning rate was 0.0002, which decayed linearly with the number of iterations. The training cycle was set to 60 times to achieve the best performance, and the batch size was 12. During network training, the input image size was randomly cropped to 256×256 pixels, and the two generators (G D and G J ) and the parameters of the discriminator (D) are updated alternately. All other parameters follow the default settings of CycleGAN. For other comparison methods, the present invention maintains their original optimal parameter settings. In particular, these methods are all implemented based on the PyTorch framework, the input image size is cropped to 256×256 pixels, and they are all trained for 200 cycles to obtain realistic dust rendering results. In the comparative experiment with the existing traditional dust image rendering algorithm, this embodiment will follow their recommended experimental settings.

[0135] 3 Comparison of algorithms

[0136] In order to fully verify the performance of the model, the proposed DustGAN model is compared with the synthesis method proposed by Huang et al. Huang et al. describe the imaging process of dust images by improving the atmospheric scattering model, and formula (5) is the physical model proposed by them. Since the algorithm of this embodiment performs model training in an unsupervised manner, the rendered sandstorm color is directly learned from the input real reference image, while the comparison algorithm synthesizes the sandstorm image by observing the preset dust color. Therefore, this embodiment will follow the recommended method in the experimental setting, input the preset sandstorm color value and use the stereo pair matching algorithm to estimate the transmittance map.

[0137] 4 Evaluation indicators

[0138] How to quantitatively evaluate the generation results of image blur is a challenging and open problem. The purpose of dust image rendering is to simulate the image degradation process under dust weather, making the image quality worse. However, in the field of image processing under natural weather, the image quality evaluation index is usually to compare the difference between the original image and the reference image. For the evaluation of image generation algorithms, it is often necessary to comprehensively consider the non-reference evaluation indicators of the visual quality, structural information and color information of the image. Therefore, taking into account the special degradation effects under dust weather, such as low visibility, color shift and loss of image details, this embodiment combines the two measurement indicators of image information entropy (IE) and Fréchet Inception Distance (FID) for quantitative comparison.

[0139] Image information entropy (IE) is an indicator that measures the complexity and amount of information in an image, and can reflect the richness of details and textures in the image. A higher information entropy value usually means that the image contains more information and details. For an image, information entropy can be obtained by calculating the probability of each grayscale value appearing. For a grayscale image (grayscale levels from 0 to 255), the formula is as follows:

[0140]

[0141] in, is the gray level x i The probability of occurrence is usually obtained by normalizing the image histogram.

[0142] Fréchet Inception Distance (FID) is used to measure the similarity between the generated image and the real image in the feature space distribution. A lower FID value indicates that the quality of the generated image is closer to the real image. Therefore, by combining these two indicators, the performance of the dust image rendering algorithm can be more comprehensively evaluated, including its performance in visual quality, structural information, and color information. The formula is as follows:

[0143]

[0144] Among them, the Inception model is used to extract the features of the real image and the generated image, and two feature distributions are obtained, which are represented as multidimensional normal distribution (μ r ,∑ r ) and (μ g ,∑ g ), where μ is the mean vector and ∑ is the covariance matrix. ||μ r -μ g || 2 is the Euclidean distance between mean vectors, and Tr represents the trace of the matrix, that is, the sum of the diagonal elements.

[0145] In addition, when performing performance analysis and verification on popular image dehazing algorithms, the present invention combines the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to evaluate the algorithm performance and image quality.

[0146] Quantitative analysis

[0147] This embodiment selects 10 scenes similar to real natural images and real dust images as test samples, and takes the average of each indicator as the final result. In the comparative analysis, the performance of the rendering results of the DustGAN algorithm is evaluated from the two indicators of IE value and FID value. Table 1 gives the IE value and FID value of the real image, dust-free image, physical model synthetic dust image, and DustGAN algorithm rendered dust image. Among them, the IE value and FID value of the DustGAN algorithm are the average values ​​calculated under 22 different dust color test conditions.

[0148] Table 2 is a summary of the IE and FID values ​​of the DustGAN algorithm under 22 dust color test conditions.

[0149] Table 1 Quantitative comparison of synthetic images and corresponding similar scene images

[0150]

[0151] Table 222 IE and FID values ​​under test conditions of dust colors

[0152]

[0153] Image information entropy IE reflects the complexity and uncertainty of the image. The closer the information entropy value is to the real image, the more the pixel values ​​in the image tend to be the same, and the more the image tends to be single and regular. In the scenario of sandstorm image generation, it is expected that the generated image has a relatively smooth dust distribution and a single tone. Therefore, the information entropy of the generated sandstorm image should be as low as possible. The smaller the FID distance, the closer the generated sandstorm image is to the real image. Excessive content differences between different images may make the measurement inaccurate. As shown in Table 1, from the perspective of IE value, the IE value of the DustGAN algorithm is 0.0973 higher than that of the comparison algorithm image, which shows that the DustGAN algorithm has a higher restoration ability in maintaining image quality than the comparison algorithm, and can better restore the details and texture of the real scene; from the perspective of FID value, that is, the characteristic statistical data distance between the rendering result of the DustGAN algorithm and the real sandstorm image. The sandstorm image rendered by the DustGAN algorithm is 17.1814 lower than that of the comparison algorithm. This shows that the dust image rendered in this embodiment is closer to the real dust image, which verifies the effectiveness of the DustGAN dust image rendering algorithm from an objective perspective.

[0154] Qualitative analysis

[0155] This example makes a qualitative comparison between the rendering results and the real reference image in terms of visual effects, detail expression, and color contrast. The rendering results of dust images in different scenes and the real dust images in similar scenes are shown. Figure 5(a) Real dust image. It can be seen that real dust images usually have the following characteristics: (1) Due to the occlusion and scattering of dust particles, the outlines of objects in the real image are not clear, and the whole image appears hazy; (2) The details of distant and near scenes are different. Scenes far away from the camera are usually covered by dust particles in the air, making fine textures and details difficult to recognize.

[0156] Figure 5 (b) is the input clear and dust-free image. Figure 5 (c) is a dust image synthesized by Huang et al. using a physical model. Figure 5 (d) to (x) are dust images synthesized by the present invention using the DustGAN algorithm. Figure 5 (c) with Figure 5 (d) to (x) respectively show the rendering results of the comparison algorithm and the algorithm of the present invention. Analyzing from the overall visual effect of the image, both of them reflect the hazy feeling of reduced clarity under the dust rendering. In terms of the detailed expression of the image scene, the physical model synthesis result of Figure (c) renders the dust in the distant view relatively uniformly, but lacks the rendering of the near view, which makes the objects in the near view have discontinuous hole artifacts. In contrast, the dust image generated by the DustGAN proposed by the present invention has a more effective depth perception of the dust concentration, that is, the object gradually blurs from near to far, which is more consistent with the dust distribution of the real image; Analyzing from the color effect of the image, due to the appearance of the hole artifact, Figure (c) still retains the bright color of the original image in the near view, while the output result of the algorithm of this embodiment maintains the color distribution characteristics consistent with the real dust image in subjective vision due to the color consistency learning and image depth rendering refinement.

[0157] The color distribution of real dust images has significant prior features such as channel displacement, distribution concentration and order. Channel displacement refers to the offset of the R, G, and B channels of the dust image due to the Mie scattering effect; distribution concentration means that the R, G, and B channels are relatively evenly distributed in the entire value range [0-255]; order means that the three channels are usually arranged in the order of G, B, and R.

[0158] Figure 6 The RGB color histograms corresponding to the dust image rendered on the clear image by the algorithm of the present invention and the real dust image are shown. Figure 2 (a) shows that the three color channels of a clear, dust-free image overlap substantially over a large range; in contrast, Figure 6 As shown in (c), the RGB distribution of the real dust image has shifted, and each color channel is confined to a fixed narrow range. Figure 6(b) shows that the histogram distribution of the generated image meets all the characteristics of the real dust image, which further verifies the realism of the algorithm in rendering dust images and its applicability in the field of dust image enhancement.

[0159] Performance Analysis of Image Dehazing Algorithms

[0160] In this experiment, the dataset used is the constructed sand and dust image dataset SCID (Sand Color Image Dataset), which contains 22 sand and dust images of different colors. In addition, the popular dehazing algorithm DedustGAN is used to test the dataset and verify its effectiveness in restoring image quality.

[0161] The SCID dataset has the following characteristics: (1) High quality. The image enhancement dataset SCID is proposed through the DustGAN algorithm. It has high accuracy in dust scene enhancement and solves the color distortion and noise amplification problems that are easily caused by traditional methods. It fills the gap in the current lack of datasets in the field of dust image enhancement; (2) Diversity. SCID contains 22 representative dust colors, covering different tones from light sand to dark brown. It aims to simulate the complexity of various dust weather in the real environment and has strong generalization ability; (3) Reality. The SCID dataset simulates a variety of natural scenes under dust weather, ensuring the realism of the images during training and testing, thereby providing a more challenging and practical testing environment for the actual application of dehazing algorithms in real scenes.

[0162] In the experiment, Table 3 shows the PSNR and SSIM values ​​of the DedustGAN image dehazing algorithm under 22 different dust color test conditions to evaluate the algorithm's dehazing performance. From the results, it can be seen that the performance of the DedustGAN algorithm on different dust color datasets is relatively consistent, with PSNR values ​​mostly maintained at around 27.9, and SSIM values ​​mostly between 0.85 and 0.87. A higher PSNR value indicates that the image quality loss after dehazing is small, and a higher SSIM value indicates that the image structure after dehazing is well maintained. These data show that DedustGAN has a stable and excellent performance in removing the influence of different colors of dust, and can effectively improve image clarity and retain image details, further verifying the robustness and effectiveness of the algorithm in processing complex dust scenes.

[0163] Table 3 Summary of PSNR and SSIM values ​​under 22 dust color test conditions

[0164]

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of unpaired learning simulation methods for dust images based on ASM and GANs under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An unpaired learning simulation method for dust images based on ASM and GANs, characterized by: An unpaired learning network based on ASM and GAN is used for dust image rendering. A depth estimation network is used to establish the ASM model and a generative adversarial network is used for image enhancement. The images generated by the two networks are then fused using a no-reference image fusion strategy.

2. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 1 is characterized by: Use Monodepth2 to build an ASM model to render dust in the image, including: Depth estimation process: load the pre-trained Monodepth2 encoder and decoder models, use the encoder to extract image features, and use the decoder to generate a depth map; Dust rendering process: Normalize the predicted depth map, load the original image and dust effect map, and perform weighted fusion using the normalized value of the depth map: J dust =0.8*(0.8*Image org *Depth normalized +1.3*Image color *(l-Depth normalized )) Among them, J dust Is the generated dust rendering image, Image org is the original image, Depth normalized is the depth map, Image color is the dust effect map; the dust rendering result J is obtained by linearly combining the content of the original image with the dust effect map dust .

3. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 1 is characterized by: The unpaired learning network comprises two branches. First, two preliminary rendering images J are generated from the two branches. dust and J refine , and then fuse them through the no-reference image fusion strategy to obtain the final dust removal output J fused .

4. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 3 is characterized by: In the unpaired learning network, for the first branch; Given a clear and dust-free image, denoted as I free ; First, the dust-free image I free Input to the depth estimation network G D , generate a preliminary rendered image J through the Monodepth2 model dust As the final output of the first branch; For the second branch, given a set of unpaired clear dust-free and dust-containing images, denoted as I free and J dust ; First, a discriminator D is used to transform the clean image I free Input to the refined network G J ; The discriminator D receives the unpaired clean image I free and Dust Image J dust Perform weakly supervised adversarial learning and update G J Learning I free and J dust The mapping relationship between them; finally, another preliminary rendering image J is obtained refine As the final output of the second branch; After obtaining two preliminary rendering images J dust and J refine Finally, the image is fused into the final rendering output J through the no-reference image fusion strategy fused .

5. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 4 is characterized by: The depth estimation network G D It includes: an encoder based on ResNet-18, including 18 convolutional layers and multiple residual blocks, and a multi-scale decoding network as a decoder, including 4-scale deconvolution layers and upsampling layers; the refined network G J It is a ResNet with 9 residual blocks; the discriminator D is PatchGAN, which contains 5 convolutional layers.

6. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 3 is characterized by: The proposed no-reference image fusion strategy is based on the natural image quality estimator (NIQE) and the perceptual-based image quality estimator (PIQE), combining the gradient modulus and chromaticity information in image quality assessment for perceptual fusion.

7. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 6 is characterized by: The no-reference image fusion strategy first calculates the NIQE score: score1 NIQE and score2 NIQE , and PIQE score: score1 PIQE and score2 PIQE J dust and J refine Evaluation is performed; the lower the NIQE and PIQE scores, the better the image quality; then the NIQE weight W is calculated NIQE and PIQE weight W PIQE ; Among them, W NIQE and W PIQE is the weight coefficient; Further calculate the final weight W dust and W refine : Finally, use the corresponding weight fusion J dust and J refine , generate the final dust removal image J fused : I fused =J dust ×W dust +J refine ×W refine 。 8. The unpaired learning simulation method for dust images based on ASM and GANs according to claim 3 is characterized by: The total loss function of the unpaired learning network includes the adversarial loss and identification loss The adversarial loss Used to update the sub-network G in an adversarial manner J and the discriminator D to ensure that J refine The authenticity of is defined as: The recognition loss Used to ensure color consistency between input and output and suppress G J Unrealistic artifacts generated; use Loss is defined as: Final loss function for: Among them, λ is the weight parameter.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the unpaired learning simulation method for dust images based on ASM and GANs are implemented as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the unpaired learning simulation method for dust images based on ASM and GANs are implemented as described in any one of claims 1 to 8.

Images