Image defogging method, device and system and storage medium

Abstract

The invention provides an image defogging method, device and system and a storage medium, and belongs to the technical field of image defogging, and the method comprises the steps: importing an original image training set and an image test set, and carrying out the preprocessing of the original image training set, and obtaining a target image training set and a depth image real noise data set; constructing a diffusion model, and performing model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model; and performing image defogging processing on the image test set through the image defogging model to obtain an image defogging result. According to the method, the problem of incomplete defogging caused by inaccurate modeling of foggy scene distribution by the model is solved, the problems of excessive defogging of near objects and insufficient defogging of far objects are also avoided, meanwhile, the difficulty of model learning is reduced, and more accurate defogging processing is provided for scenes with different depths.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of image defogging, and in particular to an image defogging method, device, system and storage medium. Background Art

[0002] In the field of image processing and computer vision, the impact of haze on image quality is an urgent problem to be solved, because it is directly related to the accuracy and reliability of downstream tasks (such as target detection, image segmentation, object recognition, etc.). For example, in the scenario of autonomous driving, the quality of the image captured by the camera directly affects the decision-making process of the driving system, and the foggy images caused by haze may cause serious safety hazards. Therefore, the development of effective image dehazing technology is of great practical significance.

[0003] Image dehazing refers to restoring a clean image from a foggy image to improve the visual quality. Traditional dehazing methods can be roughly divided into two categories: image enhancement-based methods and physical model-based methods. Image enhancement-based methods usually improve the visual effect of the image by adjusting the brightness, contrast or hue of the image. Physical model-based methods attempt to reversely derive a fog-free image through understanding the atmospheric scattering model. However, traditional methods lack the estimation of haze depth or the estimation is inaccurate, resulting in poor results in dealing with severe haze conditions, and simple models and enhancement methods cannot handle complex optical phenomena. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an image defogging method, device, system and storage medium in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: an image defogging method comprises the following steps:

[0006] Importing an original image training set and an image test set, preprocessing the original image training set to obtain a target image training set and a depth image real noise data set;

[0007] Constructing a diffusion model, and performing model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model;

[0008] The image defogging model is used to perform image defogging processing on the image test set to obtain an image defogging result.

[0009] Another technical solution of the present invention to solve the above technical problem is as follows: an image defogging device, comprising:

[0010] Import module, used to import original image training set and image test set;

[0011] A preprocessing module, used to preprocess the original image training set to obtain a target image training set and a depth image real noise data set;

[0012] A model analysis module is used to construct a diffusion model, and perform model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model;

[0013] The image defogging result obtaining module is used to perform image defogging processing on the image test set through the image defogging model to obtain the image defogging result.

[0014] Based on the above-mentioned image defogging method, the present invention also provides an image defogging system.

[0015] Another technical solution of the present invention to solve the above technical problem is as follows: an image defogging system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image defogging method as described above is implemented.

[0016] Based on the above-mentioned image defogging method, the present invention also provides a computer-readable storage medium.

[0017] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image defogging method as described above is implemented.

[0018] The beneficial effects of the present invention are as follows: a target image training set and a depth image real noise data set are obtained by preprocessing an original image training set, an image defogging model is obtained by model analysis of a diffusion model using the target image training set and the depth image real noise data set, and an image defogging result is obtained by performing image defogging processing on an image test set using the image defogging model. This improves the problem of incomplete defogging caused by inaccurate modeling of the distribution of foggy scenes, avoids the problem of excessive defogging of nearby objects and insufficient defogging of distant objects, and at the same time reduces the difficulty of model learning, providing more accurate defogging processing for scenes of different depths. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of an image defogging method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the diffusion model structure of the image defogging method provided by an embodiment of the present invention;

[0021] Figure 3A module block diagram of an image defogging device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0023] Figure 1 A schematic diagram of a flow chart of an image defogging method provided by an embodiment of the present invention.

[0024] like Figure 1 As shown, an image defogging method includes the following steps:

[0025] Importing an original image training set and an image test set, preprocessing the original image training set to obtain a target image training set and a depth image real noise data set;

[0026] Constructing a diffusion model, and performing model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model;

[0027] The image defogging model is used to perform image defogging processing on the image test set to obtain an image defogging result.

[0028] It should be understood that the subsets ITS and OTS of the image dehazing dataset RESIDE are used as training sets (ie, original image training sets), and O-haze containing 45 image pairs is used as a test set (ie, image test set).

[0029] In the above embodiment, the target image training set and the depth image real noise data set are obtained by preprocessing the original image training set, the image defogging model is obtained by model analysis of the diffusion model with the target image training set and the depth image real noise data set, and the image defogging result is obtained by performing image defogging processing on the image test set with the image defogging model. The problem of incomplete defogging caused by inaccurate modeling of the distribution of foggy scenes is improved, and the problem of excessive defogging of nearby objects and insufficient defogging of distant objects is avoided. At the same time, the difficulty of model learning is reduced, and more accurate defogging processing is provided for scenes of different depths.

[0030] Optionally, as an embodiment of the present invention, the original image training set includes a plurality of original foggy training images and original fog-free training images corresponding to each of the original foggy training images.

[0031] The process of preprocessing the original image training set to obtain the target image training set and the depth image real noise data set includes:

[0032] Respectively scaling each of the original foggy training images and each of the original fog-free training images to obtain scaled foggy training images corresponding to each of the original foggy training images and scaled fog-free training images corresponding to each of the original foggy training images;

[0033] Partitioning the scene depths of each of the scaled fog-free training images respectively to obtain depth partition maps corresponding to each of the original foggy training images;

[0034] Importing Gaussian noise, adding the Gaussian noise to each of the depth partition maps to obtain a true depth image noise corresponding to each of the original foggy training images, and collecting all the true depth image noises to obtain a depth image true noise dataset;

[0035] Respectively adding the real noise of each of the depth images to the scaled fog-free training images corresponding to each of the original foggy training images to obtain the noisy fog-free training images corresponding to each of the original foggy training images;

[0036] Each of the noise-free foggy training images and the scaled foggy training images corresponding to each of the original foggy training images are respectively connected channel by channel to obtain a target training image corresponding to each of the original foggy training images, and all of the target training images are collected to obtain a target image training set.

[0037] It should be understood that a pre-processing operation is performed before entering the network model (ie, the diffusion model).

[0038] Specifically, 86125 pairs of fog-free images (i.e., original fog-free training images) and foggy images (i.e., original foggy training images) in the RESIDE training set are used as training sets, and they are scaled to 128×128 size with 3 channels, denoted as {x i ,y i} (i.e., the scaled fog-free training image and the scaled foggy training image).

[0039] Specifically, at the network input, the randomly sampled Gaussian noise ε~N(0, I) and the depth partition map d corresponding to the sample are added at time t. i Add to get the noise with depth information (i.e., the real noise of the depth image), added to the fog-free image x i (i.e., the zoomed fog-free training image) to obtain the noise sample (i.e., noisy and fog-free training images); then, the noise samples (i.e., noisy and foggy training image) and foggy image y i (i.e., the scaled foggy training image) is connected channel by channel, taking the foggy image as a condition to obtain the new input Xi (i.e., target training image), with a size of 128×128×6.

[0040] In the above embodiment, the original image training set is preprocessed to obtain the target image training set and the depth image real noise data set, thereby avoiding the problem of excessive defogging of nearby objects and insufficient defogging of distant objects. At the same time, it reduces the difficulty of model learning and provides more accurate defogging processing for scenes of different depths.

[0041] Optionally, as an embodiment of the present invention, the process of partitioning the scene depth of each of the scaled fog-free training images to obtain a depth partition map corresponding to each of the original foggy training images includes:

[0042] Performing depth estimation on each of the scaled fog-free training images using a depth estimation model to obtain an original depth map corresponding to each of the original foggy training images;

[0043] Scaling each of the original depth maps respectively to obtain a target depth map corresponding to each of the original foggy training images;

[0044] The depth partitioning is performed on each of the target depth maps through the depth embedding layer to obtain a depth partition map corresponding to each of the original foggy training images.

[0045] Specifically, the fog-free image in the training set (i.e., the scaled fog-free training image) is fed into the depth anything depth estimation model to obtain the depth map D i (i.e. original depth map), D i (i.e., the original depth map) is a matrix of the same size as the input image. Each value in the matrix represents the depth information of the corresponding position in the image. It is then scaled to 128×128 and evenly divided into four parts according to the depth value in the depth embedding layer (i.e., the depth embedding layer). Each part corresponds to a different depth level, so that the model learns to remove fog while learning the relative distance of the scene depth of the input image, i.e., distant view, distant medium view, near medium view, and near view. According to the atmospheric scattering model and life experience, the greater the scene depth, the thicker the fog tends to be, and the fog concentration in areas with similar depths is often similar. Therefore, different depth areas should adopt different defogging schemes, and the model needs to learn an additional defogging prompt information, i.e., the relative size of the scene depth.

[0046] In the above embodiment, each scaled fog-free training image is partitioned by scene depth to obtain a depth partition map, which enables the model to learn the relative distance of the scene depth of the input image while learning defogging, providing more accurate defogging processing for scenes of different depths.

[0047] Optionally, as an embodiment of the present invention, the target depth map includes a plurality of pixel positions and original depth values ​​corresponding to each of the pixel positions.

[0048] The process of performing depth partitioning on each of the target depth maps through the depth embedding layer to obtain depth partition maps corresponding to each of the original foggy training images comprises:

[0049] The maximum value of all the original depth values ​​in each of the target depth maps is respectively screened out, and the maximum depth value of each of the target depth maps is obtained after screening;

[0050] Respectively filter out the minimum value of all the original depth values ​​in each of the target depth maps, and obtain the minimum depth value of each of the target depth maps after filtering;

[0051] If the original depth value corresponding to each pixel position satisfies the first formula, the first preset value is used as the target depth value corresponding to the pixel position, and the first formula is:

[0052]

[0053] Among them, D i (x1) is the original depth value corresponding to the x1-th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map;

[0054] If the original depth value corresponding to each of the pixel positions satisfies the second formula, the second preset value is used as the target depth value corresponding to the pixel position, and the second formula is:

[0055]

[0056] Among them, D i (x2) is the original depth value corresponding to the x2th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map;

[0057] If the original depth value corresponding to each pixel position satisfies the third formula, the third preset value is used as the target depth value corresponding to the pixel position. The third formula is:

[0058]

[0059] Among them, D i (x3) is the original depth value corresponding to the x3th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map;

[0060] If the original depth value corresponding to each pixel position satisfies the fourth formula, the fourth preset value is used as the target depth value corresponding to the pixel position. The fourth formula is:

[0061]

[0062] Among them, D i (x4) is the original depth value corresponding to the x4th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map;

[0063] The target depth values ​​corresponding to all the pixel positions in each of the target depth maps are plotted respectively to obtain depth partition maps corresponding to each of the original foggy training images.

[0064] Preferably, the first preset value may be 0, the second preset value may be 1, the third preset value may be 2, and the fourth preset value may be 3.

[0065] Specifically, the depth map D i The depth value (i.e., original depth value) at each pixel position in the target depth map is mapped to new partition values ​​0, 1, 2, and 3 according to the interval in which it is located. The maximum value of the depth map is d max (i.e. the maximum depth value), the minimum value is d min (i.e. the minimum depth value), The pixel position of the depth map is x, and the depth partition map d is obtained according to the following rules i :If D i (x)∈[d min , d min +δ), then d i (x) = 0; if D i (x)∈[d min +δ,d min +2δ), then d i (x) = 1; if D i (x)∈[d min +2δ,d min +3δ), then d i (x) = 2; if D i (x)∈[d min +3δ,d max ], then d i (x)=3.

[0066] In the above embodiment, each target depth map is depth partitioned by the depth embedding layer to obtain a depth partition map, which enables the model to learn the relative distance of the input image scene depth while learning defogging, providing more accurate defogging processing for scenes of different depths.

[0067] Optionally, as an embodiment of the present invention, Figure 1 and 2 As shown, the diffusion model includes a first 3×3 convolutional layer, a downsampling stage, an intermediate stage, an upsampling stage, and a second 3×3 convolutional layer.

[0068] The process of constructing a diffusion model, performing model analysis on the diffusion model through the target image training set and the depth image real noise data set, and obtaining an image defogging model includes:

[0069] Performing feature extraction on each of the target training images through the first 3×3 convolutional layer to obtain first training image features corresponding to each of the original foggy training images;

[0070] Downsampling and analyzing each of the first training image features respectively in the downsampling stage to obtain second training image features corresponding to each of the original foggy training images;

[0071] Performing feature enhancement on each of the second training image features in the intermediate stage to obtain third training image features corresponding to each of the original foggy training images;

[0072] Performing upsampling analysis on each of the third training image features in the upsampling stage to obtain fourth training image features corresponding to each of the original foggy training images;

[0073] Extracting features of each of the fourth training images through the second 3×3 convolutional layer to obtain depth image prediction noise corresponding to each of the original foggy training images;

[0074] The loss function corresponding to each of the original foggy training images is calculated by performing loss function calculation on each of the predicted noises of the depth images and the real noises of the depth images corresponding to each of the original foggy training images through the fifth formula, and the loss function corresponding to each of the original foggy training images is obtained. The fifth formula is:

[0075]

[0076] Where L(θ) j is the loss function corresponding to the jth original foggy training image, t is the time, T is the preset duration, x j is the scaled fog-free training image corresponding to the jth original foggy training image, is the real noise of the depth image corresponding to the jth original foggy training image, Predict the noise of the depth image corresponding to the jth original foggy training image, ||||1 is the L1 norm;

[0077] The parameters of the diffusion model are updated according to the Adam optimizer and all the loss functions to obtain an image defogging model.

[0078] It should be understood that the data processing steps of the first 3×3 convolutional layer and the second 3×3 convolutional layer are the same, and only the processed data are different.

[0079] It should be understood that the structure of the diffusion model noise estimation network is defined, and the noise error loss is calculated using the training method of the conditional diffusion model to obtain the network parameters of the defogging diffusion model.

[0080] Specifically, the noise estimation network transforms the foggy image y i and time step t as input to obtain noise with depth information (i.e., depth image prediction noise); specifically, the network is composed of a UNet structure consisting of downsampling (i.e., the downsampling stage), an intermediate stage, and upsampling (i.e., the upsampling stage).

[0081] It should be understood that the 3×3 Conv layer (i.e., the first 3×3 convolutional layer) is used to extract the input X i (i.e., the target training image).

[0082] Specifically, the noise estimation network of the partitioned diffusion model is iteratively trained, denoted as The agreed variable iteration records the number of iterations of the current network training (iteration is initially 0); the objective function (i.e., loss function) of the noise estimation network is defined as: Among them, T = 1000, at any time t, the model Output (i.e. depth image prediction noise) and noise with depth information (i.e., the real noise of the depth image) calculates the L1 loss function and updates the network parameters θ.

[0083] Specifically, when this round of iteration ends, set iteration = iteration + 1; use the Adam optimizer to train the network; when iteration reaches 2000K times, the network training stops and the training model ddpm_dehaze.pt file is saved.

[0084] It should be understood that the feature resolution is changed to 3×128×128 by a 3×3 Conv layer (i.e., the second 3×3 convolution layer), which is the output of the network, i.e., a noise block with depth information. (i.e. depth image prediction noise).

[0085] Specifically, using noise with depth information (i.e., the real noise of the depth image) is used as supervision, so that the network can learn the depth information of the image, so as to realize the diffusion and removal of fog partitions at different depths, thereby maximizing the defogging efficiency and more effectively removing fog at a long distance.

[0086] In the above embodiment, the image defogging model is obtained by performing model analysis on the diffusion model through the target image training set and the real noise data set of the depth image, which improves the problem of incomplete defogging caused by inaccurate modeling of the distribution of foggy scenes, avoids the problem of excessive defogging of nearby objects and insufficient defogging of distant objects, and realizes the network's diffusion and removal of fog partitions at different depths, maximizes the defogging efficiency, and more effectively removes long-distance fog, providing more accurate defogging processing for scenes at different depths.

[0087] Optionally, as an embodiment of the present invention, the downsampling stage includes a plurality of sequentially arranged first residual blocks, a plurality of sequentially arranged downsampling layers, a plurality of sequentially arranged second residual blocks, and a first self-attention layer having the same number as the second residual blocks and arranged in sequence,

[0088] The process of performing downsampling analysis on each of the first training image features respectively in the downsampling stage to obtain second training image features corresponding to each of the original foggy training images includes:

[0089] Taking every three of the first residual blocks as a group, thereby obtaining a plurality of first residual block groups;

[0090] Extracting features of each of the first training images through the first residual block group to obtain fifth training image features corresponding to each of the original foggy training images;

[0091] Downsampling each of the fifth training image features through the first downsampling layer, and inputting the downsampling results into the second first residual block group until passing through the last downsampling layer, thereby obtaining sixth training image features corresponding to each of the original foggy training images;

[0092] Extracting features of each of the sixth training images through the first second residual block to obtain seventh training image features corresponding to each of the original foggy training images;

[0093] The feature relationship of each of the seventh training image features is captured through the first first self-attention layer, and the captured results are input into the second second residual block until passing through the last first self-attention layer, thereby obtaining the second training image features corresponding to each of the original foggy training images.

[0094] It should be understood that when entering the downsampling stage, it first passes through four groups of alternating residual blocks (ResBlock) (i.e., the first residual block) and downsampling layers (Downsample), each group of which contains 3 ResBlocks and 1 Downsample, and then passes through three groups of alternating residual blocks (ResBlock) (i.e., the second residual block) and self-attention layers (Self-Attention) (i.e., the first self-attention layer). During this period, the number of channels gradually increases, and the image resolution gradually halves, resulting in a feature size of 256×8×8 (i.e., the second training image feature).

[0095] Preferably, the number of the first residual blocks may be 12, the number of the downsampling layers may be 4, and the numbers of the second residual blocks and may both be 3.

[0096] It should be understood that the data processing steps of the first residual block and the second residual block are the same, and only the processed data are different.

[0097] Specifically, feature extraction is performed on each of the first training image features through the first first residual block group, and the output result of the first first residual block group is input into the first downsampling layer, and the output result of the first downsampling layer is input into the second first residual block group, and the output result of the second first residual block group is input into the second downsampling layer, and so on, until the last downsampling layer is passed, thereby obtaining the sixth training image features corresponding to each of the original foggy training images.

[0098] Specifically, feature extraction is performed on each of the sixth training image features through the first second residual block, and the output result of the first second residual block is input into the first first self-attention layer, and the output result of the first first self-attention layer is input into the second second residual block, and the output result of the second second residual block is input into the second first self-attention layer, and so on, until the last first self-attention layer is passed, thereby obtaining the second training image features corresponding to each of the original foggy training images.

[0099] In the above embodiment, the first training image features are downsampled and analyzed in the downsampling stage to obtain the second training image features, which can obtain more accurate image features and depth information, reduce the difficulty of model learning, and provide more accurate dehazing processing for scenes of different depths.

[0100] Optionally, as an embodiment of the present invention, the intermediate stage includes a third residual block, a second self-attention layer and a fourth residual block,

[0101] The process of performing feature enhancement on each of the second training image features in the intermediate stage to obtain third training image features corresponding to each of the original foggy training images includes:

[0102] Extracting features of each of the second training images by using the third residual block to obtain eighth training image features corresponding to each of the original foggy training images;

[0103] Capturing feature relationships of each of the eighth training image features through the second self-attention layer to obtain ninth training image features corresponding to each of the original foggy training images;

[0104] The fourth residual block is used to extract features of each of the ninth training images to obtain third training image features corresponding to each of the original foggy training images.

[0105] It should be understood that the data processing steps of the third residual block and the fourth residual block are the same as those of the first residual block, and only the processed data are different. The data processing steps of the first self-attention layer and the second self-attention layer are the same, and only the processed data are different.

[0106] Specifically, the intermediate stage includes one ResBlock (i.e., the third residual block), one Self-Attention layer (i.e., the second self-attention layer), and one ResBlock (i.e., the fourth residual block), while maintaining the feature resolution.

[0107] In the above embodiment, the third training image features are obtained by performing feature enhancement on each second training image feature in the intermediate stage, thereby enhancing the low-resolution features, avoiding the problem of excessive defogging of nearby objects and insufficient defogging of distant objects, and reducing the difficulty of model learning.

[0108] Optionally, as an embodiment of the present invention, the upsampling stage includes a plurality of sequentially arranged fifth residual blocks, a third self-attention layer having the same number as the fifth residual blocks, a plurality of sequentially arranged upsampling blocks, and a plurality of sequentially arranged sixth residual blocks,

[0109] The process of performing upsampling analysis on each of the third training image features in the upsampling stage to obtain fourth training image features corresponding to each of the original foggy training images includes:

[0110] Extracting features of each of the third training images through the first fifth residual block to obtain tenth training image features corresponding to each of the original foggy training images;

[0111] Capturing the feature relationship of each of the tenth training image features through the first third self-attention layer, and inputting the captured results into the second fifth residual block until passing through the last third self-attention layer, thereby obtaining the eleventh training image features corresponding to each of the original foggy training images;

[0112] Taking every three of the sixth residual blocks as a group, thereby obtaining a plurality of second residual block groups;

[0113] Upsampling each of the eleventh training image features by a first upsampling block to obtain a twelfth training image feature corresponding to each of the original foggy training images;

[0114] The first second residual block group is used to extract features of each of the twelfth training images, and the results of the feature extraction are input into the second upsampling block until the last second residual block group is passed, thereby obtaining fourth training image features corresponding to each of the original foggy training images.

[0115] Preferably, the number of the fifth residual block and the third self-attention layer can be 3, the number of the upsampling blocks can be 4, and the number of the sixth residual blocks can be 12.

[0116] It should be understood that the data processing steps of the fifth residual block and the sixth residual block are the same as those of the first residual block, and only the processed data are different. The data processing steps of the third self-attention layer are the same as those of the first self-attention layer, and only the processed data are different.

[0117] Specifically, the structure of the upsampling stage is symmetrical to that of the downsampling stage, and the input of each layer is the concatenation of the input features of the layer and the features of the corresponding layer in the downsampling stage according to the number of channels.

[0118] Specifically, feature extraction is performed on each of the third training image features through the first fifth residual block, and the output result of the first fifth residual block is input into the first third self-attention layer, and the output result of the first third self-attention layer is input into the second fifth residual block, and the output result of the second fifth residual block is input into the second third self-attention layer, and so on, until the last third self-attention layer is passed, thereby obtaining the eleventh training image feature corresponding to each of the original foggy training images.

[0119] Specifically, feature extraction is performed on each of the eleventh training image features through the first upsampling block, and the output result of the first upsampling block is input into the first second residual block group, and the output result of the first second residual block group is input into the second upsampling block, and the output result of the second upsampling block is input into the second second residual block group, and so on, until the last second residual block group is passed, thereby obtaining the fourth training image features corresponding to each of the original foggy training images.

[0120] In the above embodiment, the third training image features are upsampled and analyzed in the upsampling stage to obtain the fourth training image features, which improves the problem of incomplete defogging caused by inaccurate modeling of the distribution of foggy scenes, avoids excessive defogging of nearby objects and insufficient defogging of distant objects, and realizes the network's diffusion and removal of fog partitions at different depths, maximizes the defogging efficiency, and more effectively removes long-distance fog, providing more accurate defogging processing for scenes at different depths.

[0121] Optionally, as another embodiment of the present invention, the present invention improves the problem of incomplete defogging caused by the inaccurate modeling of the distribution of foggy scenes caused by the diffusion model. By introducing image depth information and integrating it into the noise generation process of the diffusion model, more accurate defogging processing is provided for scenes of different depths. Specifically, the partitioned diffusion model first divides the image into multiple regions according to the depth map, and each region represents a different depth range. Then, according to the depth information and haze concentration of each region, the diffusion parameters of the defogging model are dynamically adjusted to ensure that the defogging processing of each region can achieve the best effect. This method not only reduces the difficulty of model learning, but also enables the network to adapt to the defogging tasks of scenes of different depths. The present invention makes full use of the generation ability of the diffusion model, and adds scene depth information to the noise to guide image defogging, providing a new direction for image defogging tasks.

[0122] Optionally, as another embodiment of the present invention, the present invention aims to solve the problem that the diffusion model is difficult to model foggy scenes. The existing image restoration based on the diffusion model usually processes the entire image uniformly and fails to reasonably utilize the depth information, which easily leads to incomplete and uneven defogging. Therefore, a partitioned diffusion model image defogging method based on scene depth is proposed. By adding the scene depth information partitions to the noise of the diffusion process, the parameters of the diffusion model are dynamically adjusted according to the depth information and noise intensity of each area to ensure that the defogging processing of each area can achieve the best effect.

[0123] Optionally, as another embodiment of the present invention, the present invention uses O-haze containing 45 image pairs as a test set, and after importing the pre-trained model weights into the test code, the following sampling algorithm is used:

[0124] The implicit sampling strategy is used to set the sampling steps to 20 steps, and t~[1,T],T=1000 is divided into 20 time periods. i and at time t x i Noise-perturbed image Input to the noise estimation network The estimated noisy image with depth information is obtained, denoted as Then after The sampling algorithm samples it to obtain a progressively denoised image By analogy, after S-1 steps of sampling, the restored haze-free image is finally obtained. SSIM and PSNR are calculated with the real haze-free image to evaluate the results.

[0125] Optionally, as another embodiment of the present invention, the present invention uses peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to compare with six existing deep learning-based image dehazing algorithms, as shown in Table 1. The O-haze test set was used for evaluation to compare the PSNR / SSIM values ​​of different image dehazing methods. Table 1 shows the PSNR / SSIM values ​​of the present invention and the six deep learning-based image dehazing methods.

[0126] Table 1

[0127]

[0128] Figure 3 A module block diagram of an image defogging device provided by an embodiment of the present invention.

[0129] Optionally, as another embodiment of the present invention, Figure 3 As shown, an image defogging device comprises:

[0130] Import module, used to import original image training set and image test set;

[0131] A preprocessing module, used to preprocess the original image training set to obtain a target image training set and a depth image real noise data set;

[0132] A model analysis module is used to construct a diffusion model, and perform model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model;

[0133] The image defogging result obtaining module is used to perform image defogging processing on the image test set through the image defogging model to obtain the image defogging result.

[0134] Optionally, another embodiment of the present invention provides an image defogging system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image defogging method described above is implemented. The system may be a computer or other system.

[0135] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image defogging method as described above is implemented.

[0136] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image defogging method, characterized in that: The steps include: Importing an original image training set and an image test set, preprocessing the original image training set to obtain a target image training set and a depth image real noise data set; Constructing a diffusion model, and performing model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model; The image defogging model is used to perform image defogging processing on the image test set to obtain an image defogging result.

2. The image defogging method according to claim 1, characterized in that: The original image training set includes a plurality of original foggy training images and original fog-free training images corresponding to each of the original foggy training images. The process of preprocessing the original image training set to obtain the target image training set and the depth image real noise data set includes: Respectively scaling each of the original foggy training images and each of the original fog-free training images to obtain scaled foggy training images corresponding to each of the original foggy training images and scaled fog-free training images corresponding to each of the original foggy training images; Partitioning the scene depths of each of the scaled fog-free training images respectively to obtain depth partition maps corresponding to each of the original foggy training images; Importing Gaussian noise, adding the Gaussian noise to each of the depth partition maps to obtain a true depth image noise corresponding to each of the original foggy training images, and collecting all the true depth image noises to obtain a depth image true noise dataset; Respectively adding the real noise of each of the depth images to the scaled fog-free training images corresponding to each of the original foggy training images to obtain the noisy fog-free training images corresponding to each of the original foggy training images; Each of the noise-free foggy training images and the scaled foggy training images corresponding to each of the original foggy training images are respectively connected channel by channel to obtain a target training image corresponding to each of the original foggy training images, and all of the target training images are collected to obtain a target image training set.

3. The image defogging method according to claim 2, characterized in that: The process of partitioning the scene depth of each of the scaled fog-free training images to obtain a depth partition map corresponding to each of the original foggy training images includes: Performing depth estimation on each of the scaled fog-free training images using a depth estimation model to obtain an original depth map corresponding to each of the original foggy training images; Scaling each of the original depth maps respectively to obtain a target depth map corresponding to each of the original foggy training images; The depth partitioning is performed on each of the target depth maps through the depth embedding layer to obtain a depth partition map corresponding to each of the original foggy training images.

4. The image defogging method according to claim 3, characterized in that: The target depth map includes a plurality of pixel positions and original depth values ​​corresponding to each of the pixel positions. The process of performing depth partitioning on each of the target depth maps through the depth embedding layer to obtain depth partition maps corresponding to each of the original foggy training images comprises: The maximum value of all the original depth values ​​in each of the target depth maps is respectively screened out, and the maximum depth value of each of the target depth maps is obtained after screening; Respectively filter out the minimum value of all the original depth values ​​in each of the target depth maps, and obtain the minimum depth value of each of the target depth maps after filtering; If the original depth value corresponding to each pixel position satisfies the first formula, the first preset value is used as the target depth value corresponding to the pixel position, and the first formula is: Among them, D i (x1) is the original depth value corresponding to the x1-th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map; If the original depth value corresponding to each of the pixel positions satisfies the second formula, the second preset value is used as the target depth value corresponding to the pixel position, and the second formula is: Among them, D i (x2) is the original depth value corresponding to the x2th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map; If the original depth value corresponding to each pixel position satisfies the third formula, the third preset value is used as the target depth value corresponding to the pixel position. The third formula is: Among them, D i (x3) is the original depth value corresponding to the x3th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map; If the original depth value corresponding to each pixel position satisfies the fourth formula, the fourth preset value is used as the target depth value corresponding to the pixel position. The fourth formula is: Among them, D i (x4) is the original depth value corresponding to the x4th pixel position in the i-th target depth map, is the minimum depth value of the i-th target depth map, is the maximum depth value of the i-th target depth map; The target depth values ​​corresponding to all the pixel positions in each of the target depth maps are plotted respectively to obtain depth partition maps corresponding to each of the original foggy training images.

5. The image defogging method according to claim 2, characterized in that: The diffusion model includes a first 3×3 convolutional layer, a downsampling stage, an intermediate stage, an upsampling stage, and a second 3×3 convolutional layer. The process of constructing a diffusion model, performing model analysis on the diffusion model through the target image training set and the depth image real noise data set, and obtaining an image defogging model includes: Performing feature extraction on each of the target training images through the first 3×3 convolutional layer to obtain first training image features corresponding to each of the original foggy training images; Downsampling and analyzing each of the first training image features respectively in the downsampling stage to obtain second training image features corresponding to each of the original foggy training images; Performing feature enhancement on each of the second training image features in the intermediate stage to obtain third training image features corresponding to each of the original foggy training images; Performing upsampling analysis on each of the third training image features in the upsampling stage to obtain fourth training image features corresponding to each of the original foggy training images; Extracting features of each of the fourth training images through the second 3×3 convolutional layer to obtain depth image prediction noise corresponding to each of the original foggy training images; The loss function corresponding to each of the original foggy training images is calculated by performing loss function calculation on each of the predicted noises of the depth images and the real noises of the depth images corresponding to each of the original foggy training images through the fifth formula, and the loss function corresponding to each of the original foggy training images is obtained. The fifth formula is: Where L(θ) j is the loss function corresponding to the jth original foggy training image, t is the time, T is the preset duration, x j is the scaled fog-free training image corresponding to the jth original foggy training image, is the real noise of the depth image corresponding to the jth original foggy training image, Predict the noise of the depth image corresponding to the jth original foggy training image, || ||1 is the L1 norm; The parameters of the diffusion model are updated according to the Adam optimizer and all the loss functions to obtain an image defogging model.

6. The image defogging method according to claim 5, characterized in that: The downsampling stage includes a plurality of sequentially arranged first residual blocks, a plurality of sequentially arranged downsampling layers, a plurality of sequentially arranged second residual blocks, and a plurality of sequentially arranged first self-attention layers, the number of which is the same as that of the second residual blocks and which are sequentially arranged. The process of performing downsampling analysis on each of the first training image features respectively in the downsampling stage to obtain second training image features corresponding to each of the original foggy training images includes: Taking every three of the first residual blocks as a group, thereby obtaining a plurality of first residual block groups; Extracting features of each of the first training images through the first residual block group to obtain fifth training image features corresponding to each of the original foggy training images; Downsampling each of the fifth training image features through the first downsampling layer, and inputting the downsampling results into the second first residual block group until passing through the last downsampling layer, thereby obtaining sixth training image features corresponding to each of the original foggy training images; Extracting features of each of the sixth training images through the first second residual block to obtain seventh training image features corresponding to each of the original foggy training images; The feature relationship of each of the seventh training image features is captured through the first first self-attention layer, and the captured results are input into the second second residual block until passing through the last first self-attention layer, thereby obtaining the second training image features corresponding to each of the original foggy training images.

7. The image defogging method according to claim 5, characterized in that: The intermediate stage includes a third residual block, a second self-attention layer, and a fourth residual block. The process of performing feature enhancement on each of the second training image features in the intermediate stage to obtain third training image features corresponding to each of the original foggy training images includes: Extracting features of each of the second training images by using the third residual block to obtain eighth training image features corresponding to each of the original foggy training images; Capturing feature relationships of each of the eighth training image features through the second self-attention layer to obtain ninth training image features corresponding to each of the original foggy training images; The fourth residual block is used to extract features of each of the ninth training images to obtain third training image features corresponding to each of the original foggy training images.

8. The image defogging method according to claim 5, characterized in that: The upsampling stage includes a plurality of sequentially arranged fifth residual blocks, a third self-attention layer having the same number as the fifth residual blocks, a plurality of sequentially arranged upsampling blocks, and a plurality of sequentially arranged sixth residual blocks, The process of performing upsampling analysis on each of the third training image features in the upsampling stage to obtain fourth training image features corresponding to each of the original foggy training images includes: Extracting features of each of the third training images through the first fifth residual block to obtain tenth training image features corresponding to each of the original foggy training images; Capturing the feature relationship of each of the tenth training image features through the first third self-attention layer, and inputting the captured results into the second fifth residual block until passing through the last third self-attention layer, thereby obtaining the eleventh training image features corresponding to each of the original foggy training images; Taking every three of the sixth residual blocks as a group, thereby obtaining a plurality of second residual block groups; Upsampling each of the eleventh training image features by a first upsampling block to obtain a twelfth training image feature corresponding to each of the original foggy training images; The first second residual block group is used to extract features of each of the twelfth training images, and the results of the feature extraction are input into the second upsampling block until the last second residual block group is passed, thereby obtaining fourth training image features corresponding to each of the original foggy training images.

9. An image defogging device, characterized in that: include: Import module, used to import original image training set and image test set; A preprocessing module, used to preprocess the original image training set to obtain a target image training set and a depth image real noise data set; A model analysis module is used to construct a diffusion model, and perform model analysis on the diffusion model through the target image training set and the depth image real noise data set to obtain an image defogging model; The image defogging result obtaining module is used to perform image defogging processing on the image test set through the image defogging model to obtain the image defogging result.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image defogging method according to any one of claims 1 to 8 is implemented.

Images