Semi-supervised Image Dehazing Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a semi-supervised image dehazing method, apparatus, computer device, and storage medium. The method includes: training a synthetic image dataset according to a preliminary constructed dehazing model, a preset discriminator, and a real clear dataset to obtain a first loss value; training a real image dataset according to the preliminary constructed dehazing model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value; updating the parameters of the preliminary constructed dehazing model based on the first loss value and the second loss value to obtain an updated dehazing model; and performing dehazing processing on a to-be-processed image according to the updated dehazing model to obtain a dehazed image. By using this method, through training based on different image datasets, more features of true and false data can be better learned, and then a dehazing image closer to the real situation can be generated, which can effectively process non-uniform foggy image scenes and reduce the dependence on a large number of foggy and fog-free image pairs.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and particularly to a semi-supervised image dehazing method, device, computer device, and storage medium. Background Art

[0002] With the rapid development of autonomous driving, intelligent monitoring, and remote sensing technologies, the importance of computer vision in practical applications has been continuously increasing. However, image processing in the real world faces various complex challenges, among which the impact of haze weather on image quality is particularly significant. Haze restricts the line of sight, affects the clarity, contrast, and details of the image, and seriously hinders the normal progress of visual tasks. Especially in non-uniform haze scenes, the distribution of haze is usually irregular and dynamically changing, which poses a greater challenge to image dehazing.

[0003] In related technologies, convolutional neural networks (CNNs) have been widely used in the dehazing task of foggy images, and can learn the features of images through large-scale data, automatically learn the mapping relationship from foggy images to haze-free images, and thus have stronger adaptability. In particular, convolutional neural networks (CNNs) can capture detailed information and global context in images through hierarchical structures, thereby effectively improving the dehazing effect. Many studies adopt an end-to-end network structure, where the input is a foggy image and the network output is a clear image after dehazing. In addition, some network models further optimize the dehazing effect by introducing techniques such as residual learning and generative adversarial networks (GANs).

[0004] However, current related technologies usually require a large amount of labeled data for training, especially paired foggy and haze-free images. However, pairs of foggy and haze-free images in the real world are scarce, and the shooting conditions are complex and variable, so it is difficult to collect enough high-quality training data. And many deep learning methods are prone to problems such as color distortion and contrast unevenness when processing non-uniform foggy images. Since the distribution of haze is often non-uniform and the concentration of fog varies greatly in different regions, it makes it difficult for dehazing algorithms to maintain the accuracy of color and details in different regions simultaneously. At the same time, deep learning models usually have a high computational complexity, especially when processing high-resolution images, the required computational resources and time costs are very high, which limits the application of these models in real-time processing or embedded devices. Summary of the Invention

[0005] Based on this, it is necessary to provide a semi-supervised image dehazing method, device, computer device, computer-readable storage medium, and computer program product for the above technical problems, which can solve the cost problem of collecting a large number of foggy images and corresponding clear images for training and effectively process non-uniform foggy image scenarios.

[0006] In a first aspect, the present application provides a semi-supervised image dehazing method. The method includes:

[0007] Training a synthetic image dataset according to a initially constructed dehazing model, a preset discriminator, and a real clear dataset to obtain a first loss value;

[0008] Training a real image dataset according to the initially constructed dehazing model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value;

[0009] Updating the parameters of the initially constructed dehazing model based on the first loss value and the second loss value to obtain an updated dehazing model;

[0010] Performing dehazing processing on the image to be processed according to the updated dehazing model to obtain a dehazed image.

[0011] In one embodiment, training a synthetic image dataset according to the initially constructed dehazing model, a preset discriminator, and a real clear dataset to obtain a first loss value includes:

[0012] Performing image processing on the synthetic image dataset according to the initially constructed dehazing model to obtain a synthetic dehazed dataset;

[0013] Analyzing and discriminating the synthetic dehazed dataset according to the real clear dataset, the preset discriminator, and a preset first loss function to obtain a first loss value.

[0014] In one embodiment, analyzing and discriminating the synthetic dehazed dataset according to the real clear dataset, the preset discriminator, and a preset first loss function to obtain a first loss value includes:

[0015] Extracting information from the real clear dataset and the synthetic dehazed dataset to obtain real clear high and low frequency information and synthetic dehazed high and low frequency information;

[0016] Concatenating the real clear dataset and the real clear high and low frequency information to obtain a real clear concatenated set;

[0017] Concatenating the synthetic dehazed dataset and the synthetic dehazed high and low frequency information to obtain a synthetic dehazed concatenated set;

[0018] Training according to the real clear concatenated set, the synthetic dehazed concatenated set, the preset discriminator, and the preset first loss function to obtain a first loss value.

[0019] In one embodiment, training according to the real clear concatenated set, the synthetic dehazed concatenated set, the preset discriminator, and the preset first loss function to obtain a first loss value includes:

[0020] Calculate the loss based on a real clear dataset, a synthetic haze-removal dataset, and a mean squared error loss function to obtain a mean squared error loss value;

[0021] Calculate the loss based on a synthetic haze-removal stitching set, a preset discriminator, and an adversarial loss function to obtain an adversarial loss value;

[0022] Set the mean squared error loss value and the adversarial loss value as the first loss value.

[0023] In one embodiment, train a real image dataset according to a preliminary constructed haze-removal model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value, including:

[0024] Perform image processing on the real image dataset according to the preliminary constructed haze-removal model to obtain a real haze-removal dataset;

[0025] Extract features from the real image dataset, the real haze-removal dataset, and the synthetic clear dataset respectively to obtain a real image feature set, a real haze-removal feature set, and a synthetic clear feature set;

[0026] Perform difference analysis on the real image feature set, the real haze-removal feature set, and the synthetic clear feature set to obtain a feature similarity dataset;

[0027] Calculate the loss of the feature similarity dataset according to a preset second loss function to obtain a second loss value.

[0028] In one embodiment, calculate the loss of the feature similarity dataset according to a preset second loss function to obtain a second loss value, including:

[0029] Calculate the loss of the feature similarity dataset based on a contrast loss function to obtain a contrast loss value;

[0030] Calculate the loss of the feature similarity dataset based on a dark channel loss function to obtain a dark channel loss value;

[0031] Calculate the loss of the feature similarity dataset based on a total variation loss function to obtain a total variation loss value;

[0032] Set the contrast loss value, the dark channel loss value, and the total variation loss value as the second loss value.

[0033] In one embodiment, update the parameters of the preliminary constructed haze-removal model based on the first loss value and the second loss value to obtain an updated haze-removal model, including:

[0034] Calculate the total loss according to the first loss value and the second loss value to obtain a total loss value;

[0035] Update the parameters of the initially constructed haze removal model based on the total loss value to obtain the updated haze removal model.

[0036] In a second aspect, the present application also provides a semi-supervised image haze removal device. The device includes:

[0037] A first training module for training the synthetic image dataset according to the initially constructed haze removal model, a preset discriminator, and a real clear dataset to obtain a first loss value;

[0038] A second training module for training the real image dataset according to the initially constructed haze removal model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value;

[0039] A parameter update module for updating the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain the updated haze removal model;

[0040] An image haze removal module for performing haze removal processing on the image to be processed according to the updated haze removal model to obtain the haze-removed image.

[0041] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Train the synthetic image dataset according to the initially constructed haze removal model, a preset discriminator, and a real clear dataset to obtain a first loss value;

[0043] Train the real image dataset according to the initially constructed haze removal model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value;

[0044] Update the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain the updated haze removal model;

[0045] Perform haze removal processing on the image to be processed according to the updated haze removal model to obtain the haze-removed image.

[0046] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Train the synthetic image dataset according to the initially constructed haze removal model, a preset discriminator, and a real clear dataset to obtain a first loss value;

[0048] Train the real image dataset according to the initially constructed haze removal model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value;

[0049] Update the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain an updated haze removal model;

[0050] Perform haze removal processing on the image to be processed according to the updated haze removal model to obtain a haze-removed image.

[0051] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Train the synthetic image dataset according to the initially constructed haze removal model, a preset discriminator, and a real clear dataset to obtain a first loss value;

[0053] Train the real image dataset according to the initially constructed haze removal model, a contrast regularization strategy, and a synthetic clear dataset to obtain a second loss value;

[0054] Update the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain an updated haze removal model;

[0055] Perform haze removal processing on the image to be processed according to the updated haze removal model to obtain a haze-removed image.

[0056] The above semi-supervised image haze removal method, device, computer device, storage medium, and computer program product train the generated synthetic image for haze removal through the constructed haze removal model, then use the discriminator to perform difference analysis on the haze-removed synthetic image and the clear image, and calculate the loss value of the haze removal training, that is, the first loss value; then use the constructed haze removal model to perform haze removal on the real image, and then perform feature extraction on the real haze-removed image, the real image, and the synthetic clear image respectively, and then use contrast regularization for difference analysis and calculate the loss value between the feature maps, that is, the second loss value; combine the first loss value and the second loss value to update the parameters and weights of the constructed haze removal model. Through training based on different image datasets, it can better learn more characteristics of true and false data, and then better enable the haze removal model to remove haze from real hazy images to generate haze-removed images closer to the real situation, and solve the cost problem of collecting a large number of hazy images and corresponding clear images for training, and can effectively process non-uniform hazy image scenes, reducing the dependence on a large number of hazy and haze-free image pairs. Description of the Drawings

[0057] Figure 1 It is a schematic flowchart of a semi-supervised image haze removal method in an embodiment;

[0058] Figure 2 It is a schematic framework diagram of a semi-supervised image haze removal model in an embodiment;

[0059] Figure 3 It is a schematic flowchart of the steps for calculating the first loss value in an embodiment;

[0060] Figure 4 It is a schematic flowchart of the steps for calculating the second loss value in an embodiment;

[0061] Figure 5 It is a structural block diagram of a semi-supervised image dehazing device in an embodiment;

[0062] Figure 6 It is an internal structure diagram of a computer device in an embodiment. Specific embodiments

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] In one embodiment, as Figure 1 shown, a semi-supervised image dehazing method is provided. In this embodiment, the method is exemplified by being applied to a terminal. It can be understood that the method can also be applied to a server and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0065] Step 102, training a synthetic image dataset according to a preliminary constructed dehazing model, a preset discriminator, and a real clear dataset to obtain a first loss value.

[0066] Exemplarily, a model for performing dehazing operations on hazy images is preliminarily constructed, that is, a preliminary constructed dehazing model. The preliminary constructed dehazing model includes a Swin-Dehaze network and adopts an encoder-decoder structure. The Swin-Dehaze network is constructed based on Swin Transformer, and the feature extraction ability of the network is enhanced by introducing the Transformer structure. This network structure can more effectively capture the global and local features of the image when processing the image dehazing task.

[0067] Select an appropriate number of synthetic image datasets from the synthetic dataset RESIDE (REalistic Single Image Dehazing Dataset, an image dehazing dataset). The synthetic image dataset includes a synthetic hazy image dataset. Input the synthetic hazy image dataset into the preliminary constructed dehazing model, and use the preliminary constructed dehazing model to remove the haze of the synthetic hazy image to obtain a synthetic dehazed image.

[0068] After that, the real clear image corresponding to the synthesized foggy image and the synthesized defogged image are simultaneously input into the discriminator. The discriminator is used to identify the difference between the synthesized defogged image and the corresponding real clear image, and calculate the loss of the synthesized defogged image compared with the real clear image during the defogging process.

[0069] The above process is a supervised defogging learning process.

[0070] Step 104: Train the real image dataset according to the initially constructed defogging model, the contrast regularization strategy, and the synthesized clear dataset to obtain a second loss value.

[0071] Exemplarily, a certain number of foggy images, i.e., unlabeled real foggy images, are randomly selected from URHI and RTTS, and the same number of clear images, i.e., the synthesized clear dataset, are randomly selected from the synthesized dataset ITS (indoor training set).

[0072] Input the real image dataset, i.e., the unlabeled real foggy images, into the initially constructed defogging model to obtain the defogged real images. Feature extraction is performed on the real image dataset, the defogged real images, and the synthesized clear dataset respectively to obtain the feature maps of these images. Then, the contrast regularization strategy is used to calculate the similarity and the loss during the defogging process, i.e., the second loss value, so as to update the constructed defogging model using the similarity and the loss during the defogging process in the subsequent steps.

[0073] The above process is an unsupervised defogging learning process.

[0074] Step 106: Update the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model.

[0075] Exemplarily, after completing the alternating training using the same defogging model, the calculated loss values during the alternating training process are calculated according to a preset total loss calculation method, so as to update the initially constructed defogging model based on the total loss in the subsequent steps, i.e., obtain the updated defogging model.

[0076] The framework schematic diagram of the initially constructed defogging model trained above is as Figure 2 shown. In addition to the above training process, it can also be evaluated through objective evaluation indicators such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and the performance of the updated defogging model can also be verified using the comprehensive target test set (SOTS) and the mixed subjective test set (HSTS).

[0077] Step 108: Perform defogging processing on the image to be processed according to the updated defogging model to obtain a defogged image.

[0078] Exemplarily, after completing the training of the defogging model and obtaining the updated defogging model, the foggy image to be processed is input into the updated defogging model, and the fog of the foggy image to be processed is removed by using the updated defogging model, that is, the defogged image corresponding to the foggy image to be processed is obtained.

[0079] In the above semi-supervised image defogging method, the generated synthetic image is defogged and trained by the constructed defogging model, and then the discriminator is used to perform difference analysis on the defogged synthetic image and the clear image, and the loss value of the defogging training, that is, the first loss value, is calculated; then the constructed defogging model is used to perform defogging operation on the real image, and then the real defogged image, the real image and the synthetic clear image are respectively subjected to feature extraction, and then difference analysis is performed by using contrast regularization and the loss value between the feature maps, that is, the second loss value, is calculated; the parameters and weights of the constructed defogging model are updated by combining the first loss value and the second loss value. Through training based on different image data sets, more features of true and false data can be better learned, and then the defogging model can better defog the real foggy image to generate a defogged image closer to the real situation, and solve the cost problem of collecting a large number of foggy images and corresponding clear images for training, and can effectively process non-uniform foggy image scenes and reduce the dependence on a large number of foggy and fog-free image pairs.

[0080] In an exemplary embodiment, training the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain the first loss value includes:

[0081] Performing image processing on the synthetic image data set according to the initially constructed defogging model to obtain a synthetic defogged data set; analyzing and discriminating the synthetic defogged data set according to the real clear data set, the preset discriminator and the preset first loss function to obtain the first loss value.

[0082] Exemplarily, input the synthetic foggy image into the Swin-Dehaze network, that is, the initially constructed defogging model, and complete the removal of the fog of the synthetic foggy image through the Swin-Dehaze network. Specifically:

[0083] After inputting the synthetic foggy image perform a 3X3 convolution operation on it to obtain shallow features , where H is the height of the image, W is the width of the image, and C is the number of channels; then input the shallow features into a 3-layer symmetric encoder-decoder architecture to extract deep features to obtain deep features . Then perform a 3×3 convolution and soft reconstruction operation on the deep features to restore its corresponding fog-free image .

[0084] The Swin-Dehaze network generally consists of an Encoder, a Bottleneck, and a Decoder. The Encoder includes a downsampling layer and Swin Transformer blocks. The downsampling layer uses disjoint convolution to divide the synthesized hazy image into non-overlapping small blocks and increase the number of channels. Five residual blocks are set in the Bottleneck module to extract more complex and deeper features in the low-resolution space and remove haze simultaneously. The Decoder includes an upsampling layer and Swin Transformer blocks. The upsampling layer uses Pixel Shuffle to aggregate the corresponding downsampling layer patches and reduce the number of channels.

[0085] The above process is the process of initially constructing a dehazing model to remove the haze from the synthesized hazy image. After obtaining the synthesized dehazing dataset, the synthesized dehazing dataset and the real clear dataset are input into the discriminator together, so that the feature representations of the synthesized dehazing dataset and the real clear dataset are as close or far away as possible.

[0086] In an exemplary embodiment, analyzing and discriminating the synthesized dehazing dataset according to the real clear dataset, a preset discriminator, and a preset first loss function to obtain a first loss value, including:

[0087] Extracting information from the real clear dataset and the synthesized dehazing dataset to obtain the high and low frequency information of the real clear and the high and low frequency information of the synthesized dehazing; splicing the real clear dataset and the high and low frequency information of the real clear to obtain a real clear splicing set; splicing the synthesized dehazing dataset and the high and low frequency information of the synthesized dehazing to obtain a synthesized dehazing splicing set; training according to the real clear splicing set, the synthesized dehazing splicing set, the preset discriminator, and the preset first loss function to obtain a first loss value.

[0088] Exemplarily, after completing the dehazing operation, performing Fourier transforms on the synthesized dehazing dataset and the real clear dataset respectively to obtain the high frequency information and low frequency information of the synthesized dehazing dataset, that is, the high and low frequency information of the synthesized dehazing, and the high frequency information and low frequency information of the real clear dataset, that is, the high and low frequency information of the real clear.

[0089] Then, the synthesized dehazing image and its corresponding high frequency information and low frequency information are spliced to obtain the enhanced features of the synthesized dehazing image. Similarly, the real clear image is processed in the same way to obtain the enhanced features of the real clear image. The enhanced features of the synthesized dehazing image and the enhanced features of the real clear image are input into the discriminator. The discriminator is trained by minimizing the difference between the true and false samples, so as to learn to distinguish the dehazing image from the real haze-free image. During the training process, the loss of generating the synthesized dehazing image, that is, the first loss value, is calculated based on the preset first loss function.

[0090] During the training process, according to the min-max game problem expression of the GAN (Generative Adversarial) network:

[0091]

[0092] The optimization problem can be expressed as a min-max optimization formula as follows:

[0093]

[0094] Where, is a continuous operation, J is the real clear image, The low-frequency component corresponding to the real clear image, The high-frequency component corresponding to the real clear image, is the synthesized haze-removed image, is the low-frequency component corresponding to the synthesized haze-removed image, The high-frequency component corresponding to the synthesized haze-removed image.

[0095] In an exemplary embodiment, as Figure 3 shown, training is performed according to the real clear stitching set, the synthesized haze-removed stitching set, the preset discriminator, and the preset first loss function to obtain the first loss value, including:

[0096] Step 302, based on the real clear dataset, the synthesized haze-removed dataset, and the mean squared error loss function, calculate the loss to obtain the mean squared error loss value.

[0097] Exemplarily, during the process of the discriminator analyzing the difference between the synthesized haze-removed dataset and the real clear dataset, the mean squared error loss function is used to calculate the loss of the haze-removing process, specifically:

[0098]

[0099] Where, is the vector form of the synthesized haze-removed image, J is the vector form of the real clear image, is the number of labeled samples in the mini-batch.

[0100] Step 304, based on the synthesized haze-removed stitching set, the preset discriminator, and the adversarial loss function, calculate the loss to obtain the adversarial loss value.

[0101] Exemplarily, through adversarial learning, the generative adversarial network can effectively improve the result. To guide the network to generate more natural and haze-free images, the adversarial loss of the generative adversarial network needs to be utilized. The adversarial loss is specifically:

[0102]

[0103] Where, For continuous operation, is the discriminator network.

[0104] Step 306, set the mean squared error loss value and the adversarial loss value as the first loss value.

[0105] Exemplarily, after calculating the mean squared error loss value and the adversarial loss value, it is necessary to wait for the subsequent training to complete, and use them together with the losses calculated during the subsequent training process as the overall optimization objective. Therefore, the mean squared error loss value and the adversarial loss value are set as the losses for this training stage.

[0106] In an exemplary embodiment, the real image dataset is trained according to the initially constructed dehazing model, the contrast regularization strategy, and the synthetic clear dataset to obtain the second loss value, including:

[0107] Perform image processing on the real image dataset according to the initially constructed dehazing model to obtain the real dehazed dataset; perform feature extraction on the real image dataset, the real dehazed dataset, and the synthetic clear dataset respectively to obtain the real image feature set, the real dehazed feature set, and the synthetic clear feature set; perform difference analysis on the real image feature set, the real dehazed feature set, and the synthetic clear feature set to obtain the feature similarity dataset; calculate the loss of the feature similarity dataset according to the preset second loss function to obtain the second loss value.

[0108] Among them, the contrast regularization strategy includes the preset second loss function.

[0109] Exemplarily, a certain number of foggy images, i.e., unlabeled real foggy images, are randomly selected from URHI and RTTS, and the same number of clear images, i.e., the synthetic clear dataset, are randomly selected from the synthetic dataset ITS (indoor training set).

[0110] After selecting the image dataset, the unlabeled real foggy images, i.e., the real image dataset, are input into the Swin-Dehaze network, i.e., the initially constructed dehazing model, to obtain the real dehazed dataset. Then the real image dataset, the real dehazed dataset, and the synthetic clear dataset are all input into the pre-trained VGG-19 model to extract the feature maps of the input images. Then, based on the contrast regularization strategy, positive and negative pairs are constructed. The positive pairs in the positive and negative pairs include the synthetic clear images in the synthetic clear dataset and the real dehazed images in the real dehazed dataset, and the negative pairs in the positive and negative pairs include the real dehazed images in the real dehazed dataset and the real foggy images in the real image dataset.

[0111] After constructing the positive and negative pairs, calculate the similarity between the feature maps corresponding to the two types of images in the positive pairs, and calculate the difference between the feature maps corresponding to the two types of images in the negative pairs using a distance calculation formula (including but not limited to the L1 distance). At the same time, design a loss function, that is, preset a second loss function, to minimize the distance between the positive pairs and maximize the distance between the negative pairs, thereby obtaining a second loss value.

[0112] In addition to constructing positive and negative pairs, the contrast regularization expressions between the synthesized clear images in the synthesized clear dataset, the real dehazed images in the real dehazed dataset, and the real hazy images in the real image dataset are as follows:

[0113]

[0114] Among them, is the contrast regularization term, are the features extracted in the same latent feature space.

[0115] In an exemplary embodiment, as Figure 4 shown, calculate the loss of the feature similarity dataset according to the preset second loss function to obtain a second loss value, including:

[0116] Step 402, calculate the loss of the feature similarity dataset based on the contrast loss function to obtain a contrast loss value.

[0117] Exemplarily, use a pre-trained VGG-19 model to extract the feature maps of different samples to capture the feature differences between the samples. Based on these feature maps, use the Pixel-wise Contrastive Loss as a constraint, specifically:

[0118]

[0119] Among them, is the i-th hidden feature extracted from a fixed pre-trained model, is the L1 distance between x and y, is the weight coefficient.

[0120] Step 404, calculate the loss of the feature similarity dataset based on the dark channel loss function to obtain a dark channel loss value.

[0121] Exemplarily, in order to help the model generate dehazed images with similar statistical characteristics to real clear images, channel prior constraints and gradient prior losses are adopted. During the image degradation process, the sparsity of the dark channel will be significantly reduced. Therefore, in order to effectively restore the clarity of the image, it is usually necessary to introduce dark channel constraints to enhance the ability of the dehazing network to restore image details and depth information. The dark channel constraint is specifically:

[0122]

[0123] where x and y are pixel coordinates, is the c-th color channel, is the image patch centered at x.

[0124] L1 regularization is used to constrain the sparsity of the predicted image, specifically:

[0125]

[0126] where is the vector form of the dark channel of the predicted image.

[0127] Step 406: Calculate the loss for the feature similarity dataset based on the total variation loss function to obtain the total variation loss value.

[0128] Exemplarily, by analyzing the gradient features of a large number of clear images, it can be found that the gradient changes of these images are usually relatively gentle in the horizontal and vertical directions, which reflects the structural characteristics of natural images. Based on this observation, using the total variation loss as a constraint can effectively guide the unsupervised defogging network to generate defogged images that are statistically similar to real clear images. The total variation loss promotes the predicted image to maintain a smooth change through L1 regularization of the image gradient, avoiding over-smoothing or detail loss, thereby retaining the structural information and details in the image. The specific form of this total variation loss is:

[0129]

[0130] where is the horizontal operation differential matrix, is the vertical operation differential matrix.

[0131] Step 408: Set the contrast loss value, the dark channel loss value, and the total variation loss value as the second loss value.

[0132] Exemplarily, after obtaining the above losses, they are collectively referred to as the second loss value, and combined with the previously obtained first loss value as the overall optimization objective.

[0133] In an exemplary embodiment, updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain the updated defogging model includes:

[0134] Calculating the total loss value according to the first loss value and the second loss value; updating the parameters of the initially constructed defogging model based on the total loss value to obtain the updated defogging model.

[0135] Exemplarily, after obtaining the first loss value and the second loss value, for the overall loss of the network, it is necessary to construct an overall loss function to optimize the entire network, specifically as follows:

[0136]

[0137] Among them, 、 、 、 are the weights of four loss terms respectively.

[0138] After determining the overall loss, the parameters and weights of the Swin-Dehaze network are updated by backpropagation and gradient descent methods to obtain the updated dehazing model.

[0139] In an exemplary embodiment, the present application provides a semi-supervised image dehazing method, and the method includes:

[0140] Performing image processing on the synthetic image dataset according to the initially constructed dehazing model to obtain a synthetic dehazing dataset.

[0141] Extracting information from the real clear dataset and the synthetic dehazing dataset to obtain real clear high and low frequency information and synthetic dehazing high and low frequency information.

[0142] Concatenating the real clear dataset and the real clear high and low frequency information to obtain a real clear concatenated set.

[0143] Concatenating the synthetic dehazing dataset and the synthetic dehazing high and low frequency information to obtain a synthetic dehazing concatenated set.

[0144] Calculating the loss based on the real clear dataset, the synthetic dehazing dataset and the mean square error loss function to obtain the mean square error loss value.

[0145] Calculating the loss based on the synthetic dehazing concatenated set, the preset discriminator and the adversarial loss function to obtain the adversarial loss value.

[0146] Setting the mean square error loss value and the adversarial loss value as the first loss value.

[0147] Performing image processing on the real image dataset according to the initially constructed dehazing model to obtain a real dehazing dataset.

[0148] Performing feature extraction on the real image dataset, the real dehazing dataset and the synthetic clear dataset respectively to obtain a real image feature set, a real dehazing feature set and a synthetic clear feature set.

[0149] Performing difference analysis on the real image feature set, the real dehazing feature set and the synthetic clear feature set to obtain a feature similarity dataset.

[0150] Calculate the loss of the feature similarity dataset based on the contrast loss function to obtain the contrast loss value.

[0151] Calculate the loss of the feature similarity dataset based on the dark channel loss function to obtain the dark channel loss value.

[0152] Calculate the loss of the feature similarity dataset based on the total variation loss function to obtain the total variation loss value.

[0153] Set the contrast loss value, the dark channel loss value, and the total variation loss value as the second loss value.

[0154] Calculate the total loss based on the first loss value and the second loss value to obtain the total loss value.

[0155] Update the parameters of the initially constructed defogging model based on the total loss value to obtain the updated defogging model.

[0156] Perform defogging processing on the image to be processed according to the updated defogging model to obtain the defogged image.

[0157] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown sequentially according to the arrows, these steps do not necessarily need to be executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0158] Based on the same inventive concept, an embodiment of the present application also provides a semi-supervised image defogging device for implementing the semi-supervised image defogging method described above. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the semi-supervised image defogging device provided below can refer to the limitations on the semi-supervised image defogging method in the above text, and will not be repeated here.

[0159] In one embodiment, as Figure 5 shown, a semi-supervised image defogging device is provided, including: a first training module 502, a second training module 504, a parameter update module 506, and an image defogging module 508, where:

[0160] The first training module 502 is used to train the synthetic image dataset according to the initially constructed haze removal model, the preset discriminator, and the real clear dataset to obtain the first loss value;

[0161] The second training module 504 is used to train the real image dataset according to the initially constructed haze removal model, the contrast regularization strategy, and the synthetic clear dataset to obtain the second loss value;

[0162] The parameter update module 506 is used to update the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain the updated haze removal model;

[0163] The image haze removal module 508 is used to perform haze removal processing on the image to be processed according to the updated haze removal model to obtain the haze-removed image.

[0164] In an exemplary embodiment, the first training module 502 is further used to perform image processing on the synthetic image dataset according to the initially constructed haze removal model to obtain a synthetic haze-removed dataset; analyze and discriminate the synthetic haze-removed dataset according to the real clear dataset, the preset discriminator, and the preset first loss function to obtain the first loss value.

[0165] In an exemplary embodiment, the first training module 502 is further used to extract information from the real clear dataset and the synthetic haze-removed dataset to obtain the real clear high and low frequency information and the synthetic haze-removed high and low frequency information; splice the real clear dataset and the real clear high and low frequency information to obtain a real clear splicing set; splice the synthetic haze-removed dataset and the synthetic haze-removed high and low frequency information to obtain a synthetic haze-removed splicing set; train according to the real clear splicing set, the synthetic haze-removed splicing set, the preset discriminator, and the preset first loss function to obtain the first loss value.

[0166] In an exemplary embodiment, the first training module 502 is further used to calculate the loss based on the real clear dataset, the synthetic haze-removed dataset, and the mean square error loss function to obtain the mean square error loss value; calculate the loss based on the synthetic haze-removed splicing set, the preset discriminator, and the adversarial loss function to obtain the adversarial loss value; set the mean square error loss value and the adversarial loss value as the first loss value.

[0167] In an exemplary embodiment, the second training module 504 is further used to perform image processing on the real image dataset according to the initially constructed haze removal model to obtain a real haze-removed dataset; perform feature extraction on the real image dataset, the real haze-removed dataset, and the synthetic clear dataset respectively to obtain a real image feature set, a real haze-removed feature set, and a synthetic clear feature set; perform difference analysis on the real image feature set, the real haze-removed feature set, and the synthetic clear feature set to obtain a feature similarity dataset; calculate the loss based on the preset second loss function for the feature similarity dataset to obtain the second loss value.

[0168] In an exemplary embodiment, the second training module 504 is further configured to calculate a loss of the feature similarity data set based on a contrast loss function to obtain a contrast loss value; calculate a loss of the feature similarity data set based on a dark channel loss function to obtain a dark channel loss value; calculate a loss of the feature similarity data set based on a total variation loss function to obtain a total variation loss value; and set the contrast loss value, the dark channel loss value, and the total variation loss value as the second loss value.

[0169] In an exemplary embodiment, the parameter update module 506 is further configured to calculate a total loss based on the first loss value and the second loss value to obtain a total loss value; and update parameters of the initially constructed dehazing model based on the total loss value to obtain an updated dehazing model.

[0170] Each module in the above semi-supervised image dehazing device can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in a processor in a computer device in a hardware form or independent thereof, or stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the above respective modules.

[0171] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structural diagram may be as Figure 6 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a semi-supervised image dehazing method.

[0172] Those skilled in the art can understand that Figure 6 the structure shown in

[0173] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0175] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0177] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0178] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0179] The above-described embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A semi-supervised image dehazing method, characterized in that, The method includes: Inputting the synthesized foggy image dataset into the initially constructed dehazing model, and using the initially constructed dehazing model to remove the fog from the synthesized foggy images to obtain a synthesized dehazed dataset; Based on the real clear dataset and the synthesized dehazed dataset, during the process of the discriminator analyzing the differences between the synthesized dehazed dataset and the real clear dataset, using the mean square error loss function to calculate the loss during the dehazing process to obtain the mean square error loss value; Performing Fourier transform on the real clear dataset and the synthesized dehazed dataset to obtain the real clear high and low frequency information and the synthesized dehazed high and low frequency information; and splicing the real clear dataset and the real clear high and low frequency information to obtain a real clear splicing set; splicing the synthesized dehazed dataset and the synthesized dehazed high and low frequency information to obtain a synthesized dehazed splicing set; Inputting the synthesized dehazed splicing set and the real clear splicing set into the discriminator, and during the process of the discriminator training by minimizing the differences between true and false samples, using the adversarial loss function to calculate the loss to obtain the adversarial loss value; Setting the mean square error loss value and the adversarial loss value as the first loss value; training the real image dataset according to the initially constructed dehazing model, the contrast regularization strategy, and the synthesized clear dataset to obtain the second loss value; wherein, the contrast regularization strategy includes a preset second loss function; Updating the parameters of the initially constructed dehazing model based on the first loss value and the second loss value to obtain an updated dehazing model; Performing dehazing processing on the image to be processed according to the updated dehazing model to obtain a dehazed image; The training the real image dataset according to the initially constructed dehazing model, the contrast regularization strategy, and the synthesized clear dataset to obtain the second loss value includes: performing image processing on the real image dataset according to the initially constructed dehazing model to obtain a real dehazed dataset; respectively performing feature extraction on the real image dataset, the real dehazed dataset, and the synthesized clear dataset to obtain a real image feature set, a real dehazed feature set, and a synthesized clear feature set; performing difference analysis on the real image feature set, the real dehazed feature set, and the synthesized clear feature set to obtain a feature similarity dataset; calculating the loss of the feature similarity dataset according to the preset second loss function to obtain the second loss value; inputting the unlabeled real foggy images, i.e., the real image dataset, into the Swin-Dehaze network, i.e., the initially constructed dehazing model, to obtain a real dehazed dataset; then inputting the real image dataset, the real dehazed dataset, and the synthesized clear dataset into the pre-trained VGG-19 model to extract the feature maps of the input images; then constructing positive and negative pairs based on the contrast regularization strategy, where the positive pairs in the positive and negative pairs include the synthesized clear images in the synthesized clear dataset and the real dehazed images in the real dehazed dataset, and the negative pairs in the positive and negative pairs include the real dehazed images in the real dehazed dataset and the real foggy images in the real image dataset.

2. The method according to claim 1, characterized in that The preset second loss function includes a contrast loss function, a dark channel loss function, and a total variation loss function; the calculating the loss of the feature similarity data set according to the preset second loss function to obtain a second loss value includes: Calculating the loss of the feature similarity data set based on the contrast loss function to obtain a contrast loss value; Calculating the loss of the feature similarity data set based on the dark channel loss function to obtain a dark channel loss value; Calculating the loss of the feature similarity data set based on the total variation loss function to obtain a total variation loss value; Setting the contrast loss value, the dark channel loss value, and the total variation loss value as the second loss value.

3. The method according to claim 1, wherein The updating the parameters of the initially constructed haze removal model based on the first loss value and the second loss value to obtain an updated haze removal model includes: Calculating the total loss based on the first loss value and the second loss value to obtain a total loss value; Updating the parameters of the initially constructed haze removal model based on the total loss value to obtain an updated haze removal model.

4. A semi-supervised image dehazing device, characterized in that, The device includes: A first training module, configured to input a synthetic hazy image data set into an initially constructed haze removal model, use the initially constructed haze removal model to remove the haze of the synthetic hazy image to obtain a synthetic haze-free data set; based on the real clear data set and the synthetic haze-free data set, during the discriminator analyzes the difference between the synthetic haze-free data set and the real clear data set, calculate the loss of the haze removal process using the mean square error loss function to obtain a mean square error loss value; perform Fourier transform on the real clear data set and the synthetic haze-free data set to obtain real clear high and low frequency information and synthetic haze-free high and low frequency information; and splice the real clear data set and the real clear high and low frequency information to obtain a real clear splicing set; splice the synthetic haze-free data set and the synthetic haze-free high and low frequency information to obtain a synthetic haze-free splicing set; input the synthetic haze-free splicing set and the real clear splicing set into the discriminator, and use the adversarial loss function to calculate the loss during the process of the discriminator training by minimizing the difference between true and false samples to obtain an adversarial loss value; set the mean square error loss value and the adversarial loss value as the first loss value; The second training module is used to train the real image dataset according to the initially constructed dehazing model, the contrast regularization strategy, and the synthetic clear dataset to obtain a second loss value; wherein, the contrast regularization strategy includes a preset second loss function; it is also used to perform image processing on the real image dataset according to the initially constructed dehazing model to obtain a real dehazed dataset; respectively extract features from the real image dataset, the real dehazed dataset, and the synthetic clear dataset to obtain a real image feature set, a real dehazed feature set, and a synthetic clear feature set; perform difference analysis on the real image feature set, the real dehazed feature set, and the synthetic clear feature set to obtain a feature similarity dataset; calculate the loss of the feature similarity dataset according to the preset second loss function to obtain a second loss value; it is also used to input the unlabeled real hazy image, that is, the real image dataset, into the Swin-Dehaze network, that is, the initially constructed dehazing model, to obtain a real dehazed dataset; then the real image dataset, the real dehazed dataset, and the synthetic clear dataset are all input into the pre-trained VGG-19 model to extract the feature maps of the input images; then, based on the contrast regularization strategy, positive and negative pairs are constructed. The positive pairs in the positive and negative pairs include the synthetic clear images in the synthetic clear dataset and the real dehazed images in the real dehazed dataset, and the negative pairs in the positive and negative pairs include the real dehazed images in the real dehazed dataset and the real hazy images in the real image dataset; The parameter update module is used to update the parameters of the initially constructed dehazing model based on the first loss value and the second loss value to obtain an updated dehazing model; The image dehazing module is used to perform dehazing processing on the image to be processed according to the updated dehazing model to obtain a dehazed image.

5. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 3.

Images