Semi-supervised image defogging method and device, computer equipment and storage medium

Abstract

The invention relates to a semi-supervised image defogging method and device, computer equipment and a storage medium. The method comprises the following steps: training a synthetic image data set according to an initially constructed defogging model, a preset discriminator and a real clear data set to obtain a first loss value; training the real image data set according to the initially constructed defogging model, the comparison regularization strategy and the synthesized clear data set to obtain a second loss value; performing parameter updating on the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model; and carrying out defogging processing on the to-be-processed image according to the updated defogging model to obtain a defogged image. By adopting the method, more characteristics of true and false data can be better learned through training based on different image data sets, so that a defogged image closer to a real situation is generated, a non-uniform fog image scene can be effectively processed, and the dependence on a large number of foggy and fogless image pairs is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a semi-supervised image defogging method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of autonomous driving, intelligent monitoring and remote sensing technology, the importance of computer vision in practical applications continues to increase. However, image processing in the real world faces various complex challenges, among which haze weather has a particularly significant impact on image quality. Haze limits the viewing distance, affects the clarity, contrast and details of the image, and seriously hinders the normal progress of visual tasks. Especially in non-uniform haze scenes, the distribution of haze is usually irregular and dynamically changing, which brings greater challenges to image dehazing.

[0003] Among the related technologies, convolutional neural networks (CNNs) are widely used in the task of defogging foggy images. They can learn the features of images through large-scale data and automatically learn the mapping relationship from foggy images to fog-free images, so they have stronger adaptability. In particular, convolutional neural networks (CNNs) can capture detailed information and global context in images through hierarchical structures, thereby effectively improving the defogging effect. Many studies use an end-to-end network structure, in which the input is a foggy image and the network output is a clear image after defogging. In addition, some network models further optimize the defogging effect by introducing technologies such as residual learning and generative adversarial networks (GANs).

[0004] However, current related technologies usually require a large amount of labeled data for training, especially pairs of foggy and fog-free images. However, pairs of foggy and fog-free images in the real world are relatively scarce, and the shooting conditions are complex and changeable, so it is difficult to collect enough high-quality training data. In addition, many deep learning methods are prone to color distortion and uneven contrast when dealing with non-uniform fog images. Since the distribution of haze is often non-uniform, the concentration of fog varies greatly in different areas, which makes it difficult for dehazing algorithms to maintain the accuracy of color and details in different areas at the same time. At the same time, deep learning models usually have high computational complexity, especially when processing high-resolution images, the required computing resources and time cost are very high, which limits the application of these models in real-time processing or embedded devices. Summary of the invention

[0005] Based on this, it is necessary to provide a semi-supervised image defogging method, device, computer equipment, computer-readable storage medium and computer program product that can solve the cost problem of collecting a large number of fog images and corresponding clear images for training and effectively handle non-uniform fog scenes in response to the above technical problems.

[0006] In a first aspect, the present application provides a semi-supervised image defogging method. The method comprises: The synthetic image dataset is trained according to the initially constructed dehazing model, the preset discriminator and the real clear dataset to obtain a first loss value; The real image dataset is trained according to the initially constructed dehazing model, the comparative regularization strategy and the synthetic clear dataset to obtain the second loss value; Based on the first loss value and the second loss value, the parameters of the initially constructed defogging model are updated to obtain an updated defogging model; The image to be processed is defogged according to the updated defogging model to obtain a defogged image.

[0007] In one embodiment, training the synthetic image dataset according to the initially constructed defogging model, the preset discriminator and the real clear dataset to obtain a first loss value includes: Perform image processing on the synthetic image data set according to the initially constructed defogging model to obtain a synthetic defogging data set; The synthetic dehazed data set is analyzed and judged according to the real clear data set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0008] In one embodiment, analyzing and discriminating the synthetic dehazed data set according to the real clear data set, the preset discriminator and the preset first loss function to obtain the first loss value includes: Extract information from real clear data sets and synthetic dehazed data sets to obtain real clear high and low frequency information and synthetic dehazed high and low frequency information; The real clear data set and the real clear high and low frequency information are spliced ​​to obtain a real clear spliced ​​set; The synthetic dehazing data set and the synthetic dehazing high and low frequency information are spliced ​​to obtain a synthetic dehazing spliced ​​set; Training is performed according to the real clear splicing set, the synthetic dehazed splicing set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0009] In one embodiment, training is performed according to the real clear stitching set, the synthetic defogging stitching set, the preset discriminator and the preset first loss function to obtain the first loss value, including: The loss is calculated based on the real clear data set, the synthetic dehazed data set and the mean square error loss function to obtain the mean square error loss value; The loss is calculated based on the synthetic dehazing concatenation set, the preset discriminator and the adversarial loss function to obtain the adversarial loss value; Set the mean square error loss value and the adversarial loss value to the first loss value.

[0010] In one embodiment, training a real image dataset according to the initially constructed dehazing model, the contrast regularization strategy, and the synthetic clear dataset to obtain a second loss value includes: Perform image processing on the real image dataset according to the initially constructed dehazing model to obtain a real dehazing dataset; Feature extraction is performed on the real image dataset, the real defogging dataset and the synthetic clear dataset respectively to obtain the real image feature set, the real defogging feature set and the synthetic clear feature set; Perform difference analysis on real image feature set, real defogging feature set and synthetic clear feature set to obtain feature similarity dataset; The loss of the feature similarity data set is calculated according to the preset second loss function to obtain a second loss value.

[0011] In one embodiment, performing loss calculation on the feature similarity data set according to a preset second loss function to obtain a second loss value includes: Based on the contrast loss function, the loss of the feature-similar data set is calculated to obtain the contrast loss value; Based on the dark channel loss function, the loss of the feature similarity data set is calculated to obtain the dark channel loss value; Based on the total variation loss function, the loss of feature-similar data sets is calculated to obtain the total variation loss value; The contrast loss value, the dark channel loss value, and the total variation loss value are set as second loss values.

[0012] In one embodiment, updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain the updated defogging model includes: Calculate the total loss according to the first loss value and the second loss value to obtain the total loss value; The parameters of the initially constructed dehazing model are updated based on the total loss value to obtain an updated dehazing model.

[0013] In a second aspect, the present application also provides a semi-supervised image defogging device. The device comprises: A first training module is used to train the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain a first loss value; The second training module is used to train the real image data set according to the initially constructed dehazing model, the comparison regularization strategy and the synthetic clear data set to obtain a second loss value; A parameter updating module, used for updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model; The image defogging module is used to defog the image to be processed according to the updated defogging model to obtain a defogged image.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: The synthetic image dataset is trained according to the initially constructed dehazing model, the preset discriminator and the real clear dataset to obtain a first loss value; The real image dataset is trained according to the initially constructed dehazing model, the comparative regularization strategy and the synthetic clear dataset to obtain the second loss value; Based on the first loss value and the second loss value, the parameters of the initially constructed defogging model are updated to obtain an updated defogging model; The image to be processed is defogged according to the updated defogging model to obtain a defogged image.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented: The synthetic image dataset is trained according to the initially constructed dehazing model, the preset discriminator and the real clear dataset to obtain a first loss value; The real image dataset is trained according to the initially constructed dehazing model, the comparative regularization strategy and the synthetic clear dataset to obtain the second loss value; Based on the first loss value and the second loss value, the parameters of the initially constructed defogging model are updated to obtain an updated defogging model; The image to be processed is defogged according to the updated defogging model to obtain a defogged image.

[0016] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: The synthetic image dataset is trained according to the initially constructed dehazing model, the preset discriminator and the real clear dataset to obtain a first loss value; The real image dataset is trained according to the initially constructed dehazing model, the comparative regularization strategy and the synthetic clear dataset to obtain the second loss value; Based on the first loss value and the second loss value, the parameters of the initially constructed defogging model are updated to obtain an updated defogging model; The image to be processed is defogged according to the updated defogging model to obtain a defogged image.

[0017] The above-mentioned semi-supervised image defogging method, device, computer equipment, storage medium and computer program product perform defogging training on the generated synthetic image through the constructed defogging model, and then use the discriminator to perform difference analysis between the defogged synthetic image and the clear image, and calculate the loss value of the defogging training, that is, the first loss value; then use the constructed defogging model to perform defogging operation on the real image, and then extract features of the real defogged image, the real image and the synthetic clear image respectively, and then use contrast regularization to perform difference analysis and calculate the loss value between the feature maps, that is, the second loss value; combine the first loss value and the second loss value to update the parameters and weights of the constructed defogging model, through training based on different image data sets, it can better learn more characteristics of true and false data, and then better enable the defogging model to defog the real foggy image to generate a defogged image that is closer to the real situation, and solve the cost problem of collecting a large number of foggy images and corresponding clear images for training, can effectively handle non-uniform foggy image scenes, and reduce dependence on a large number of foggy and non-foggy image pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of a process of a semi-supervised image dehazing method in one embodiment; Figure 2 Schematic diagram of a framework of a semi-supervised image dehazing model in one embodiment; Figure 3 A schematic diagram of a flow chart of a step of calculating a first loss value in one embodiment; Figure 4 A schematic diagram of a flow chart of a step of calculating a second loss value in one embodiment; Figure 5 is a structural block diagram of a semi-supervised image defogging device in one embodiment; Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In one embodiment, Figure 1 As shown, a semi-supervised image defogging method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps: Step 102: training the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain a first loss value.

[0021] For example, a model for performing a dehazing operation on a foggy image is initially constructed, i.e., a preliminary dehazing model. The preliminary dehazing model includes a Swin-Dehaze network and adopts an encoder-decoder structure. The Swin-Dehaze network is constructed based on the SwinTransformer, and the feature extraction capability of the network is enhanced by introducing the Transformer structure. This network structure can more effectively capture the global and local features of the image when processing the image dehazing task.

[0022] Select a suitable number of synthetic image datasets from the synthetic dataset RESIDE (REalistic Single Image Dehazing Dataset), which includes a synthetic foggy image dataset. Input the synthetic foggy image dataset into the initial constructed dehazing model, and use the initial constructed dehazing model to remove the fog of the synthetic foggy image to obtain a synthetic dehazed image.

[0023] Then, the real clear image corresponding to the synthetic foggy image and the synthetic defogged image are simultaneously put into the discriminator, and the difference between the synthetic defogged image and the corresponding real clear image is identified by the discriminator, and the loss of the synthetic defogged image compared with the real clear image during the defogging process is calculated.

[0024] The above process is a supervised dehazing learning process.

[0025] Step 104 , training the real image dataset according to the initially constructed dehazing model, the contrast regularization strategy and the synthetic clear dataset to obtain a second loss value.

[0026] Exemplarily, a certain number of foggy images, namely unlabeled real foggy images, are randomly selected from URHI and RTTS, and the same number of clear images, namely the synthetic clear dataset, are randomly selected from the synthetic dataset ITS (indoor training set).

[0027] The real image dataset, i.e., the unlabeled real foggy images, is input into the initially constructed dehazing model to obtain the real images after dehazing. Feature extraction is performed on the real image dataset, the real images after dehazing, and the synthetic clear dataset respectively to obtain feature maps of these images. Then, the similarity of these feature maps and the loss in the dehazing process, i.e., the second loss value, are calculated using the contrast regularization strategy, so as to facilitate the subsequent update of the constructed dehazing model using the similarity and the loss in the dehazing process.

[0028] The above process is an unsupervised dehazing learning process.

[0029] Step 106 , updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model.

[0030] Exemplarily, after completing alternating training using the same dehazing model, the loss value calculated and determined during the alternating training is calculated according to a preset total loss calculation method, so that the initially constructed dehazing model can be subsequently updated based on the total loss to obtain an updated dehazing model.

[0031] The framework diagram of the initial dehazing model trained above is as follows: Figure 2 As shown, in addition to the above training process, it can also be evaluated by objective evaluation indicators such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and the performance of the updated dehazing model can also be verified using the synthetic target test set (SOTS) and the mixed subjective test set (HSTS).

[0032] Step 108, defogging the image to be processed according to the updated defogging model to obtain a defogged image.

[0033] Exemplarily, after completing the training of the defogging model and obtaining the updated defogging model, the foggy image to be processed is input into the updated defogging model, and the updated defogging model is used to remove the fog from the foggy image to be processed, so as to obtain the defogged image corresponding to the foggy image to be processed.

[0034] In the above-mentioned semi-supervised image defogging method, the generated synthetic image is defogged by the constructed defogging model, and then the discriminator is used to perform difference analysis between the defogged synthetic image and the clear image, and the loss value of the defogging training, i.e., the first loss value, is calculated; then the constructed defogging model is used to perform defogging operation on the real image, and then the real defogged image, the real image and the synthetic clear image are respectively subjected to feature extraction, and then contrast regularization is used to perform difference analysis and calculate the loss value between the feature maps, i.e., the second loss value; the parameters and weights of the constructed defogging model are updated in combination with the first loss value and the second loss value, and through training based on different image data sets, more characteristics of true and false data can be better learned, so as to better enable the defogging model to defog the real foggy image to generate a defogged image that is closer to the real situation, and the cost problem of collecting a large number of foggy images and corresponding clear images for training is solved, and the non-uniform foggy image scenes can be effectively processed, and the dependence on a large number of foggy and non-foggy image pairs is reduced.

[0035] In an exemplary embodiment, training the synthetic image dataset according to the initially constructed defogging model, the preset discriminator and the real clear dataset to obtain a first loss value includes: The synthetic image data set is processed according to the initially constructed dehazing model to obtain a synthetic dehazing data set; the synthetic dehazing data set is analyzed and discriminated according to the real clear data set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0036] Exemplarily, the synthetic foggy image is input into the Swin-Dehaze network, i.e., the initially constructed dehazing model, and the fog of the synthetic foggy image is removed by the Swin-Dehaze network, specifically: Synthesize a foggy image at input After that, a 3X3 convolution operation is performed on it to obtain shallow features , where H is the height of the image, W is the width of the image, and C is the number of channels; the shallow features are then input into a 3-layer symmetric encoder-decoder architecture to extract deep features to obtain deep features Then we can use deep features Perform 3×3 convolution and soft reconstruction operations to restore its corresponding haze-free image .

[0037] The Swin-Dehaze network generally consists of the Encoder, BottleNeck and Decoder ends. The Encoder end includes a downsampling layer and a Swin Transformer block. The downsampling layer uses disjoint convolution to divide the synthetic foggy image into disjoint small blocks and increase the number of channels. Five residual blocks are set in the BottleNeck module to extract more complex and deeper features in the low-resolution space and remove fog at the same time. The Decoder end includes an upsampling layer and a Swin Transformer block. The upsampling layer uses Pixel Shuffle to aggregate the corresponding downsampling layer patches and reduce the number of channels.

[0038] The above process is the process of initially building a defogging model to remove the fog from the synthetic foggy image. After obtaining the synthetic defogging dataset, the synthetic defogging dataset and the real clear dataset are input into the discriminator together, so that the feature representations of the synthetic defogging dataset and the real clear dataset are as close or far as possible.

[0039] In an exemplary embodiment, analyzing and discriminating the synthetic dehazed data set according to the real clear data set, the preset discriminator and the preset first loss function to obtain the first loss value includes: Information is extracted from the real clear data set and the synthetic defogging data set to obtain real clear high and low frequency information and synthetic defogging high and low frequency information; the real clear data set and the real clear high and low frequency information are spliced ​​to obtain a real clear splicing set; the synthetic defogging data set and the synthetic defogging high and low frequency information are spliced ​​to obtain a synthetic defogging splicing set; training is performed according to the real clear splicing set, the synthetic defogging splicing set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0040] Exemplarily, after completing the dehazing operation, the synthetic dehazing data set and the real clear data set are respectively subjected to Fourier transform to obtain the high-frequency information and low-frequency information of the synthetic dehazing data set, i.e., the synthetic dehazing high- and low-frequency information, and the high-frequency information and low-frequency information of the real clear data set, i.e., the real clear high- and low-frequency information.

[0041] Then the synthetic defogging image and its corresponding high-frequency information and low-frequency information are spliced ​​to obtain the enhanced features of the synthetic defogging image. Similarly, the real clear image is processed in the same way to obtain the enhanced features of the real clear image. The enhanced features of the synthetic defogging image and the enhanced features of the real clear image are input into the discriminator. The discriminator is trained by minimizing the difference between true and false samples, so as to learn to distinguish the defogging image from the real non-fogging image. During the training process, the loss of the synthetic defogging image, i.e., the first loss value, is calculated based on the preset first loss function.

[0042] During the training process, the minimum-maximum game problem expression of the GAN (Generative Adversarial) network is:

[0043] The optimization problem can be formulated as a min-max optimization formula as follows:

[0044] in, is a continuous operation, J is a real and clear image, The low-frequency components corresponding to the real and clear image, The high-frequency components corresponding to the real and clear image, To synthesize the dehazed image, To synthesize the low-frequency components corresponding to the dehazed image, Synthesize the high-frequency components corresponding to the dehazed image.

[0045] In an exemplary embodiment, Figure 3 As shown, training is performed according to the real clear splicing set, the synthetic defogging splicing set, the preset discriminator and the preset first loss function to obtain the first loss value, including: Step 302 , performing loss calculation based on the real clear data set, the synthetic dehazed data set and the mean square error loss function to obtain a mean square error loss value.

[0046] For example, in the process of the discriminator performing difference analysis between the synthetic defogging data set and the real clear data set, the loss of the defogging process is calculated using the mean square error loss function, specifically:

[0047] in, is the vector form of the synthetic dehazed image, J is the vector form of the real clear image, is the number of labeled samples in a mini-batch.

[0048] Step 304 , performing loss calculation based on the synthetic dehazing concatenation set, the preset discriminator and the adversarial loss function to obtain an adversarial loss value.

[0049] For example, through adversarial learning, the generative adversarial network can effectively improve the results. To guide the network to generate more natural and fog-free images, it is necessary to use the adversarial loss of the generative adversarial network. The adversarial loss is specifically:

[0050] in, For continuous operation, is the discriminator network.

[0051] Step 306: Set the mean square error loss value and the adversarial loss value to the first loss value.

[0052] For example, after calculating the mean square error loss value and the adversarial loss value, it is necessary to wait for subsequent training to be completed, and use it together with the loss calculated in the subsequent training process as the overall optimization target, so the mean square error loss value and the adversarial loss value are set as the loss of this training stage.

[0053] In an exemplary embodiment, training a real image dataset according to the initially constructed defogging model, the contrast regularization strategy, and the synthetic clear dataset to obtain a second loss value includes: According to the initially constructed defogging model, image processing is performed on the real image dataset to obtain a real defogging dataset; feature extraction is performed on the real image dataset, the real defogging dataset and the synthetic clear dataset respectively to obtain a real image feature set, a real defogging feature set and a synthetic clear feature set; difference analysis is performed on the real image feature set, the real defogging feature set and the synthetic clear feature set to obtain a feature similarity dataset; loss calculation is performed on the feature similarity dataset according to a preset second loss function to obtain a second loss value.

[0054] Among them, the contrast regularization strategy includes presetting a second loss function.

[0055] Exemplarily, a certain number of foggy images, namely unlabeled real foggy images, are randomly selected from URHI and RTTS, and the same number of clear images, namely the synthetic clear dataset, are randomly selected from the synthetic dataset ITS (indoor training set).

[0056] After selecting the image dataset, the unlabeled real foggy image, i.e. the real image dataset, is input into the Swin-Dehaze network, i.e. the initially constructed dehazing model, to obtain the real dehazing dataset. Then the real image dataset, the real dehazing dataset, and the synthetic clear dataset are all input into the pre-trained VGG-19 model to extract the feature map of the input image. Then, positive and negative pairs are constructed based on the contrast regularization strategy. The positive pairs in the positive and negative pairs include the synthetic clear images in the synthetic clear dataset and the real dehazing images in the real dehazing dataset, and the negative pairs in the positive and negative pairs include the real dehazing images in the real dehazing dataset and the real foggy images in the real image dataset.

[0057] After constructing the positive and negative pairs, the similarity between the feature maps corresponding to the two types of images in the positive pair is calculated, and the difference between the feature maps corresponding to the two types of images in the negative pair is calculated using a distance calculation formula (including but not limited to L1 distance). At the same time, a loss function is designed, that is, a second loss function is preset to minimize the distance between the positive pairs and maximize the distance between the negative pairs, thereby obtaining a second loss value.

[0058] In addition to constructing positive and negative pairs, the contrast regularization expressions between synthetic clear images in the synthetic clear dataset, real defogging images in the real defogging dataset, and real foggy images in the real image dataset are as follows:

[0059] in, To compare the regularization term, are features extracted in the same latent feature space.

[0060] In an exemplary embodiment, Figure 4 As shown, the loss of the feature similarity data set is calculated according to the preset second loss function to obtain the second loss value, including: Step 402 , performing loss calculation on the feature similarity data set based on the contrast loss function to obtain a contrast loss value.

[0061] For example, a pre-trained VGG-19 model is used to extract feature maps of different samples to capture the feature differences between samples. Based on these feature maps, a pixel-wise contrastive loss is used as a constraint, specifically:

[0062] in, To extract the i-th hidden feature from a fixed pre-trained model, is the L1 distance between x and y, is the weight coefficient.

[0063] Step 404 , performing loss calculation on the feature similarity data set based on the dark channel loss function to obtain a dark channel loss value.

[0064] For example, in order to help the model generate a dehazed image with similar statistical characteristics to a real clear image, channel prior constraints and gradient prior loss are used. During the image degradation process, the sparsity of the dark channel will be significantly reduced. Therefore, in order to effectively restore the clarity of the image, it is usually necessary to introduce dark channel constraints to enhance the dehazing network's ability to restore image details and depth information. The dark channel constraints are specifically:

[0065] Among them, x and y are pixel coordinates, is the cth color channel, is the image patch centered at x.

[0066] L1 regularization is used to constrain the sparsity of the predicted image, which is as follows:

[0067] in, It is the vector form of the dark channel of the predicted image.

[0068] Step 406 , performing loss calculation on the feature similarity data set based on the total variation loss function to obtain a total variation loss value.

[0069] For example, by analyzing the gradient features of a large number of clear images, it can be found that the gradient changes of these images in the horizontal and vertical directions are usually relatively gentle, which reflects the structural characteristics of natural images. Based on this observation, the use of total variation loss as a constraint can effectively guide the unsupervised dehazing network to generate dehazed images that are similar in statistical characteristics to real clear images. The total variation loss promotes the predicted image to maintain smooth changes by L1 regularization of the image gradient, avoiding over-smoothing or loss of details, thereby retaining the structural information and details in the image. The total variation loss is specifically:

[0070] in, is the horizontal differential matrix, is the vertical differential matrix.

[0071] Step 408: setting the contrast loss value, the dark channel loss value, and the total variation loss value as second loss values.

[0072] Exemplarily, after the above losses are obtained, they are collectively referred to as second loss values, and combined with the first loss values ​​obtained previously as the overall optimization target.

[0073] In an exemplary embodiment, updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model includes: The total loss is calculated according to the first loss value and the second loss value to obtain a total loss value; and the parameters of the initially constructed defogging model are updated based on the total loss value to obtain an updated defogging model.

[0074] For example, after obtaining the first loss value and the second loss value, for the overall loss of the network, it is necessary to construct an overall loss function to optimize the entire network, specifically:

[0075] in, , , , are the weights of the four loss terms respectively.

[0076] After determining the overall loss, the parameters and weights of the Swin-Dehaze network are updated through back-propagation and gradient descent methods to obtain the updated dehazing model.

[0077] In an exemplary embodiment, the present application provides a semi-supervised image defogging method, the method comprising: The synthetic image dataset is processed according to the initially constructed dehazing model to obtain a synthetic dehazing dataset.

[0078] Information is extracted from the real clear data set and the synthetic dehazed data set to obtain the real clear high and low frequency information and the synthetic dehazed high and low frequency information.

[0079] The real clear data set and the real clear high and low frequency information are spliced ​​to obtain a real clear spliced ​​set.

[0080] The synthetic dehazing dataset and the synthetic dehazing high and low frequency information are spliced ​​to obtain the synthetic dehazing splicing set.

[0081] The loss is calculated based on the real clear data set, the synthetic dehazed data set and the mean square error loss function to obtain the mean square error loss value.

[0082] The loss is calculated based on the synthetic dehazing splicing set, the preset discriminator and the adversarial loss function to obtain the adversarial loss value.

[0083] Set the mean square error loss value and the adversarial loss value to the first loss value.

[0084] The real image dataset is processed according to the initially constructed dehazing model to obtain a real dehazing dataset.

[0085] Feature extraction is performed on the real image dataset, the real dehazing dataset and the synthetic clear dataset respectively to obtain the real image feature set, the real dehazing feature set and the synthetic clear feature set.

[0086] The difference analysis is performed on the real image feature set, the real dehazing feature set and the synthetic clarity feature set to obtain a feature similarity dataset.

[0087] Based on the contrast loss function, the loss of feature-similar data sets is calculated to obtain the contrast loss value.

[0088] Based on the dark channel loss function, the loss of the feature similarity data set is calculated to obtain the dark channel loss value.

[0089] Based on the total variation loss function, the loss of feature-similar data sets is calculated to obtain the total variation loss value.

[0090] The contrast loss value, the dark channel loss value, and the total variation loss value are set as second loss values.

[0091] The total loss is calculated according to the first loss value and the second loss value to obtain a total loss value.

[0092] The parameters of the initially constructed dehazing model are updated based on the total loss value to obtain an updated dehazing model.

[0093] The image to be processed is defogged according to the updated defogging model to obtain a defogged image.

[0094] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiment of the present application also provides a semi-supervised image defogging device for implementing the semi-supervised image defogging method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more semi-supervised image defogging device embodiments provided below can refer to the limitations of the semi-supervised image defogging method above, and will not be repeated here.

[0096] In one embodiment, Figure 5 As shown, a semi-supervised image defogging device is provided, comprising: a first training module 502, a second training module 504, a parameter updating module 506 and an image defogging module 508, wherein: A first training module 502 is used to train the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain a first loss value; A second training module 504 is used to train the real image data set according to the initially constructed defogging model, the comparison regularization strategy and the synthetic clear data set to obtain a second loss value; A parameter updating module 506 is used to update the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model; The image defogging module 508 is used to perform defogging on the image to be processed according to the updated defogging model to obtain a defogged image.

[0097] In an exemplary embodiment, the first training module 502 is also used to perform image processing on the synthetic image data set according to the initially constructed dehazing model to obtain a synthetic dehazing data set; analyze and discriminate the synthetic dehazing data set according to the real clear data set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0098] In an exemplary embodiment, the first training module 502 is also used to extract information from the real clear data set and the synthetic defogging data set to obtain real clear high and low frequency information and synthetic defogging high and low frequency information; splicing the real clear data set and the real clear high and low frequency information to obtain a real clear splicing set; splicing the synthetic defogging data set and the synthetic defogging high and low frequency information to obtain a synthetic defogging splicing set; training is performed according to the real clear splicing set, the synthetic defogging splicing set, the preset discriminator and the preset first loss function to obtain a first loss value.

[0099] In an exemplary embodiment, the first training module 502 is also used to perform loss calculation based on a real clear data set, a synthetic dehazed data set and a mean square error loss function to obtain a mean square error loss value; perform loss calculation based on a synthetic dehazed splicing set, a preset discriminator and an adversarial loss function to obtain an adversarial loss value; and set the mean square error loss value and the adversarial loss value to the first loss value.

[0100] In an exemplary embodiment, the second training module 504 is also used to perform image processing on the real image data set according to the initially constructed defogging model to obtain a real defogging data set; perform feature extraction on the real image data set, the real defogging data set and the synthetic clear data set respectively to obtain a real image feature set, a real defogging feature set and a synthetic clear feature set; perform difference analysis on the real image feature set, the real defogging feature set and the synthetic clear feature set to obtain a feature similarity data set; perform loss calculation on the feature similarity data set according to a preset second loss function to obtain a second loss value.

[0101] In an exemplary embodiment, the second training module 504 is also used to perform loss calculation on the feature similarity data set based on the contrast loss function to obtain a contrast loss value; perform loss calculation on the feature similarity data set based on the dark channel loss function to obtain a dark channel loss value; perform loss calculation on the feature similarity data set based on the total variation loss function to obtain a total variation loss value; and set the contrast loss value, the dark channel loss value and the total variation loss value as the second loss value.

[0102] In an exemplary embodiment, the parameter updating module 506 is further used to calculate the total loss based on the first loss value and the second loss value to obtain the total loss value; and to update the parameters of the initially constructed defogging model based on the total loss value to obtain the updated defogging model.

[0103] Each module in the above-mentioned semi-supervised image defogging device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0104] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a semi-supervised image defogging method is implemented.

[0105] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0108] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A semi-supervised image dehazing method, characterized in that: The method comprises: The synthetic image dataset is trained according to the initially constructed dehazing model, the preset discriminator and the real clear dataset to obtain a first loss value; The real image data set is trained according to the initially constructed defogging model, the contrast regularization strategy and the synthetic clear data set to obtain a second loss value; wherein the contrast regularization strategy includes a preset second loss function; Based on the first loss value and the second loss value, the parameters of the initially constructed defogging model are updated to obtain an updated defogging model; Defogging the image to be processed according to the updated defogging model to obtain a defogged image; The method of training the real image dataset according to the initially constructed defogging model, comparing the regularization strategy and the synthetic clear dataset to obtain a second loss value includes: performing image processing on the real image dataset according to the initially constructed defogging model to obtain a real defogging dataset; performing feature extraction on the real image dataset, the real defogging dataset and the synthetic clear dataset respectively to obtain a real image feature set, a real defogging feature set and a synthetic clear feature set; performing difference analysis on the real image feature set, the real defogging feature set and the synthetic clear feature set to obtain a feature similarity dataset; performing loss calculation on the feature similarity dataset according to a preset second loss function to obtain a second loss value.

2. The method according to claim 1, characterized in that The step of training the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain a first loss value includes: Perform image processing on the synthetic image data set according to the initially constructed defogging model to obtain a synthetic defogging data set; The synthetic dehazed data set is analyzed and discriminated according to the real clear data set, the preset discriminator and the preset first loss function to obtain a first loss value.

3. The method according to claim 2, characterized in that The step of analyzing and discriminating the synthetic dehazed data set according to the real clear data set, the preset discriminator and the preset first loss function to obtain a first loss value includes: Extracting information from the real clear data set and the synthetic defogging data set to obtain real clear high- and low-frequency information and synthetic defogging high- and low-frequency information; splicing the real clear data set and the real clear high and low frequency information to obtain a real clear splicing set; Splicing the synthetic defogging data set with the synthetic defogging high and low frequency information to obtain a synthetic defogging splicing set; Training is performed according to the real clear splicing set, the synthetic defogging splicing set, the preset discriminator and the preset first loss function to obtain a first loss value.

4. The method according to claim 3, characterized in that The preset first loss function includes a mean square error loss function and an adversarial loss function; the training is performed according to the real clear splicing set, the synthetic defogging splicing set, the preset discriminator and the preset first loss function to obtain a first loss value, including: Perform loss calculation based on the real clear data set, the synthetic dehazed data set and a mean square error loss function to obtain a mean square error loss value; Perform loss calculation based on the synthetic dehazing splicing set, the preset discriminator and the adversarial loss function to obtain an adversarial loss value; The mean square error loss value and the adversarial loss value are set as first loss values.

5. The method according to claim 1, characterized in that The preset second loss function includes a contrast loss function, a dark channel loss function and a total variation loss function; and the step of performing loss calculation on the feature similarity data set according to the preset second loss function to obtain a second loss value includes: Performing loss calculation on the feature similarity data set based on the contrast loss function to obtain a contrast loss value; Performing loss calculation on the feature similarity data set based on a dark channel loss function to obtain a dark channel loss value; Based on the total variation loss function, the loss of the feature similarity data set is calculated to obtain a total variation loss value; The contrast loss value, the dark channel loss value and the total variation loss value are set as second loss values.

6. The method according to claim 1, characterized in that The updating of the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model includes: Calculate the total loss according to the first loss value and the second loss value to obtain a total loss value; The parameters of the initially constructed defogging model are updated based on the total loss value to obtain an updated defogging model.

7. A semi-supervised image defogging device, characterized in that: The device comprises: A first training module is used to train the synthetic image data set according to the initially constructed defogging model, the preset discriminator and the real clear data set to obtain a first loss value; A second training module is used to train the real image data set according to the initially constructed defogging model, the contrast regularization strategy and the synthetic clear data set to obtain a second loss value; wherein the contrast regularization strategy includes a preset second loss function; it is also used to perform image processing on the real image data set according to the initially constructed defogging model to obtain a real defogging data set; perform feature extraction on the real image data set, the real defogging data set and the synthetic clear data set respectively to obtain a real image feature set, a real defogging feature set and a synthetic clear feature set; perform difference analysis on the real image feature set, the real defogging feature set and the synthetic clear feature set to obtain a feature similarity data set; perform loss calculation on the feature similarity data set according to the preset second loss function to obtain a second loss value; A parameter updating module, used for updating the parameters of the initially constructed defogging model based on the first loss value and the second loss value to obtain an updated defogging model; The image defogging module is used to perform defogging on the image to be processed according to the updated defogging model to obtain a defogged image.

8. The device according to claim 7, characterized in that The first training module is also used to perform image processing on the synthetic image data set according to the initially constructed defogging model to obtain a synthetic defogging data set; analyze and discriminate the synthetic defogging data set according to the real clear data set, a preset discriminator and a preset first loss function to obtain a first loss value.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images