4K image defogging method, device and storage medium based on weakly supervised contrast learning

Abstract

The present invention relates to the field of image defogging, and specifically discloses a 4K image defogging method, device and storage medium based on weakly supervised contrast learning, including S1, obtaining a dark channel prior image #imgabs1# after dark channel prior processing of a foggy image #imgabs0#; weighted fusion of the foggy image #imgabs2# and the processed image #imgabs3#, and finally obtaining a fused image #imgabs4#; S2, inputting the fused image #imgabs5# into a U-net convolutional neural network structure for processing, and inserting a residual block and introducing an SC attention mechanism during processing; S3, using PatchGAN as a discriminator, which divides the restored image #imgabs6# into many small overlapping areas, judging whether each small area is part of the real image, using the least squares GAN loss to train the network, and finally discriminating and obtaining the restored fog-free image #imgabs7#; S4, contrast learning loss function. The beneficial effects achieved by the present invention are: the high-resolution image defogging algorithm is short in time and can maintain a high-precision defogging effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image defogging, and in particular to a 4K image defogging method, device and storage medium based on weakly supervised contrast learning. Background Art

[0002] High-definition video technology has developed rapidly with the progress of society. Its main function is to help people find more details when acquiring video resources. "High-definition" means "high resolution". Today's mainstream high resolutions are 2k (2560×1440) and 4k (3840×2160), which are much higher than previous analog images (the highest is D1, i.e. 720×576). Image resolution has been qualitatively improved. For example, in the security industry, the application effect of video surveillance has been further affirmed in both real-time monitoring and post-investigation. However, with the development and popularization of new technologies, some problems will also arise. At present, one of the problems faced by digital high-definition images in the real world is that the details and contrast of the images will be significantly reduced in haze weather, and the effective recognition rate of the images is low. Therefore, it is becoming increasingly important to defog images taken in haze weather to actively and effectively improve image quality.

[0003] In recent years, although video dehazing algorithms have made significant progress, some problems still exist. First, both haze and detail information can cause blurred images and make them difficult to distinguish. Therefore, while removing haze, some detail information may be weakened, resulting in detail loss. Secondly, although using adjacent frame information for dehazing can effectively reduce the impact of haze in the target frame, it may also introduce noise and artifacts in adjacent frames, reducing the dehazing effect. In recent years, deep learning-based methods, convolutional neural networks (CNNs), have made significant progress in the field of image dehazing. These deep learning models can learn image features by training large amounts of data, thereby achieving more accurate distinction between details and haze without relying on adjacent frame information.

[0004] However, deep learning methods also have shortcomings. Traditional deep learning models usually have large parameters, high floating-point operations, large data throughput, and complex operation logic. These characteristics result in the inference operation consuming a large amount of computing resources, including floating-point units (GPUs), memory, communication bandwidth, and high-performance CPUs. These limitations make them difficult to deploy on resource-limited edge devices, which usually include smartphones, tablets, sensor nodes, embedded systems, IoT devices, etc., which usually have limited computing resources, storage capacity, and energy supply. In addition, image enhancement tasks such as high-resolution video image dehazing still face the following challenges: it is difficult to collect pairs of ultra-high-definition real-world foggy and clean image pairs; achieving ultra-high-definition real-time dehazing places higher requirements on the efficiency and accuracy of the algorithm.

[0005] In summary, this scheme will adopt a combination of global features and local features to provide stable, accurate, and fast UHD dehazing capabilities on resource-constrained and low-power edge devices, thereby improving the efficiency and practicality of the algorithm. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 4K image defogging method, device and storage medium based on weakly supervised contrast learning, so as to achieve a high-resolution image defogging algorithm with short time consumption and high-precision defogging effect.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] In a first aspect, one of the embodiments of the present application provides a 4K image defogging method based on weakly supervised contrast learning, comprising the following steps:

[0009] S1, image preparation processing;

[0010] The input foggy image After dark channel prior processing, it is corrected by the atmospheric scattering model to obtain the dark channel prior image. ;

[0011] The dark channel prior image With foggy image Perform weighted fusion to obtain a fused image ;

[0012] S2, generate image;

[0013] The fused image The input is into the U-net convolutional neural network structure, encoded through convolution operations, multi-scale features are extracted, and then the spatial resolution of the image is gradually restored through the decoder to output the restored image , Refers to clear images; in the stage of extracting multi-scale features, a residual block is inserted, and the SC attention mechanism is introduced based on the residual block;

[0014] S3, identification image;

[0015] The image will be restored The image is divided into multiple small blocks, and then the PatchGAN discriminator is used to determine whether each small block is part of the real image. During the network training process, the least squares GAN is used to calculate the adversarial loss between the generator and the discriminator to guide the generator to generate a more realistic dehazed image - that is, to obtain a preliminarily restored haze-free image. ;

[0016] S4, contrastive learning loss function;

[0017] S401, construct positive samples and negative samples;

[0018] From the input foggy image Randomly select N+1 image blocks from the N+1 image blocks. ; Restored image from output Find an image block in , image block With image blocks The relative position of the foggy image Randomly select N+1 image blocks from the image, removing the image block In addition, there are N image blocks left; then, the image blocks With image blocks Form a pair of positive samples; the image block Together with the remaining N image blocks, they form N groups of negative samples;

[0019] Use a noisy contrastive estimation module to maximize the mutual information between positive samples - that is, , The mutual information between the corresponding image blocks in is as follows:

[0020] The image will be restored The image blocks in As an anchor point, the anchor point, positive sample, and negative sample are mapped into K-dimensional vectors, which are represented as , and ;

[0021] b. Calculate positive samples Compared with negative samples The probability of being selected is expressed by the cross entropy loss

[0022]

[0023] in, express and The cosine similarity between is a temperature parameter used to adjust the distance between the anchor point and other samples, usually set to 0.07 by default; It refers to the nth negative sample;

[0024] S402, pixel-by-pixel comparison loss ;

[0025] Using encoder and two multilayer perceptron networks From the input foggy image Extract features and embed them into the feature stack ;in, represents the total number of layers selected from the encoder, Indicates the selected layer; Indicates that the image The sample selected from After being input into the network, the features obtained by the calculation of the encoding layer are obtained in sequence. After one layer, there will be a feature. Here, it means that the first layer; It means that it is known that the features of the corresponding number of layers are selected and calculated. , and then pass the corresponding layer features through the multi-layer perceptron network , and then the feature perception result is obtained after calculating each sample block; yes Simple expression of Represents a feature value obtained after a sample is calculated by the selected layer; these feature stacks represent different image blocks;

[0026] Sets the spatial position in each selected layer to ,in Represents the number of spatial positions in each layer;

[0027] Each layer selects an anchor point and represents its feature as ,in Represents the number of channels in each layer; Indicates that all samples have been A set formed after calculation, S represents the number of all samples;

[0028] The features of positive samples and negative samples are represented as , ;in, Represents a set of positive samples after calculation ; S\s represents the result of removing positive sample s from all samples S, that is, all negative samples; Indicates that all negative samples are composed of a set after calculation , because each layer of the feature set has one less positive sample, so the amount of data in the set becomes ;

[0029] The final loss is expressed as:

[0030]

[0031] in, Represents the mapping relationship between input and output image blocks, there are N+1 mapping relationships;

[0032] S403, pixel-by-pixel comparison loss ;

[0033] Get clean images of the real world by shooting , obtain the corresponding haze image of the real world by shooting , these images are randomly selected from real-world images and are not paired with each other;

[0034] Generate positive and negative sample pairs: Use clean images and the initial restored haze-free image As a positive sample pair, use the haze image and the initial restored haze-free image As a negative sample pair;

[0035] Loss calculation: By constructing a pixel-level self-contrast perception loss, we compare positive and negative sample pairs to enhance the model’s discrimination ability.

[0036] Feature extraction: The pre-trained VGG-16 network is introduced to extract feature maps of positive and negative sample pairs to further improve the image restoration effect;

[0037] Therefore, in the loss calculation, the image-by-image contrast loss is described as

[0038]

[0039] in, , = 1, 2, ... represents the i-th hidden feature extracted by the VGG-16 network pre-trained on ImageNet; select the 2nd, 3rd, and 5th maximum pooling layers; represents the weight coefficient, They are 0.4, 0.6 and 1 respectively; Represents a constant value; the formula The "1" in represents the L1 norm, also known as the Manhattan distance, which is the sum of the absolute values ​​of the elements in the vector. It is used to measure the distance between two vectors. The closer the distance between two vectors, the more similar the two vectors are. In addition, the same loss is also used. To keep the image structure after dehazing the same, that is , It represents the result of a clear and fog-free image after passing through the generative network G. The purpose of the loss is to control the performance of the generated network and ensure that it does not affect the normal image processing results. It is expected that the output results are basically consistent with the input results. It refers to the unclearFigure 1 A clear picture of training;

[0040] The overall loss function is expressed as

[0041]

[0042] in, They are 1, 1, 0.0002, 5 respectively; Represents the loss value of the judgment network, adv refers to the judgment network; Indicates the judgment of the generator's generation results.

[0043] In some embodiments, the S1 comprises:

[0044] S101: Foggy image The imaging model is expressed as ;in, represents a foggy image, represents the dark channel prior image, stands for transmittance - the proportion of light retained after passing through the fog, A is the atmospheric light;

[0045] Transmittance Estimation: For a haze-free image, define its dark channel as ,in, For a clean image, Represents the value of the image in the r, g, and b channels, c represents the three channels of r, g, and b, In pixels A local window centered on the dark channel; according to the dark channel prior, the local minimum is used to estimate the transmittance , ,in, It is an attenuation coefficient with a value of 0.95, which is used to retain a certain sense of fog;

[0046] For the calculation of the dark channel prior image: through the transmittance , atmospheric light A, can restore the fog-free image , ,in, is a constant that prevents the denominator from being zero;

[0047] S102: The original foggy image Restore haze-free image Perform weighted fusion to obtain a fused image ; ,in are learnable parameters.

[0048] In some embodiments, the S2 includes: performing a 4-fold downsampling operation to convert the input fused image Encode into a low-resolution image; use 9 residual blocks in the low-resolution space to extract more complex and deep features; then generate the final restored image through 4x sampling operation and 7X7 convolution layer .

[0049] In some embodiments, the use of 9 residual blocks in the low-resolution space to extract more complex and deep features means: at the end of the residual block, an SC attention mechanism is introduced, which is divided into a channel attention mechanism and a spatial attention mechanism;

[0050] Among them, the channel attention mechanism expression is as follows:

[0051]

[0052] in, Represented as the input feature map, is mean pooling, is the maximum pooling, The layer is a combination of two layers of neural networks; the input feature map is subjected to global maximum pooling and global mean pooling based on the width and height of the feature map respectively; then it is respectively subjected to layer, The features of the layer output are summed up; finally, The activation operation generates the final channel attention feature map;

[0053] Among them, the spatial attention mechanism expression is

[0054] in, The convolution kernel size is The convolution layer of the output feature map has a channel definition of 1. First, a global maximum pooling and a global mean pooling based on the channel are performed on the input feature map. Then, the two results are fused based on the channel. Then, a convolution operation is performed to reduce the dimension to 1 channel. Finally, Generate spatial attention feature map;

[0055] The SC attention mechanism generates attention feature map information in the channel and space dimensions respectively, and multiplies these two dimensions with the original input feature map to complete the adaptive feature correction, thereby obtaining the final feature map.

[0056] In some embodiments, the S3 includes: S301, image division;

[0057] Divide the restored image into multiple small image blocks; these small image blocks are of fixed size and cover the entire image for subsequent identification and generation;

[0058] S302, PatchGAN discriminator;

[0059] Use PatchGAN as a discriminator to independently judge each small block, that is, classify the small image blocks divided in S301 to determine whether they are from real images;

[0060] S303, adversarial training;

[0061] During network training, the adversarial loss between the generator and the discriminator is optimized through the least squares GAN loss function;

[0062] S304, generating a fog-free image;

[0063] Through continuous optimization of the above steps, the generator gradually learns to generate more realistic dehazed images; eventually, the image output by the generator is close to the real image in visual effect, achieving a preliminarily restored dehazed image. .

[0064] In some embodiments, in S3: the least squares GAN loss is used to train the network and a loss function is established. The loss function is expressed as:

[0065]

[0066]

[0067] in, Represents the input of the generator G The generated result is the restored haze-free image; Represents the image generated by the discriminator D to the generator The judgment result of The output of a clean image representing the real world is close to 1, It means that we hope that the restored haze-free image is as close to 1 as possible, that is, as realistic as possible. Represents minimizing the squared difference between the discriminator's prediction of the generated sample and the target value 1; Indicates that the least squares loss is used to penalize the squared difference between the discriminator prediction and the desired target; G represents the production network, where G is used as an input parameter; D represents the discriminant network, where D is used as an input parameter; Indicates that the discriminant network is used to classify the clear image If the judgment is true / 1, it indicates that it is a clear image; if the judgment is false / 0, it indicates that it is not a clear image.

[0068] In a second aspect, one of the embodiments of the present application provides a device for a 4K image defogging method based on weakly supervised contrast learning, comprising a processor, wherein the processor is used to execute the above-mentioned 4K image defogging method based on weakly supervised contrast learning.

[0069] In a third aspect, one of the embodiments of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes the above-mentioned 4K image dehazing method based on weakly supervised contrast learning.

[0070] The present invention has the following advantages:

[0071] 1. Performance advantages: The proposed scheme significantly improves the performance of image dehazing by introducing weakly supervised contrastive learning and self-contrast perception loss. This method does not rely on a large number of paired foggy and clear images, but uses unlabeled real-world images, avoiding the problem of data pairing difficulties in traditional methods. Through the contrastive learning framework and pixel-by-pixel contrastive loss, the network can be trained more effectively, improving the accuracy and robustness of the dehazing results.

[0072] 2. Functional advantages: The proposed scheme combines the technology of adversarial training to further enhance the generalization ability and robustness of the network; this enables the model to perform well in various complex real-world scenarios, including image defogging under various environmental factors such as changing lighting and changing weather conditions;

[0073] 3. Cost advantage: Compared with traditional methods that require a large number of manually labeled paired data sets, the proposed weakly supervised learning paradigm saves the cost and time of data annotation. This is especially important for large-scale data sets, which can reduce the cost of research and development, while speeding up the training and deployment of models and achieving ultra-high-definition image dehazing capabilities;

[0074] 4. Quality advantage: Through self-contrast perception loss, the distance between the restored image and the real clean image can be optimized in the representation space, thereby improving the quality of the dehazing result; this method effectively reduces artifacts and distortion in the image, producing a more natural and clear image output;

[0075] In summary, the innovation of the solution proposed in the present invention in the field of image dehazing lies not only in the improvement of technical methods, but also in how to effectively solve the challenges in existing methods through contrastive learning and self-contrast perceptual loss, thereby significantly improving performance, functionality, cost-effectiveness and image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a logical schematic diagram of the present invention;

[0077] Figure 2Processing logic graph for SC attention mechanism;

[0078] Figure 3 These are comparison pictures after defogging by PFF and the present invention, wherein the left picture is the picture after PFF processing, and the right picture is the picture after processing by the present invention. DETAILED DESCRIPTION

[0079] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0080] It should be noted that the directions or positional relationships indicated by "left", "right", etc. are based on the directions or positional relationships shown in the accompanying drawings, or are the directions or positional relationships in which the inventive product is usually placed when in use, or are the directions or positional relationships commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0081] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0082] See also Figure 1 , the 4K image defogging method based on weakly supervised contrast learning provided by the embodiment of the present invention includes:

[0083] S1, initial image dehazing processing;

[0084] The input foggy image After dark channel prior processing, it is corrected by the atmospheric scattering model to obtain the dark channel prior image. ; Then the dark channel prior image With foggy image Perform weighted fusion to obtain a fused image ;

[0085] S2, generate image;

[0086] The fused image The input is into the U-net convolutional neural network structure, encoded through convolution operations, multi-scale features are extracted, and then the spatial resolution of the image is gradually restored through the decoder to output the restored image ; In the stage of extracting multi-scale features, a residual block is inserted, and the SC attention mechanism is introduced based on the residual block;

[0087] S3, identification image;

[0088] The image will be restored The image is divided into multiple small blocks, and then the PatchGAN discriminator is used to determine whether each small block is part of the real image. During the network training process, the least squares GAN is used to calculate the adversarial loss between the generator and the discriminator to guide the generator to generate a more realistic dehazed image - that is, to obtain a preliminarily restored haze-free image. ;

[0089] S4. Contrastive learning loss function.

[0090] The image preprocessing of S1 is further described below.

[0091] S1. Image preparation processing.

[0092] S101, dark channel prior processing.

[0093] Dark channel prior is an image processing technique commonly used for defogging. It is based on the observation that in most fog-free images, the pixel values ​​of at least one color channel are very low in some local areas. With this prior, the thickness of fog can be estimated and the fog in the image can be removed.

[0094] A dark channel prior is applied to the input foggy image, the minimum value of each pixel is calculated, and then it is corrected through the atmospheric scattering model to obtain the dehazed image.

[0095] S102: weighted image fusion.

[0096] Weighted fusion is a linear combination of the original image and the dark channel prior image to achieve a balance between the defogging effect and detail retention. By adjusting the weighting coefficient, the image processed by the dark channel prior is fused with the original foggy image to form a fused image. This process can be achieved by weighted averaging, ensuring that the details in the original image are retained as much as possible while removing the fog.

[0097] For example, during the image preprocessing in S1, the specific means may be:

[0098] S101: Foggy image The imaging model is expressed as ;in, represents a foggy image, represents the dark channel prior image, stands for transmittance - the proportion of light that is retained after passing through the fog, A is the atmospheric light; Refers to a specific non-clear image in the foggy image;

[0099] Transmittance Estimation: For a haze-free image, define its dark channel as ,in, For a clean image, Represents the value of the image in the r, g, and b channels, c represents the three channels of r, g, and b, In pixels A local window centered on the dark channel; according to the dark channel prior, the local minimum is used to estimate the transmittance , ,in, It is an attenuation coefficient with a value of 0.95, which is used to retain a certain sense of fog;

[0100] For the calculation of the dark channel prior image: through the transmittance , atmospheric light A, can restore the fog-free image , ,in, is a constant that prevents the denominator from being zero;

[0101] S102: The original foggy image Restore haze-free image Perform weighted fusion to obtain a fused image ; ,in are learnable parameters.

[0102] It should be noted that in the S1 dark channel prior, the dark channel prior utilizes the statistical characteristics of the haze-free image, which can effectively remove haze, restore the image contrast, and make the image clearer. Fusion of the original haze image can help retain some details that may be lost in the dehazing process, especially when the dark channel prior may over-dehaze, it can better retain the color information of the original image and avoid color distortion caused by over-dehazing. Then the fusion strategy adopts a weighted fusion method to combine the dark channel prior results and the original foggy image Weighted fusion is performed. This fusion method can comprehensively utilize the advantages of the dehazing algorithm and the information of the original image, so that the final image achieves a balance between the dehazing effect and detail retention, thereby obtaining a higher quality dehazing image.

[0103] It should be noted that the dark channel prior is He K, Sun J, Tang X. Single image hazeremoval using dark channel prior[J]. IEEE transactions on pattern analysisand machine intelligence, 2010, 33(12): 2341-2353. The technology used is the existing image processing technology; this scheme only uses the existing technology of dark channel prior in this scheme.

[0104] The image generated by S2 is further described below.

[0105] S2. Generate an image.

[0106] S201, U-net structure.

[0107] U-net is a convolutional neural network structure that is widely used in medical image segmentation and image restoration tasks. It is characterized by symmetrical jump connections between the encoder and decoder, which can well preserve spatial resolution information. The fused image is input into the U-net network, encoded through a series of convolution operations, and multi-scale features are extracted. Subsequently, the spatial resolution of the image is gradually restored through the decoder, and finally a fog-free restored image is output.

[0108] S202: residual block.

[0109] The residual block is a network module containing skip connections, which can alleviate the gradient vanishing and degradation problems that may occur in deep networks. In the feature extraction stage of U-net, the residual block is inserted to ensure that the deep network can effectively learn more complex features, thereby improving the dehazing effect.

[0110] S203, SC attention mechanism.

[0111] The SC attention mechanism is a mechanism that combines spatial and channel attention, which can more effectively focus on important areas in the image and improve the accuracy of feature extraction. On the basis of the residual block, the SC attention mechanism is added to enhance the response of U-net to key features by weighting the spatial and channel dimensions, thereby further improving the quality of the dehazed image.

[0112] For example, see Figure 2 , in the generated image of S2, the specific means can be:

[0113] The general approach is to use the existing U-net structure as a generator to achieve a better trade-off between parameters and performance, and to input the fused image The specific method is: through a 4-fold downsampling operation, the input fused image is mapped to a fog-free restored image. Encoded into a low-resolution image; 9 residual blocks are used in the low-resolution space to extract more complex and deep features; then a 4x sampling operation and a 7X7 convolution are performed to generate the final restored image ,in, Refers to a specific clear image generated.

[0114] It should be noted that when using 9 residual blocks in the low-resolution space to extract more complex and deep features: the SC attention mechanism is introduced at the end of the residual block, which is divided into channel attention mechanism and spatial attention mechanism;

[0115] Among them, the channel attention mechanism expression is as follows:

[0116]

[0117] in, Represented as the input feature map, is mean pooling, is the maximum pooling, The layer is a combination of two layers of neural networks; the input feature map is subjected to global maximum pooling and global mean pooling based on the width and height of the feature map respectively; then it is respectively subjected to layer, The features of the layer output are summed up; finally, The activation operation generates the final channel attention feature map;

[0118] Among them, the spatial attention mechanism expression is

[0119] in, The convolution kernel size is The convolution layer of the output feature map has a channel definition of 1. First, a global maximum pooling and a global mean pooling based on the channel are performed on the input feature map. Then, the two results are fused based on the channel. Then, a convolution operation is performed to reduce the dimension to 1 channel. Finally, Generate spatial attention feature map;

[0120] SC attention generates attention feature map information in two dimensions, channel and space, respectively, and multiplies these two dimensions with the original input feature map to complete adaptive feature correction, thereby obtaining the final feature map.

[0121] That is, in S2 image generation, the essence is to use the existing U-net structure as a generator to achieve a better trade-off between parameters and performance, and to input the fused image Mapped to a haze-free restored image.

[0122] The S3 identification image is further described below.

[0123] S301, image segmentation;

[0124] Divide the restored image into multiple small image blocks; these small image blocks are of fixed size and cover the entire image for subsequent identification and generation;

[0125] S302, PatchGAN discriminator;

[0126] Use PatchGAN as a discriminator to independently judge each small block, that is, classify the small image blocks divided in S301 to determine whether they are from real images;

[0127] S303, adversarial training;

[0128] During the network training process, the adversarial loss between the generator and the discriminator is optimized by the least squares GAN loss function. It should be noted that the goal of the generator is to generate dehazed images that are as realistic as possible, so that the PatchGAN discriminator cannot distinguish whether these image patches are real or generated; the goal of the discriminator is to distinguish between real patches and generated patches as accurately as possible.

[0129] S304, generating a fog-free image;

[0130] Through continuous optimization of the above steps, the generator gradually learns to generate more realistic dehazed images; ultimately, the image output by the generator is closer to the real image in visual effect, achieving a preliminarily restored dehazed image. .

[0131] During the training process of the entire network, the least squares loss function is used to calculate the adversarial loss between the generator and the discriminator to guide the generator to generate more realistic dehazed images. The introduction of this loss function helps to improve the visual quality of the final image and reduce the generation of artifacts.

[0132] For example, when establishing a loss function in S3, the loss function is expressed as

[0133]

[0134]

[0135] in, Represents the input of the generator G The generated result is the restored haze-free image; Represents the image generated by the discriminator D to the generator The judgment result of The output of a clean image representing the real world is close to 1, It means that we hope that the restored haze-free image is as close to 1 as possible, that is, as realistic as possible. Represents minimizing the squared difference between the discriminator's prediction of the generated sample and the target value 1; Indicates that the least squares loss is used to penalize the squared difference between the discriminator prediction and the desired target; G represents the production network, where G is used as an input parameter; D represents the discriminant network, where D is used as an input parameter; Indicates that the discriminant network is used to classify the clear image If the judgment is true / 1, it indicates that it is a clear image, and if the judgment is false / 0, it indicates that it is not a clear image. The loss of the discriminator focuses on whether it can correctly identify the real image and whether it can correctly judge the generated image as false.

[0136] It should be noted that the use of PatchGAN has the following effects: PatchGAN can ensure the consistency of the image in the local area, thereby generating more detailed and realistic image details. Since the judgment of each small block is performed independently, the computational complexity of PatchGAN is relatively low, which is suitable for processing high-resolution images. PatchGAN effectively improves the quality and authenticity of the generated image by making judgments in the local area, allowing the generator to generate more realistic fog-free images.

[0137] The S4 contrast learning loss function is further described below.

[0138] Learn an embedding space that places positive samples close to each other while spreading the embeddings of negative samples far apart, and then use contrastive learning to train the network unsupervisedly.

[0139] The specific method is:

[0140] S401, construct positive and negative samples.

[0141] From the input foggy image Randomly select N+1 image blocks from the N+1 image blocks. ; Restored image from output Find an image block in , image block With image blocks The relative position of the foggy image Randomly select N+1 image blocks from the image, removing the image block In addition, there are N image blocks left; then, the image blocks With image blocks Form a pair of positive samples; the image block Together with the remaining N image blocks, N groups of negative samples are formed; for example, after inputting a foggy image, the image is flattened to form a 200X200 image, which is 40,000 positions. The index of each value can accurately find the corresponding position in the original image. For example, the 0 index corresponds to the first row and first column in the input image. Knowing this premise, randomly select N+1 positions from the 40,000 positions as sample blocks. In this way, the sample blocks of the input foggy image are obtained. The length and width of each sample block is 1×1. N is generally set to 255. Select 1 from the 255+1 sample blocks or image blocks. The image block is used as the data in the subsequent positive sample, and the other 255 are used as the data in the subsequent negative sample; then the defogging result is output, and the size of the output result is consistent with the input, and it also corresponds to the index of 40,000 positions. An image block is found in the output defogging result, and the position of the found image block is consistent with the image block position in the above "Select 1 image block as the data in the subsequent positive sample". In this way, the found image block and the corresponding position in the input image form a pair of positive samples, and the found image block and the remaining 255 non-corresponding positions in other input images form 255 pairs of negative samples;

[0142] Use a noisy contrastive estimation module to maximize the mutual information between positive samples - that is, , The mutual information between the corresponding image blocks in is as follows:

[0143] a. Restore the image The image blocks in As an anchor point, the anchor point, positive sample, and negative sample are mapped into K-dimensional vectors, which are represented as , and ;

[0144] b. Calculate positive samples Compared with negative samples The probability of being selected is expressed by the cross entropy loss

[0145]

[0146] in, express and The cosine similarity between is a temperature parameter used to adjust the distance between the anchor point and other samples, usually set to 0.07 by default; It refers to the nth negative sample.

[0147] S402, pixel-by-pixel comparison loss .

[0148] Using encoder and two multilayer perceptron networks From the input foggy image Extract features and embed them into the feature stack ;in, represents the total number of layers selected from the encoder, Indicates the selected layer; Indicates that the image The sample selected from After being input into the network, the features obtained by the calculation of the encoding layer are obtained in sequence. After one layer, there will be a feature. Here, it means that the first layer; It means that it is known that the features of the corresponding number of layers are selected and calculated. , and then pass the corresponding layer features through the multi-layer perceptron network , and then the feature perception result is obtained after calculating each sample block; yes Simple expression of Represents a feature value obtained after a sample is calculated by the selected layer; these feature stacks represent different image blocks;

[0149] Sets the spatial position in each selected layer to ,in Represents the number of spatial positions in each layer;

[0150] Each layer selects an anchor point and represents its feature as ,in Represents the number of channels in each layer; Indicates that all samples have been A set formed after calculation, S represents the number of all samples;

[0151] The features of positive samples and negative samples are represented as , ;in, Represents a set of positive samples after calculation ; S\s represents the result of removing positive sample s from all samples S, that is, all negative samples; Indicates that all negative samples are composed of a set after calculation , because each layer of the feature set has one less positive sample, so the amount of data in the set becomes ;

[0152] The final loss is expressed as:

[0153]

[0154] It should be noted that the purpose of this loss is to make the positive samples of the input image match the corresponding positions in the output results as much as possible, so It refers to the mapping relationship between the output and the sample blocks in the output image, and there are N+1 relationships.

[0155] S403, pixel-by-pixel comparison loss .

[0156] Get clean images of the real world by shooting , obtain the corresponding haze image of the real world by shooting , these images are randomly selected from real-world images and are not paired with each other;

[0157] Generate positive and negative sample pairs: Use clean images and the initial restored haze-free image As a positive sample pair, use the haze image and the initial restored haze-free image As a negative sample pair;

[0158] Loss calculation: By constructing a pixel-level self-contrast perception loss, we compare positive and negative sample pairs to enhance the model’s discrimination ability.

[0159] Feature extraction: The pre-trained VGG-16 network is introduced to extract feature maps of positive and negative sample pairs to further improve the image restoration effect;

[0160] Therefore, in the loss calculation, the image-by-image contrast loss is described as

[0161]

[0162] in, , = 1, 2, ... represents the i-th hidden feature extracted by the VGG-16 network pre-trained on ImageNet; select the 2nd, 3rd, and 5th maximum pooling layers; represents the weight coefficient, They are 0.4, 0.6 and 1 respectively; Represents a constant value; the formula The "1" in represents the L1 norm, also known as the Manhattan distance, which is the sum of the absolute values ​​of the elements in the vector. It is used to measure the distance between two vectors. The closer the distance between two vectors, the more similar the two vectors are. In addition, the same loss is also used. To keep the image structure after dehazing the same, that is , It represents the result of a clear and fog-free image after passing through the generative network G. The purpose of the loss is to control the performance of the generated network and ensure that it does not affect the normal image processing results. It is expected that the output results are basically consistent with the input results. It refers to the unclear Figure 1 A clear picture of training;

[0163] The overall loss function is expressed as

[0164]

[0165] in, They are 1, 1, 0.0002, 5 respectively; Represents the loss value of the judgment network, adv refers to the judgment network; Indicates the judgment of the generator's generation results.

[0166] Test Trial:

[0167] This scheme is compared with the PSD, PFF, and GCA image dehazing methods to verify the dehazing effect; the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) commonly used in the field of image enhancement are used as evaluation indicators. The larger the PSNR, the higher the image quality, and the higher the SSIM, the more similar the two images are; the models corresponding to the above comparison methods are run on the 4KID, O-HAZE, and I-HAZE ultra-high-definition datasets respectively;

[0168] The experimental results are shown in Table 1. Figure 3 As shown; Table 1 is a comparative test table of Ours scheme and general PSD, PFF, and GCD image dehazing; Figure 3 This is a comparison of the defogging effects of PFF and this solution, with PFF on the left and this solution on the right;

[0169] It can be seen from Table 1 that the total time of PSD, PFF, GCA, and Ours in the dehazing process is 20s, 38s, 95s, and 14s respectively; the processing time of Ours in this scheme is shorter than that of other methods; therefore, this scheme is easier to train than other supervised methods; and it can greatly improve the algorithm operation efficiency while maintaining high image quality;

[0170] It can be concluded that the defogging effect of this scheme is very good, which is better than SD, PFF and GCA methods.

[0171] Table 1 Comparative test table of Ours scheme and general PSD, PFF, GCD image dehazing

[0172]

[0173] It should be noted that this solution has the following advantages through dark channel prior, N-net structure processing of the generator, introduction of SC attention mechanism, use of PatchGAN as the discriminator, and finally learning loss function through comparison:

[0174] (1) Solved the domain transfer problem;

[0175] Traditional image dehazing methods are usually trained on synthetic datasets, which results in insufficient generalization ability of the models in the real world;

[0176] In this scheme, the proposed method adopts a weakly supervised contrastive learning method, which uses unpaired real-world blurry and clean images to avoid the domain shift problem between synthetic data and real data; in this way, the model can better adapt to diverse conditions such as lighting and weather changes in the real world;

[0177] (2) The difficulties of data matching are overcome;

[0178] Existing image dehazing methods usually rely on a large number of paired foggy and clean images for supervised learning, which is difficult to achieve in practice, i.e., data pairing is difficult;

[0179] In this scheme, the proposed method introduces a new weakly supervised contrastive learning paradigm, in which foggy images are regarded as positive samples and clean images as negative samples; this contrastive learning method not only reduces the dependence on paired data, but also improves the data utilization efficiency of the model by effectively utilizing the unlabeled real-world image collection;

[0180] (3) Optimization design of loss function;

[0181] In this scheme, the proposed method designs a new self-contrast-aware loss function; this loss function helps the network better learn to distinguish the foggy phenomenon in the real world by adjusting the restored image in the embedding space to make it close to the positive sample - that is, the clean image of the real world, and away from the negative sample - that is, other possible foggy images.

[0182] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. 4K image dehazing method based on weakly supervised contrastive learning, Features: The following steps are involved: S1, image preparation processing; The input foggy image After dark channel prior processing, it is corrected by the atmospheric scattering model to obtain the dark channel prior image. ;in, It refers to a non-sharp image; The dark channel prior image With foggy image Perform weighted fusion to obtain a fused image ; S2, generate image; The fused image The input is into the U-net convolutional neural network structure, encoded through convolution operations, multi-scale features are extracted, and then the spatial resolution of the image is gradually restored through the decoder to output the restored image , Refers to clear images; in the stage of extracting multi-scale features, a residual block is inserted, and the SC attention mechanism is introduced based on the residual block; S3, identification image; The image will be restored The image is divided into multiple small blocks, and then the PatchGAN discriminator is used to determine whether each small block is part of the real image. During the network training process, the least squares GAN is used to calculate the adversarial loss between the generator and the discriminator to guide the generator to generate a more realistic dehazed image - that is, to obtain a preliminarily restored haze-free image. ; S4, contrastive learning loss function; S401, construct positive samples and negative samples; From the input foggy image Randomly select N+1 image blocks from the N+1 image blocks. ; Restored image from output Find an image block in , image block With image blocks The relative position of foggy image Randomly select N+1 image blocks from the image, removing the image block In addition, there are N image blocks left; then, the image blocks With image blocks Form a pair of positive samples; the image block Together with the remaining N image blocks, they form N groups of negative samples; Use a noisy contrastive estimation module to maximize the mutual information between positive samples - that is, , The mutual information between the corresponding image blocks in is as follows: b. Calculate positive samples Compared with negative samples The probability of being selected is expressed by the cross entropy loss in, express and The cosine similarity between is a temperature parameter used to adjust the distance between the anchor point and other samples, usually set to 0.07 by default; It refers to the nth negative sample; S402, pixel-by-pixel comparison loss ; Using encoder and two multilayer perceptron networks From the input foggy image Extract features and embed them into the feature stack ;in, represents the total number of layers selected from the encoder, Indicates the selected layer; Indicates that the image The sample selected from After being input into the network, the features obtained by the calculation of the encoding layer are obtained in sequence. After one layer, there will be a feature. Here, it means that the first layer; It means that it is known that the features of the corresponding number of layers are selected and calculated. , and then pass the corresponding layer features through the multi-layer perceptron network , and then the feature perception result is obtained after calculating each sample block; yes Simple expression of Represents a feature value obtained after a sample is calculated by the selected layer; these feature stacks represent different image blocks; Sets the spatial position in each selected layer to ,in Represents the number of spatial positions in each layer; Each layer selects an anchor point and represents its feature as ,in Represents the number of channels in each layer; Indicates that all samples have been A set formed after calculation, S represents the number of all samples; The features of positive samples and negative samples are represented as , ;in, Represents a set of positive samples after calculation ; S\s represents the result of removing positive sample s from all samples S, that is, all negative samples; Indicates that all negative samples are composed of a set after calculation , because each layer of the feature set has one less positive sample, so the amount of data in the set becomes ; The final loss is expressed as: in, Represents the mapping relationship between input and output image blocks, there are N+1 mapping relationships; S403, pixel-by-pixel comparison loss ; Get clean images of the real world by capturing , obtain the corresponding haze image of the real world by shooting , these images are randomly selected from real-world images and are not paired with each other; Generate positive and negative sample pairs: Take a clean image and the initial restored haze-free image As a positive sample pair, use the haze image and the initial restored haze-free image As a negative sample pair; Loss calculation: By constructing a pixel-level self-contrast perception loss, we compare positive and negative sample pairs to enhance the model’s discrimination ability. Feature extraction: The pre-trained VGG-16 network is introduced to extract feature maps of positive and negative sample pairs to further improve the image restoration effect; Therefore, in the loss calculation, the image-by-image contrast loss is described as in, , = 1, 2, ... represents the i-th hidden feature extracted by the VGG-16 network pre-trained on ImageNet; select the 2nd, 3rd, and 5th maximum pooling layers; represents the weight coefficient, They are 0.4, 0.6 and 1 respectively; Represents a constant value; the formula The "1" in represents the L1 norm, also known as the Manhattan distance, which is the sum of the absolute values ​​of the elements in the vector. It is used to measure the distance between two vectors. The closer the distance between two vectors, the more similar the two vectors are. In addition, the same loss is also used. To keep the image structure after dehazing the same, that is , It represents the result of a clear and fog-free image after passing through the generative network G. The purpose of the loss is to control the performance of the generated network and ensure that it does not affect the normal image processing results. It is expected that the output results are basically consistent with the input results. refers to the legible graphs that are trained together with the non-legible graphs; The overall loss function is expressed as in, They are 1, 1, 0.0002, 5 respectively; Represents the loss value of the judgment network, adv refers to the judgment network; Indicates the judgment of the generator's generation results.

2. The 4K image defogging method based on weakly supervised contrastive learning according to claim 1, Features: The S1 includes: S101: Foggy image The imaging model is expressed as ;in, represents a foggy image, represents the dark channel prior image, stands for transmittance - the proportion of light retained after passing through the fog, A is the atmospheric light; Transmittance Estimation: For a haze-free image, define its dark channel as ,in, For a clean image, Represents the value of the image in the r, g, and b channels, c represents the three channels of r, g, and b, In pixels A local window centered on the dark channel; according to the dark channel prior, the local minimum is used to estimate the transmittance , ,in, It is an attenuation coefficient with a value of 0.95, which is used to retain a certain sense of fog; For the calculation of the dark channel prior image: through the transmittance , atmospheric light A, can restore the fog-free image , ,in, is a constant that prevents the denominator from being zero; S102: The original foggy image Restore haze-free image Perform weighted fusion to obtain a fused image ; ,in are learnable parameters.

3. The 4K image defogging method based on weakly supervised contrastive learning according to claim 2, Features: The S2 includes: Through 4 times downsampling operation, the input fused image Encoded as low-resolution images; Use 9 residual blocks in the low-resolution space to extract more complex and deep features; Then, through a 4x sampling operation and a 7X7 convolution layer, the final restored image is generated. .

4. The 4K image defogging method based on weakly supervised contrastive learning according to claim 3, Features: The method of using 9 residual blocks in the low-resolution space to extract more complex and deep features means that at the end of the residual block, an SC attention mechanism is introduced, which is divided into a channel attention mechanism and a spatial attention mechanism; Among them, the channel attention mechanism expression is ,in, Represented as the input feature map, is mean pooling, is the maximum pooling, The layer is a combination of two layers of neural networks; the input feature map is subjected to global maximum pooling and global mean pooling based on the width and height of the feature map respectively; then it is respectively subjected to layer, The features of the layer output are summed up; finally, The activation operation generates the final channel attention feature map; Among them, the spatial attention mechanism expression is ,in, The convolution kernel size is The convolution layer of the output feature map has a channel definition of 1. First, a global maximum pooling and a global mean pooling based on the channel are performed on the input feature map. Then, the two results are fused based on the channel. Then, a convolution operation is performed to reduce the dimension to 1 channel. Finally, Generate spatial attention feature map; The SC attention mechanism generates attention feature map information in the channel and space dimensions respectively, and multiplies these two dimensions with the original input feature map to complete the adaptive feature correction, thereby obtaining the final feature map.

5. The 4K image defogging method based on weakly supervised contrastive learning according to claim 3, Features: The S3 includes: S301, image division; Divide the restored image into multiple small image blocks; these small image blocks are of fixed size and cover the entire image for subsequent identification and generation; S302, PatchGAN discriminator; Use PatchGAN as a discriminator to independently judge each small block, that is, classify the small image blocks divided in S301 to determine whether they are from real images; S303, adversarial training; During network training, the adversarial loss between the generator and the discriminator is optimized through the least squares GAN loss function; S304, generating a fog-free image; Through continuous optimization of the above steps, the generator gradually learns to generate more realistic dehazed images; eventually, the image output by the generator is close to the real image in visual effect, achieving a preliminarily restored dehazed image. .

6. The 4K image defogging method based on weakly supervised contrastive learning according to claim 5, Features: In S3, the least squares GAN loss is used to train the network and establish a loss function, which is expressed as: in, Represents the input of the generator G The generated result is the restored haze-free image; Represents the image generated by the discriminator D to the generator The judgment result of The output of a clean image representing the real world is close to 1, It means that we hope that the restored haze-free image is as close to 1 as possible, that is, as realistic as possible. Represents minimizing the squared difference between the discriminator's prediction of the generated sample and the target value 1; Indicates that the least squares loss is used to penalize the squared difference between the discriminator prediction and the desired target; G represents the production network, where G is used as an input parameter; D represents the discriminant network, where D is used as an input parameter; Indicates that the discriminant network is used to classify the clear image If the judgment is true / 1, it indicates that it is a clear image; if the judgment is false / 0, it indicates that it is not a clear image.

7. A device for a 4K image defogging method based on weakly supervised contrast learning, comprising a processor, Features: The processor is used to execute the 4K image defogging method based on weakly supervised contrast learning as described in any one of claims 1 to 6.

8. A computer readable storage medium, Features: The storage medium stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes the 4K image defogging method based on weakly supervised contrast learning as described in any one of claims 1 to 6.

Images