Single image defogging method, device, equipment and medium based on style restoration and detail supplementation

Abstract

The present invention provides a single image defogging method, apparatus, device, and medium based on style restoration and detail supplementation, relating to the field of computer vision technology. The method comprises: inputting a sample foggy image into a to-be-trained image defogging network; processing the sample foggy image using a pre-trained style restoration module to obtain a sample style-restored image; processing the sample foggy image using a to-be-trained detail supplementation module to obtain detail supplementation image features; inputting the sample style-restored image and detail supplementation image features into a to-be-trained cross-fusion module to obtain a sample defogging image; updating the model parameters of the to-be-trained detail supplementation module and the to-be-trained cross-fusion module based on the sample defogging image and the sample clean image until a trained image defogging network is obtained; and inputting the to-be-defogging image into the trained image defogging network to obtain a defogged image, thereby improving the image defogging effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a single image defogging method, device, equipment and medium based on style restoration and detail supplementation. Background Art

[0002] Computer vision technology is rapidly expanding its influence across various industries, becoming a key enabler for cutting-edge fields such as smart cities, autonomous driving, and intelligent safety systems. However, image quality largely determines the success of these applications. Factors such as inclement weather (fog, rain, and snow), mechanical issues (camera shake), physical phenomena (thermal noise and shadows), insufficient lighting, or image data loss can significantly affect image clarity and integrity, limiting the effectiveness of subsequent analysis tasks. Among the many image corruption issues, fog is the most common. Effectively restoring this type of damaged image can significantly improve the performance of subsequent vision tasks.

[0003] Single image dehazing, a key low-level vision task, plays a crucial role in practical applications such as object detection and autonomous driving. This technology aims to remove the effects of haze from a single input image through algorithms, restoring the scene's true detail and clarity, thereby providing a more reliable foundation for higher-level visual analysis. In object detection, haze reduces image contrast and introduces noise, blurring object outlines and thus impairing the performance of detection algorithms. Effective dehazing can significantly improve image quality, making object boundaries more distinct. This not only enhances detection accuracy but also enhances classification accuracy. Clear images facilitate feature extraction. For tasks that rely on texture or shape information for classification, such as pedestrian and vehicle recognition, accurate image restoration leads to better classification results. Single image dehazing has important and widespread applications. Autonomous driving systems rely on accurate environmental perception, and hazy weather severely impacts the information captured by cameras. Effective dehazing algorithms can help vehicles better understand their surroundings, avoid potential hazards, and ensure safe driving. In addition, image dehazing can enhance the data quality of visual sensors, making data fusion with other sensors such as LiDAR and millimeter-wave radar more effective, thereby improving the robustness and reliability of the entire system.

[0004] Currently, single-image dehazing methods typically directly learn the overall difference between foggy and clear images. This means they learn directly from a very difficult task, resulting in a high learning difficulty, which in turn impairs the image dehazing effect to a certain extent. This leads to poor image dehazing results when dealing with complex and changing real-world scenes. Therefore, how to improve image dehazing performance is a technical problem that needs to be solved urgently in this invention. Summary of the Invention

[0005] Based on the above technical problems, the present invention provides a single image defogging method, device, equipment and medium based on style restoration and detail supplementation, aiming to improve the effect and performance of image defogging.

[0006] A first aspect of the present invention provides a single image defogging method based on style restoration and detail supplementation, the method comprising:

[0007] Inputting a sample foggy image into a to-be-trained image defogging network, the to-be-trained image defogging network comprising at least: a pre-trained style restoration module, a to-be-trained detail supplementation module, and a to-be-trained cross-fusion module; the detail supplementation module comprises multi-directional and multi-dimensional convolutions for extracting and supplementing image details; the style restoration module is implemented based on a large pre-trained visual model for restoring the image style; and the cross-fusion module is configured to repeatedly aggregate the output information of the detail supplementation module and the style restoration module;

[0008] Processing the sample foggy image using the pre-trained style restoration module to obtain a sample style restored image;

[0009] Processing the sample foggy image by the detail supplementation module to be trained to obtain detail supplementation image features;

[0010] Inputting the sample style restoration image and the detail supplement image features into the cross fusion module to be trained to obtain a sample defogging image;

[0011] Based on the sample defogging image and the sample clean image corresponding to the sample foggy image, updating the model parameters of the detail supplement module to be trained and the cross fusion module to be trained until a trained image defogging network is obtained;

[0012] The image to be defogged is input into the trained image defogging network to obtain a defogged image output by the trained image defogging network.

[0013] A second aspect of the present invention provides a single image defogging device based on style restoration and detail supplementation, the device comprising:

[0014] An image input module is configured to input a sample foggy image into an image defogging network to be trained. The image defogging network to be trained includes at least: a pre-trained style restoration module, a detail supplementation module to be trained, and a cross-fusion module to be trained; the detail supplementation module is composed of multi-directional and multi-dimensional convolutions to extract and supplement image details; the style restoration module is implemented based on a large pre-trained visual model to restore the image style; and the cross-fusion module is configured to repeatedly aggregate the output information of the detail supplementation module and the style restoration module.

[0015] a style processing module, configured to process the sample foggy image using the pre-trained style restoration module to obtain a sample style restored image;

[0016] a detail processing module, configured to process the sample foggy image through the detail supplementation module to be trained to obtain detail supplementation image features;

[0017] a feature fusion module, configured to input the features of the sample style restored image and the detail supplemented image into the cross fusion module to be trained, to obtain a sample defogging image;

[0018] a model training module, configured to update model parameters of the detail supplement module to be trained and the cross-fusion module to be trained based on the sample defogging image and the sample clean image corresponding to the sample foggy image, until a trained image defogging network is obtained;

[0019] The image defogging module is used to input the image to be defogged into the trained image defogging network to obtain the defogged image output by the trained image defogging network.

[0020] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the single image defogging method based on style restoration and detail supplementation according to the first aspect of the embodiment of the present invention is implemented.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the single image defogging method based on style restoration and detail supplementation according to the first aspect of the embodiment of the present invention is implemented.

[0022] In the single-image dehazing method based on style restoration and detail supplementation provided by the present invention, style restoration and detail supplementation are performed on a sample foggy image using a style restoration module and a detail supplementation module, respectively, to obtain sample style-restored image and detail supplementation image features. The sample style-restored image and detail supplementation image features are then combined using a cross-fusion module. By repeatedly aggregating high-level information, a clear dehazed image is ultimately obtained. Thus, the present invention proposes an image dehazing network with a style restoration module and a detail supplementation module as two sub-networks. In this image dehazing network, the image dehazing task is divided into two sub-tasks, "style restoration" and "detail supplementation," to perform style restoration and detail supplementation on the foggy image, respectively. This reduces the learning difficulty of the image dehazing network, effectively achieves image dehazing, and improves the effect and performance of image dehazing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 1 is a schematic diagram illustrating the style and content differences between foggy and clear image pairs in different scenes according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a single image defogging method based on style restoration and detail supplementation according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of implementation details of a detail supplement module according to an embodiment of the present invention;

[0027] Figure 4 1 is a schematic diagram of the overall structure of an image defogging network according to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram comparing three training methods according to an embodiment of the present invention;

[0029] Figure 6 1 is a diagram showing the comparison of training results of SRMs trained in three different ways on SOTS-indoor in the RESIDE dataset, according to an embodiment of the present invention;

[0030] Figure 7 is a flowchart of training and inference of an image dehazing network according to an embodiment of the present invention;

[0031] Figure 8is a flowchart of training and reasoning of an image dehazing network according to another embodiment of the present invention;

[0032] Figure 9 This is a visual comparison diagram of a single image dehazing method based on style restoration and detail supplementation and other mainstream methods shown in an embodiment of the present invention;

[0033] Figure 10 This is another visual comparison diagram of a single image dehazing method based on style restoration and detail supplementation and other mainstream methods shown in an embodiment of the present invention;

[0034] Figure 11 This is a structural block diagram of a single image defogging device based on style restoration and detail supplementation provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] In view of the fact that current deep learning-based image dehazing methods usually directly learn the overall difference between foggy images and clear image pairs, which means learning directly from a very difficult task, resulting in a technical defect that damages the image dehazing effect, the present invention finds that the degradation of foggy images usually involves two aspects: style transformation and hiding of details. To illustrate this point, the present invention evaluates the style and content differences between image pairs consisting of foggy images and clear images in two different scenarios. The style difference can be represented by the Gram matrix, which is an indicator in the style transfer task. If the mean square distance of the Gram matrix between the image pairs is smaller, it can be considered that the styles of the two images are more similar. The Gram matrix is ​​calculated as follows:

[0037] ;

[0038] Among them, H, W and C represent the dimensions of the image H×W×C, represents the kth element of the i-th channel, Represents the element in the i-th row and j-th column of the Gram matrix. In order to calculate the content difference between two images, the present invention uses Contextual Loss ( ) function to calculate. The specific calculation process is as follows, that is, the content and detail differences between pairs can be calculated using context loss ( ) to indicate:

[0039] CX(X, Y) is used to calculate the number of features in Y that are nearest neighbors to a feature in X. X and Y are the feature values ​​obtained after passing through a feature extraction network Φ. Φ in this invention uses a Swin network. The above definitions of style and content difference metrics can be used to calculate the specific changes in style and content of an image after being affected by fog.

[0040] like Figure 1 As shown, Figure 1 : is a schematic diagram showing the style and content differences between foggy and clear image pairs in different scenes according to an embodiment of the present invention. Figure 1 As can be seen from the figure, when both images in the same image pair are clear or foggy, the Gram matrix is ​​smaller, even though the details shown in these images are completely different ( is relatively large). When a foggy image and a clear image of the same scene are paired, the Gram matrix is ​​larger, while This indicates that: 1. The style difference between foggy and clear images is very significant, and style recovery can play a key role in single-task image dehazing; 2. There are also content differences between foggy and clear images, and detail supplementation can be an important auxiliary means to improve dehazing performance.

[0041] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present invention proposes an image defogging network with style restoration and detail supplementation as two sub-networks. In this image defogging network, the image defogging task is divided into two sub-tasks of "style restoration" and "detail supplementation" to perform style restoration and detail supplementation on foggy images respectively, thereby reducing the learning difficulty of the image defogging network, effectively realizing image defogging, and improving the effect and performance of image defogging.

[0042] Hereinafter, specific examples of this solution will be described in more detail with reference to the accompanying drawings.

[0043] Please refer to Figure 2 , Figure 2 FIG. 1 is a flowchart showing a single image defogging method based on style restoration and detail supplementation according to an embodiment of the present invention. Figure 2 As shown, the single image defogging method based on style restoration and detail supplementation provided in this embodiment includes at least the following steps:

[0044] Step S11: inputting the sample foggy image into the image defogging network to be trained, wherein the image defogging network to be trained comprises at least: a pre-trained style restoration module, a detail supplementation module to be trained, and a cross-fusion module to be trained.

[0045] In this embodiment, the image dehazing network to be trained includes at least: a pre-trained style restoration module, a pre-trained detail supplementation module, and a pre-trained cross-fusion module. The detail supplementation module, composed of multi-directional and multi-dimensional convolutions, is used to extract and supplement image details. The style restoration module is implemented based on a large pre-trained visual model and is used to restore the image style. The cross-fusion module is used to repeatedly aggregate the output information of the detail supplementation module and the style restoration module.

[0046] This embodiment prepares a sample data set for training the image defogging network, and the sample data set includes: a sample foggy image and a sample clean image corresponding to the sample foggy image; the sample clean image, that is, the label corresponding to the sample foggy image, is a clear image after defogging corresponding to the sample foggy image; the sample foggy image is a foggy image sample used for training the image defogging network.

[0047] In this embodiment, during the training process of the image defogging network to be trained, a sample foggy image is first input into the image defogging network to be trained.

[0048] Step S12: Processing the sample foggy image through the pre-trained style restoration module to obtain a sample style restored image.

[0049] In this embodiment, after a sample foggy image is input into the image dehazing network to be trained, it is then fed into a pre-trained style restoration module and a detail supplementation module to be trained. The sample foggy image is processed in parallel by these modules. For example, when a sample foggy image enters the image dehazing network to be trained, it is copied into two copies: one copy is fed into the pre-trained style restoration module, and the other copy is fed into the detail supplementation module to be trained.

[0050] In this embodiment, after a sample foggy image is input into a pretrained style restoration module, the pretrained style restoration module can perform style restoration processing on the sample foggy image, resulting in a sample style-restored image output by the style restoration module. This sample style-restored image is the image after style restoration of the sample foggy image. The pretrained style restoration module in this embodiment is a pretrained style restoration module. During the training of the image dehazing network to be trained, the network parameters of the pretrained style restoration module are fixed and are not updated.

[0051] Step S13: Processing the sample foggy image by the detail supplementation module to be trained to obtain detail supplementation image features.

[0052] In this embodiment, after the sample foggy image is input into the detail supplement module to be trained, the sample foggy image can be processed by the detail supplement module to be trained to obtain the detail supplement image features output by the detail supplement module. The detail supplement image features are image features after the sample foggy image is extracted and supplemented with details in multiple directions and dimensions.

[0053] Step S14: inputting the sample style restored image and the detail supplemented image features into the cross-fusion module to be trained to obtain a sample defogging image.

[0054] In this embodiment, after obtaining the features of the sample style-restored image and the detail-supplemented image, the features of the sample style-restored image and the detail-supplemented image can be input into a cross-fusion module to be trained. This cross-fusion module then fuses the features of the sample style-restored image and the detail-supplemented image to obtain a sample dehazed image as output by the cross-fusion module. For example, the cross-fusion module to be trained can combine the features of the sample style-restored image and the detail-supplemented image to generate a clear image (i.e., the sample dehazed image) by repeatedly aggregating high-level information. The sample dehazed image is the clean image obtained by dehazing the sample hazy image during the training process of the image dehazing network to be trained.

[0055] Step S15: Based on the sample defogging image and the sample clean image corresponding to the sample foggy image, the model parameters of the detail supplement module to be trained and the cross fusion module to be trained are updated until a trained image defogging network is obtained.

[0056] In this embodiment, the model parameters of the detail supplement module to be trained and the cross-fusion module to be trained can be updated based on the sample defogging image corresponding to the sample foggy image and the sample clean image corresponding to the sample foggy image until a trained image defogging network is obtained.

[0057] Step S16: inputting the image to be defogged into the trained image defogging network to obtain a defogged image output by the trained image defogging network.

[0058] In this embodiment, the trained image dehazing network is used to perform dehazing on an image. During the application of the trained image dehazing network, the image to be dehazed can be input into the trained image dehazing model to obtain a dehazed image output by the trained image dehazing model. The image to be dehazed is the image to be dehazed, and the dehazed image is the image after the dehazing process has been performed on the image to be dehazed.

[0059] In this embodiment, the style restoration module and detail supplementation module are used to perform style restoration and detail supplementation on a sample foggy image, respectively, to obtain features of the sample style-restored image and detail supplementation image. The features of the sample style-restored image and detail supplementation image are then combined by a cross-fusion module. By repeatedly aggregating high-level information, a clear defogged image is ultimately obtained. Thus, the present invention proposes an image defogging network with a style restoration module and a detail supplementation module as two sub-networks. In this image defogging network, the image defogging task is divided into two sub-tasks, "style restoration" and "detail supplementation," to perform style restoration and detail supplementation on a foggy image, respectively. This reduces the learning difficulty of the image defogging network, effectively achieves image defogging, and improves the effect and performance of image defogging.

[0060] Currently, in the field of single image dehazing tasks, simple convolution is often used for feature extraction, which limits the network's feature expression and learning capabilities, thereby limiting its ability to effectively express and model complex information. To address this issue, in combination with the above embodiments, in one embodiment, the present invention further provides a single image dehazing method based on style restoration and detail supplementation. In this method, the above step S13 may specifically include steps S21 to S23:

[0061] Step S21: inputting the sample foggy image into the first convolution, the second convolution, the third convolution and the fourth convolution respectively for processing to obtain a first image feature, a second image feature, a third image feature and a fourth image feature.

[0062] In this embodiment, the goal of the detail supplementation module is to focus on extracting and supplementing details from all directions. To achieve this, the detail supplementation module of this embodiment does not have any image generation function to ensure the effective extraction of feature values ​​in all directions and dimensions, thereby facilitating the supplementation of details in different directions and sizes.

[0063] In this embodiment, the detail supplement module uses a variety of convolutions of different sizes and directions. The detail supplement module includes at least: a first convolution, a second convolution, a third convolution, and a fourth convolution. The first convolution, the second convolution, the third convolution, and the fourth convolution are responsible for detail supplementation in four dimensions: point level, horizontal level, vertical level, and plane level, respectively. Among them, the first convolution can be Pixel-Wise Convolution (P-WC), the second convolution can be Horizontal-Wise Convolution (H-WC), the third convolution can be Vertical-Wise Convolution (V-WC), and the fourth convolution can be Plane-Wise Convolution (PL-WC).

[0064] In the detail supplement module, the sample foggy image is input into the first convolution, second convolution, third convolution and fourth convolution for processing respectively, and the first image feature of the first convolution output, the second image feature of the second convolution output, the third image feature of the third convolution output and the fourth image feature of the fourth convolution output are obtained respectively.

[0065] Step S22: Connect the sample foggy image, the first image feature, the second image feature, the third image feature, and the fourth image feature to obtain a fifth image feature.

[0066] In this embodiment, after obtaining the first image feature, the second image feature, the third image feature and the fourth image feature, the sample foggy image, the first image feature, the second image feature, the third image feature and the fourth image feature are connected to obtain a comprehensive feature value, that is, the fifth image feature.

[0067] Step S23: adjusting the spatial resolution of the fifth image feature to obtain the detail supplementary image feature.

[0068] In this embodiment, after obtaining the fifth image feature, it is necessary to adjust the spatial resolution of the fifth image feature to obtain the detail supplement image feature output by the detail supplement module. In a specific example, this embodiment adjusts the spatial resolution of the fifth image feature through a PixelUnshuffle operation. PixelUnshuffle and PixelShuffle are inverse operations of each other. PixelUnshuffle is an effective downsampling method that is crucial for converting low-dimensional, large-sized tensors into high-dimensional, small-sized tensors, while PixelShuffle performs the opposite conversion.

[0069] In addition, in a specific example, the implementation details of various convolutions in the detail supplementation module are shown in Table 1, where padding is used to keep the output of the convolution and the output size the same.

[0070] Table 1 Detailed parameters of each convolution in the detail supplement module

[0071]

[0072] In one embodiment, if Figure 3 As shown, Figure 3 FIG. 1 is a schematic diagram showing implementation details of a detail supplement module according to an embodiment of the present invention. Figure 3In the Detail Replenishment Module (DRM), various convolutions of varying sizes and orientations are used. These convolutions are of four types: P-WC, H-WC, V-WC, and PL-WC. These convolutions are responsible for detail replenishment in four dimensions: point-level, horizontal-level, vertical-level, and plane-level. In the DRM, the features extracted from the input image (e.g., the sample foggy image) by P-WC, H-WC, V-WC, and PL-WC are concatenated with the input image to form a composite feature value. The spatial resolution is then adjusted through a PixelUnshuffle operation to obtain the detail replenishment image features corresponding to the input image.

[0073] In combination with the above embodiments, in one implementation, the present invention further provides a single image defogging method based on style restoration and detail supplementation. In this method, the above step S14 may specifically include steps S31 to S34:

[0074] Step S31: in the cross-fusion module to be trained, performing an upsampling operation and a first spatial resolution adjustment operation on the sample style restoration image to obtain a sixth image feature.

[0075] In this embodiment, in the cross-fusion module to be trained, the sample style restoration image output by the style restoration module can be subjected to an upsampling operation and a first spatial resolution adjustment operation in sequence to obtain the sixth image feature. The first spatial resolution adjustment operation in this embodiment can be a PixelUnshuffle operation.

[0076] Step S32: Connect the sixth image feature and the detail supplementary image feature to obtain a seventh image feature.

[0077] In this embodiment, after the sixth image feature is obtained, the sixth image feature and the detail supplement image feature output by the detail supplement module may be connected to obtain the seventh image feature.

[0078] Step S33: The seventh image feature is processed by multiple dimensionality reduction modules to obtain an intermediate feature map.

[0079] In this embodiment, the cross-fusion module includes multiple dimensionality reduction modules, each of which includes a dimensionality reduction convolution (DRC) and an activation function (such as GeLU). After obtaining the seventh image feature, the seventh image feature can be sequentially input into multiple dimensionality reduction modules for processing, thereby performing continuous dimensionality reduction and obtaining an intermediate feature map. Each time a feature is input into a dimensionality reduction module, it first undergoes dimensionality reduction convolution processing and then an activation function.

[0080] Step S34: performing a second spatial resolution adjustment operation on the intermediate feature map to obtain the sample defogging image.

[0081] In this embodiment, a second spatial resolution adjustment operation is performed on the obtained intermediate feature map to obtain a sample dehazed image output by the cross-fusion module. The second spatial resolution adjustment operation may be a PixelShuffle operation.

[0082] In combination with the above embodiments, in one implementation, the present invention further provides a single image defogging method based on style restoration and detail supplementation. In this embodiment, the style restoration module at least includes: an improved MAE encoder and a MAE decoder, and the above step S12 may specifically include step S41:

[0083] Step S41: the sample foggy image is processed sequentially through the improved MAE encoder and the MAE decoder to obtain the sample style restored image.

[0084] In this embodiment, the main task of the style recovery module is to restore the style of the image. Style features are high-level semantic features, and using a deeper and larger network is conducive to achieving better style feature recovery. Based on this, this embodiment uses a large visual model to extract semantic features to achieve style recovery, which can alleviate the common problems of overfitting and poor generalization performance of single-task image restoration networks. Therefore, the style recovery module of this embodiment is implemented based on the large visual pre-trained model MAE. MAE (Masked Auto Encoder) is a self-supervised pre-trained large visual model. MAE includes: MAE decoder and MAE encoder. This embodiment uses MAE as the baseline network for the style recovery module, and the style recovery module includes: MAE decoder and improved MAE encoder.

[0085] In this embodiment, after a sample foggy image is input into the style restoration module, it is processed sequentially through the improved MAE encoder and decoder, resulting in a sample style-restored image output by the style restoration module. In one specific implementation, to ensure that the input and output image sizes are consistent with the original MAE, this embodiment reduces the resolution of the input image (e.g., the sample foggy image) before feeding it into the style restoration module to accommodate the MAE network.

[0086] In order to enable MAE to perform image dehazing from style restoration, this embodiment implements dehazing MAE by using adapters before and after each MAE encoder layer of the MAE encoder. Specifically, the improved MAE encoder of this embodiment includes: multiple dehazing MAE encoding layer structures connected in sequence. The input sample foggy image is processed in sequence by multiple dehazing MAE encoding layer structures, and then processed by the MAE decoder to finally obtain a sample style restored image. Each dehazing MAE encoding layer structure includes: a first adapter, a MAE encoder layer, and a second adapter connected in sequence.

[0087] In this embodiment, to enable MAE to achieve style recovery capabilities, a first adapter is added before each MAE encoder layer, and a second adapter is added after each MAE encoder layer, based on the data transmission direction in the MAE encoder. The first adapter can be a Channel-Wise Adapter (CW-A), and the second adapter can be a Patch-Wise Adapter (PW-A). The first adapter includes a linear layer and an activation layer, responsible for feature transformation across multiple patches; the second adapter involves a single linear transformation, modifying the characteristics of each feature value before it is processed by the encoder layer.

[0088] In this embodiment, through the first and second adapters, the original self-attention mechanism can effectively identify information of different dimensions, thereby allowing the original MAE to acquire new capabilities, namely, dehazing capabilities. The feature value output by each dehazing MAE coding layer structure in this embodiment can be expressed as: batch_size (the amount of data in each training batch) * patch_num (the number of blocks each image is divided into) * patch_size (the size of each block). Specifically, the output of each dehazing MAE coding layer structure can be obtained using the following formula:

[0089] ;

[0090] Among them, Represents the eigenvalue sent to the tth dehazing MAE coding layer structure, represents the MAE t-th layer encoder, Represent the linear layer weights of the adapters before and after the MAE t-th layer encoder, that is, the weights of the first adapter and the second adapter corresponding to the MAE t-th layer encoder respectively; Represents the bias of the linear layer of the adapter before and after the t-th layer encoder, and GeLU is the activation function. It is worth noting that this embodiment uses the original decoder of the pre-trained model MAE, rather than just using the high-level information generated by the encoder for downstream tasks. Therefore, when designing the adapter, this embodiment chooses a simple linear transformation to preserve the original solution space of the decoder, and uses the identity matrix to initialize the adapter weights to ensure its normal operation. That is, in the above formula will be initialized to the identity matrix.

[0091] In this embodiment, during the training process of the style restoration module, in order to retain more pre-trained knowledge, the network parameters of the MAE encoder layer and the MAE decoder are frozen to prevent any gradient calculation or weight update. At the same time, only the network parameters of the adapters (i.e., the first adapter and the second adapter) are updated, that is, only the adapters perform gradient calculation and weight update.

[0092] Currently, datasets used for dehazing are primarily composed of synthetic data, which may not fully cover all possible real-world degradation scenarios. This can lead to unstable performance of image dehazing networks when encountering unseen scenarios. Furthermore, these datasets are often limited in size, which limits the amount of training, potentially leading to overfitting (where the model overfits to the training data and fails to generalize well to new data). Consequently, the generalization ability of image dehazing networks trained with these datasets is easily limited. To address this issue, this embodiment designs a dehazing module (i.e., a style restoration module) based on a self-masking autoencoder (MAE) to leverage the advantages of PTMs (pre-training models) in image dehazing tasks. This allows the network to acquire rich prior knowledge, thereby improving the generalization ability of the image dehazing network and further ensuring its practical application in various scenarios.

[0093] In one embodiment, if Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram of the overall structure of an image defogging network according to an embodiment of the present invention. Figure 4In [1], the image dehazing network consists of three main modules: the Style Restoration Module (SRM), the Detail Replenishment Module (DRM), and the Cross Fusion Module (CFM). When a foggy image enters the image dehazing network, it is copied into two copies. One copy is sent to the DRM to extract and replenish details, resulting in detail-replenished image features. The other copy is sent to the SRM to restore style, resulting in a style-restored image. After the foggy image passes through the SRM and DRM, all output features are input to the CFM to combine the output information of the SRM and DRM. Through repeated aggregation of high-level information, a clear and clean image is ultimately generated. The SRM includes a MAE decoder and an improved MAE encoder. The improved MAE encoder consists of N dehazing MAE coding layers (dehazing MAE coding layer structure), each of which consists of a CW-A, a MAE encoder layer, and a PW-A. CFM combines the outputs of DRM and SRM to fuse style and detail information. In CFM, the dimensionality reduction operation is achieved by adjusting the number of input channels and output channels. The goal of this is to enable the network to autonomously learn how to utilize style information and detail information. To achieve this, this embodiment increases the feature dimensions of DRM and SRM to a very high level when designing the network, and then connects them together to obtain a 672-dimensional feature map. Next, dimensionality reduction convolution is used to continuously reduce the dimensionality of the feature map to obtain a 48-dimensional feature map. Finally, a PixelShuffle operation is performed, and this process will produce a clear image. In this way, the single image dehazing method proposed in the embodiment of the present invention has better generalization performance and stronger dehazing and restoration capabilities.

[0094] In combination with the above embodiments, in one embodiment, the present invention further provides a single image dehazing method based on style restoration and detail supplementation. In this embodiment, the pre-trained style restoration module is obtained by training based on the style restoration module to be trained. The style restoration module to be trained includes: a pre-trained MAE encoder layer, a pre-trained MAE decoder, a first adapter to be trained, and a second adapter to be trained. The training steps of the style restoration module to be trained may specifically include steps S51 to S57:

[0095] Step S51: inputting a sample clean image corresponding to the sample foggy image into the style restoration module to be trained to obtain a first sample image.

[0096] In this embodiment, when training the image dehazing network, in order to enable MAE to effectively learn the dehazing ability through style restoration, a two-stage progressive multi-task training method is adopted to train the style restoration module and the detail supplementation module according to different subtasks, so that they have the ability of style restoration and detail supplementation respectively, which helps to effectively apply the pre-trained large model to the image dehazing task.

[0097] In the first stage of training, the training of the style restoration module is completed first, and during the training of the style restoration module, all weights except the first adapter to be trained and the second adapter to be trained are frozen.

[0098] When training the style restoration module, a sample clean image corresponding to the sample foggy image is first input into the style restoration module to be trained, obtaining a first sample image output by the style restoration module to be trained. The sample dataset used to train the style restoration module is the same as the sample dataset used to train the image defogging network. That is, the sample foggy images used to train the style restoration module are the same as the sample foggy images used to train the image defogging network.

[0099] Step S52: Obtain a first loss based on the sample clean image and the first sample image.

[0100] In this embodiment, after obtaining the first sample image, the first loss may be calculated based on the sample clean image corresponding to the sample foggy image and the first sample image corresponding to the sample foggy image.

[0101] Step S53: inputting the sample foggy image into the style restoration module to be trained to obtain a second sample image.

[0102] In this embodiment, the sample foggy image may also be input into the style restoration module to be trained to obtain a second sample image output by the style restoration module to be trained.

[0103] Step S54: obtaining a second loss based on the second sample image and the first sample image.

[0104] In this embodiment, after the second sample image is obtained, the second loss may be calculated based on the first sample image corresponding to the sample foggy image and the second sample image corresponding to the sample foggy image.

[0105] Step S55: Obtaining an image reconstruction loss based on the first loss and the second loss.

[0106] In this embodiment, after the first loss and the second loss are obtained, the image reconstruction loss can be obtained based on the first loss and the second loss.

[0107] Step S56: obtaining a style loss based on the second sample image and the sample clean image.

[0108] In this embodiment, the style loss may also be calculated based on the obtained second sample image and the sample clean image.

[0109] In a specific example, the style loss It can be expressed by the following formula:

[0110]

[0111] Here, l represents an intermediate layer of a feature extraction network (such as the style recovery module in this embodiment). In one embodiment, the five layers of Relu-1, Relu-6, Relu-11, Relu-20, and Relu-26 from VGG19 are used. Represents the Gram matrix of the sample clean image corresponding to the lth layer, The Gram matrix representing the second sample image output by the lth layer.

[0112] Step S57: updating the network parameters of the first adapter to be trained and the second adapter to be trained based at least on the image reconstruction loss and the style loss, until the pre-trained style restoration module is obtained.

[0113] In this embodiment, after obtaining the image reconstruction loss and the style loss, the network parameters of the first adapter to be trained and the second adapter to be trained can be updated at least based on the image reconstruction loss and the style loss until a trained first adapter and a trained second adapter are obtained, thereby obtaining a pre-trained style recovery module based on the trained first adapter, the trained second adapter, the pre-trained MAE encoder layer, and the pre-trained MAE decoder.

[0114] In one example, this embodiment proposes a progressive multi-task training method to train the style restoration module. In this stage (i.e., the first stage of training), two propagations are performed to obtain the image reconstruction loss. In the first propagation, a sample clean image Clear is input to the style restoration module to be trained. The task of the style restoration module is to restore the sample clean image to obtain the first sample image. . Use the mean square error (MSE) to evaluate the sample clean image Clear and the first sample image The gap between the first loss In the second propagation process, the sample hazy image Hazy is sent to the style recovery module to be trained. The sample hazy image is processed by the style recovery module to obtain the second sample image In this embodiment, the second sample image is not And the sample clean image Clear is compared, but the second sample image and the first sample image Perform MSE evaluation between them and record it as the second loss . The image reconstruction loss The first loss +Second loss Composition, by back-propagating image reconstruction loss To update the style restoration module, this process is called "progressive multi-task" training. Through this progressive multi-task loss of this embodiment, the ability of the style restoration network in image restoration tasks is improved, thereby enhancing the defogging ability.

[0115] In order to show the advantages of progressive multi-task loss in the training process, in one embodiment, this embodiment also uses the training method of progressive multi-task loss to compare with the other two training methods using "single-task direct loss" and "multi-task direct loss". Among them, "single-task direct loss" can be described as the loss between the second sample image and the sample clean image, that is, ; “Multi-task direct loss” can be described as The specific relationship between the three can be referred to Figure 5 As shown, Figure 5 1 is a schematic diagram showing a comparison between three training methods according to an embodiment of the present invention.

[0116] In combination with the above embodiments, in one embodiment, the present invention further provides a single image defogging method based on style restoration and detail supplementation. In this method, the above step S57 may specifically include steps S61 and S62:

[0117] Step S61: Calculate the total loss based on the image reconstruction loss and the image reconstruction loss weight, and the style loss and the style loss weight.

[0118] In this embodiment, the image reconstruction loss corresponds to an image reconstruction loss weight, and the style loss corresponds to a style loss weight. The image reconstruction loss weight and the style loss weight can be freely set. The total loss can be calculated based on the image reconstruction loss and the image reconstruction loss weight, as well as the style loss and the style loss weight.

[0119] In a specific example, the total loss is: ;in, is the image reconstruction loss, is the style loss, is the image reconstruction loss weight, is the style loss weight.

[0120] Step S62: updating the network parameters of the first adapter to be trained and the second adapter to be trained based on the total loss until the pre-trained style restoration module is obtained.

[0121] In this embodiment, after obtaining the total loss, the network parameters of the first adapter to be trained and the second adapter to be trained can be updated based on the total loss until a trained first adapter and a trained second adapter are obtained, thereby obtaining a pre-trained style restoration module.

[0122] Among them, in the initial stage of training of the style recovery module, that is, during the first convergence of the total loss, the image reconstruction loss weight is set to 1 and the style loss weight is set to 0 to enable the style recovery module to have image reconstruction capabilities; after the first convergence of the total loss, the image reconstruction loss weight is set to 1 and the style loss weight is set to 1000, the accuracy of the model will increase again and converge again, thereby enhancing the style recovery capability of the style recovery module.

[0123] In addition, in order to illustrate the rationality and effectiveness of the training strategy proposed in this embodiment, key indicators in the training process are also measured. Figure 6 As shown, Figure 6 This is a diagram showing the training comparison results of SRM trained in three different ways on SOTS-indoor in the RESIDE dataset, according to an embodiment of the present invention. Figure 6 By generating The image is compared with the clear image, and the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are calculated respectively. Compared with the other two methods ("direct single-task learning" and "direct multi-task learning"), the "progressive multi-task learning" training method has obvious advantages in training speed and accuracy. After further training for 26 epochs, the total loss converges and can be Set to 1, The model converges further when it is set to 1000. The present invention designs a dehazing MAE for style restoration by adjusting the pre-trained model MAE, and adopts a two-stage “progressive multi-task” approach to train the network, so that the MAE can effectively learn the dehazing capability through style restoration.

[0124] In combination with the above embodiments, in one embodiment, the present invention further provides a single image defogging method based on style restoration and detail supplementation. In this method, the above step S15 may specifically include steps S71 to S73:

[0125] Step S71: Based on the sample defogging image and the sample clean image corresponding to the sample foggy image, a first loss function is calculated to obtain a third loss, and the model parameters of the detail supplement module to be trained and the cross-fusion module to be trained are updated based on the third loss.

[0126] In this embodiment, in a two-stage training method, after the first stage of training of the style restoration module to be trained is completed, resulting in a pre-trained style restoration module, the pre-trained style restoration module is completely frozen, and its gradients and parameters are no longer updated, before proceeding to the second stage of training. In the second stage of training, the parameters of the pre-trained style restoration module are fixed, while the model parameters of the detail supplementation module to be trained and the cross-fusion module to be trained are updated.

[0127] In this embodiment, the first loss function can be performed based on the sample clean image corresponding to the sample defogging image and the sample foggy image. The third loss is calculated. Based on the third loss, the model parameters of the detail supplement module to be trained and the cross fusion module to be trained are updated. In a specific example, the first loss function is L2Loss, that is, for:

[0128] ;

[0129] Among them, y represents the sample clean image, Represents the sample dehazed image, and i represents the i-th pixel value of the image.

[0130] Step S72: After the third loss converges, the second loss function is calculated based on the sample defogged image and the sample clean image corresponding to the sample foggy image to obtain a fourth loss, and the model parameters of the detail supplement module to be trained and the cross fusion module to be trained are continued to be updated based on the fourth loss until the fourth loss converges to obtain a trained detail supplement module and a trained cross fusion module.

[0131] In this embodiment, after the third loss converges, the second loss function is calculated based on the sample clean image corresponding to the sample defogging image and the sample foggy image to obtain the fourth loss. The second loss function is a loss function that incorporates SSIM. In a specific example, the second loss function for:

[0132] ;

[0133] Among them, y represents the sample clean image, represents the sample dehazed image, and Represent the average pixel values ​​of the sample clean image and the sample dehazed image, and Represent the variance of the sample dehazed image and the sample clean image respectively, represents the covariance between the sample clean image and the sample dehazed image. and In order to prevent the denominator from becoming 0, in this embodiment, and .

[0134] After obtaining the fourth loss, the model parameters of the detail supplement module to be trained and the cross fusion module to be trained can be updated based on the fourth loss until the fourth loss converges to obtain the trained detail supplement module and the trained cross fusion module.

[0135] Step S73: obtaining the trained image defogging network based on the pre-trained style restoration module, the trained detail supplement module and the trained cross-fusion module.

[0136] In this embodiment, after obtaining the trained detail supplement module and the trained cross fusion module, a trained image dehazing network can be obtained based on the pre-trained style restoration module, the trained detail supplement module and the trained cross fusion module.

[0137] In a specific example, in the second stage of training, batch_size is set to 4 and the learning rate is initialized to Continue to use cosine decay to update the learning rate and maintain the minimum learning rate .

[0138] Image dehazing plays a vital role in security and surveillance systems, and can significantly improve the monitoring effects of these systems. By removing image blur and color distortion caused by haze, the dehazing algorithm can restore clearer and more realistic scene details, thereby improving the quality and reliability of video surveillance. In one embodiment, an image restoration preprocessing module (such as an image dehazing preprocessing module) can be deployed in the system using the single image dehazing method based on style restoration and detail supplementation shown in any of the aforementioned embodiments. The visual systems in these fields can obtain more accurate data support and make more informed decisions. The deployment of the system is divided into a training phase and an inference phase, such as Figure 7 As shown, Figure 7This is a flowchart of the training and inference of an image dehazing network according to an embodiment of the present invention. The training process is as follows:

[0139] Constructing Datasets: In the experimental section, this example evaluates the performance of the model on three different datasets: RESIDE, Haze4K, and NH-Haze. The RESIDE dataset is widely used for dehazing real and synthetic images. It consists of five subsets: the Indoor Training Set (ITS), the Outdoor Training Set (OTS), the Synthetic Object Test Set (SOTS), the Real Task-Driven Test Set (RTTS), and the Hybrid Subjective Test Set (HSTS). In the experiments, this example primarily uses the ITS and OTS subsets for training and evaluates the model on the SOTS subset. The Haze4K dataset consists of 3,000 synthetic training images and 1,000 synthetic test images, and this example also evaluates the key metrics of the model. In addition, training and testing were performed on the real-scene dataset NH-Haze.

[0140] Read image data: Read the corresponding image data and ground truth data from the training dataset. The batch size of loaded training image data for each GPU is 32.

[0141] Configure training parameters: Use PyTorch 2.0 to build the model and train on a system with four NVIDIA 3090 GPUs or higher. See Table 2 for specific training parameters. For specific training strategies, see the single image dehazing method based on style restoration and detail addition described in any of the above examples.

[0142] The model outputs the prediction result. The restoration result obtained by model inference is the same RGB image as the input image.

[0143] Backpropagation after loss calculation: During the training process, the loss between the input image and the output image is calculated. The specific loss function design can be found in the single image dehazing method based on style restoration and detail supplementation shown in any of the above embodiments.

[0144] Saving training results: After training is complete, the final model is obtained and saved offline, such as in HDF5, TensorFlowSavedModel, ONNX, etc., and the model can be exported using the corresponding API or tools. The model can also be verified on the validation set.

[0145] The reasoning process during implementation can be divided into the following steps:

[0146] Loading models: Use ONNX, TensorRT, and other methods to load offline models. Using tools such as ONNX and TensorRT can significantly improve the efficiency of model loading and inference, and support cross-platform flexibility.

[0147] Data Reading: Images are read and decoded from the camera in real time. During the data reading process, the program initializes the camera connection, captures image frames in real time, and decodes them, converting them into analyzable digital information for subsequent image processing and analysis algorithms. The camera is initialized using OpenCV or a similar computer vision library. Then, parameters such as frame rate and resolution are set using the camera's API to begin capturing the image stream in real time. After each frame is captured, the built-in decoder converts the raw image data (e.g., H.264 encoding) into matrix data in RGB or grayscale format for subsequent processing. This process is typically executed continuously in a loop to ensure the continuity and real-time nature of the image data.

[0148] Feed the read image into the model and infer

[0149] The output of the model is the dehazed image, which is then fed into the next subsequent visual task to be completed.

[0150] Table 2 System hardware configuration and software version information

[0151]

[0152] In addition, in another embodiment, the single image dehazing method based on style restoration and detail supplementation proposed in the above embodiment is often deployed and implemented as part of image processing software. With the rise of computational photography in the past two years, removing haze has become one of the basic AI capabilities. Whether it is professional graphic design software or lightweight applications for ordinary users, it can provide users with a powerful tool to improve the quality of photos and other visual content. Especially in post-processing of photography, the dehazing function will play an irreplaceable role. System deployment is divided into training phase and inference phase, such as Figure 8 As shown, Figure 8 FIG. 4 is a flowchart of training and inference of an image dehazing network according to another embodiment of the present invention.

[0153] 1. Training Phase: Since the process and parameter settings of the training phase are the same as those in the previous embodiment, they will not be described here. In short, it includes the steps of data preparation, model selection and initialization, parameter configuration, training process, verification and tuning, and saving the best model.

[0154] 2. Deployment and inference phase:

[0155] Model Loading: Offline models can be loaded using methods such as ONNX and TensorRT. Using tools like ONNX and TensorRT significantly improves model loading and inference efficiency, while also providing cross-platform flexibility. When deploying this inference module as part of image processing software, specific considerations apply. The trained model and its inference logic can be seamlessly integrated into the existing image processing software framework, ensuring smooth collaboration with other functional modules.

[0156] Data loading: Real-time data: Image frames are read from the camera and fed into the model after necessary preprocessing (such as scaling and normalization). Custom data: Users upload image or video files, which the system automatically parses and converts into a format acceptable to the model, ensuring the same preprocessing standards as real-time data.

[0157] Save results: Ensure that the inference results are presented to users in an intuitive manner. Display the detection results or classification labels through a graphical interface to improve the overall user experience.

[0158] Through the above deployment steps, the inference module can be effectively integrated into the image processing software, maintaining consistency with the training phase while fully considering the needs of different data sources and actual application scenarios. This deployment method enhances the functionality and flexibility of the software, making it more adaptable to diverse user needs.

[0159] In one embodiment, in order to verify the effectiveness of the single image defogging method based on style restoration and detail supplementation proposed in the embodiment of the present invention, Figure 9 and Figure 10 In this paper, different methods are used to present the visualization results of image dehazing algorithms for indoor and outdoor scenes. The algorithms compared with the method proposed in the embodiment of the present invention include mainstream dehazing algorithms such as ADONet, GridDehazeNet, FFANet, C2PNet, and DeHamer. It can be seen in the visualization comparison of indoor scenes that the images generated by AODNet have obvious color differences. In contrast, C2PNet performs well in both indoor and outdoor scenes. However, this method (Ours) even surpasses C2PNet in terms of naturalness and fidelity. On the more realistic dataset NH-Haze, the method of this paper still has obvious advantages. In thick fog areas, SRDR has more realistic colors and richer texture details, which is closer to real images, indicating that the method proposed in this invention has achieved state-of-the-art performance on many mainstream datasets. Among them, Figure 9 This is a visual comparison diagram of a single image dehazing method based on style restoration and detail supplementation and other mainstream methods shown in an embodiment of the present invention; Figure 10This is another visual comparison diagram of a single image dehazing method based on style restoration and detail supplementation according to an embodiment of the present invention and other mainstream methods.

[0160] In one embodiment, the performance indicators of various dehazing networks across multiple datasets are shown in Table 3, with the best results highlighted in bold. Compared with other methods, the method proposed in the present invention (Ours) has a significant PSNR advantage on all datasets. Specifically, on the RESIDE test set, the SRDR network achieved a PSNR of 43.46dB and 40.63dB in indoor and outdoor scenes, respectively, which are 0.90dB and 3.95dB higher than C2PNet, respectively. On another synthetic dataset, Haze 4K, the method proposed in the present invention achieved a PSNR of 35.11dB and an SSIM of 0.991, which is also better than other existing image dehazing methods. In addition, the performance on NH-Haze, a real-world dataset, was also evaluated. SRDR still achieved the best value in the PSNR indicator, but the SSIM indicator was slightly lower than C2PNet.

[0161] Table 3 Quantitative comparison of the present invention and other mainstream methods

[0162]

[0163] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0164] Based on the same inventive concept, an embodiment of the present invention provides a single image defogging device based on style restoration and detail supplementation. Figure 11 , Figure 11 This is a structural block diagram of a single image defogging device based on style restoration and detail supplementation provided by an embodiment of the present invention. Figure 11 As shown, the single image defogging device based on style restoration and detail supplementation in this embodiment may include:

[0165] An image input module is configured to input a sample foggy image into an image defogging network to be trained. The image defogging network to be trained includes at least: a pre-trained style restoration module, a detail supplementation module to be trained, and a cross-fusion module to be trained; the detail supplementation module is composed of multi-directional and multi-dimensional convolutions to extract and supplement image details; the style restoration module is implemented based on a large pre-trained visual model to restore the image style; and the cross-fusion module is configured to repeatedly aggregate the output information of the detail supplementation module and the style restoration module.

[0166] a style processing module, configured to process the sample foggy image using the pre-trained style restoration module to obtain a sample style restored image;

[0167] a detail processing module, configured to process the sample foggy image through the detail supplementation module to be trained to obtain detail supplementation image features;

[0168] a feature fusion module, configured to input the features of the sample style restored image and the detail supplemented image into the cross fusion module to be trained, to obtain a sample defogging image;

[0169] a model training module, configured to update model parameters of the detail supplement module to be trained and the cross-fusion module to be trained based on the sample defogging image and the sample clean image corresponding to the sample foggy image, until a trained image defogging network is obtained;

[0170] The image defogging module is used to input the image to be defogged into the trained image defogging network to obtain the defogged image output by the trained image defogging network.

[0171] Optionally, the detail supplementation module includes at least: a first convolution, a second convolution, a third convolution and a fourth convolution, wherein the first convolution, the second convolution, the third convolution and the fourth convolution are respectively responsible for detail supplementation in four dimensions: point level, horizontal level, vertical level and plane level;

[0172] Detailed supplementary modules, including:

[0173] a convolution processing module, configured to input the sample foggy image into the first convolution, the second convolution, the third convolution, and the fourth convolution respectively for processing to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature;

[0174] a feature connection module, configured to connect the sample foggy image, the first image feature, the second image feature, the third image feature, and the fourth image feature to obtain a fifth image feature;

[0175] A resolution adjustment module is used to adjust the spatial resolution of the fifth image feature to obtain the detail supplementary image feature.

[0176] Optionally, the feature fusion module includes:

[0177] a first processing module, configured to perform an upsampling operation and a first spatial resolution adjustment operation on the sample style restoration image in the cross-fusion module to be trained, to obtain a sixth image feature;

[0178] a second processing module, configured to connect the sixth image feature and the detail supplementary image feature to obtain a seventh image feature;

[0179] A third processing module, configured to process the seventh image feature through a plurality of dimensionality reduction modules to obtain an intermediate feature map; each of the plurality of dimensionality reduction modules includes: a dimensionality reduction convolution and an activation function;

[0180] The fourth processing module is used to perform a second spatial resolution adjustment operation on the intermediate feature map to obtain the sample defogging image.

[0181] Optionally, the style restoration module includes at least: an improved MAE encoder and a MAE decoder;

[0182] The improved MAE encoder comprises: a plurality of defogging MAE coding layer structures connected in sequence, each defogging MAE coding layer structure comprising: a first adapter, a MAE encoder layer and a second adapter connected in sequence;

[0183] The first adapter includes a linear layer and an activation layer, which is responsible for feature transformation across multiple blocks; the second adapter involves a single linear transformation;

[0184] Style restoration module, including:

[0185] The encoding and decoding processing module is used to process the sample foggy image through the improved MAE encoder and the MAE decoder in sequence to obtain the sample style restored image.

[0186] Optionally, the pre-trained style restoration module is obtained by training based on the style restoration module to be trained, and the style restoration module to be trained includes: a pre-trained MAE encoder layer, a pre-trained MAE decoder, a first adapter to be trained, and a second adapter to be trained; the apparatus further includes: a style restoration training module, configured to train the style restoration module to be trained, the style restoration training module including:

[0187] A first input module is configured to input a sample clean image corresponding to the sample foggy image into the style restoration module to be trained to obtain a first sample image;

[0188] A first calculation module, configured to obtain a first loss based on the sample clean image and the first sample image;

[0189] A second input module, configured to input the sample foggy image into the style restoration module to be trained to obtain a second sample image;

[0190] a second calculation module, configured to obtain a second loss based on the second sample image and the first sample image;

[0191] a third calculation module, configured to obtain an image reconstruction loss based on the first loss and the second loss;

[0192] a fourth calculation module, configured to obtain a style loss based on the second sample image and the sample clean image;

[0193] A first parameter updating module is configured to update network parameters of the first adapter to be trained and the second adapter to be trained based at least on the image reconstruction loss and the style loss until the pre-trained style restoration module is obtained.

[0194] Optionally, the first parameter updating module includes:

[0195] a total loss calculation module, configured to calculate a total loss based on the image reconstruction loss and the image reconstruction loss weight, and the style loss and the style loss weight;

[0196] a second parameter updating module, configured to update network parameters of the first adapter to be trained and the second adapter to be trained based on the total loss until the pre-trained style restoration module is obtained;

[0197] During the first convergence of the total loss, the image reconstruction loss weight is set to 1 and the style loss weight is set to 0, so that the style restoration module has image reconstruction capability;

[0198] After the total loss converges for the first time, the image reconstruction loss weight is set to 1 and the style loss weight is set to 1000 to enhance the style restoration capability of the style restoration module.

[0199] Optionally, the model training module includes:

[0200] a first updating module, configured to calculate a first loss function based on the sample defogging image and the sample clean image corresponding to the sample foggy image to obtain a third loss, and update model parameters of the detail supplement module to be trained and the cross fusion module to be trained based on the third loss;

[0201] a second updating module, configured to, after the third loss converges, calculate a second loss function based on the sample defogging image and the sample clean image corresponding to the sample foggy image to obtain a fourth loss, and continue to update the model parameters of the detail supplement module to be trained and the cross fusion module to be trained based on the fourth loss until the fourth loss converges, thereby obtaining a trained detail supplement module and a trained cross fusion module;

[0202] A network training module is used to obtain the trained image dehazing network based on the pre-trained style restoration module, the trained detail supplement module and the trained cross-fusion module.

[0203] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the single image dehazing method based on style restoration and detail supplementation as described in any of the above embodiments of the present invention.

[0204] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When executed, the processor implements the steps of the single image dehazing method based on style restoration and detail supplementation described in any of the above embodiments of the present invention.

[0205] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0206] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0207] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0211] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0212] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0213] The above is a detailed introduction to the single image defogging method, device, equipment and medium based on style restoration and detail supplementation provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A single image dehazing method based on style restoration and detail supplementation, characterized in that: The method comprises: Inputting a sample foggy image into a to-be-trained image defogging network, the to-be-trained image defogging network comprising at least: a pre-trained style restoration module, a to-be-trained detail supplementation module, and a to-be-trained cross-fusion module; the detail supplementation module comprises multi-directional and multi-dimensional convolutions for extracting and supplementing image details; the style restoration module is implemented based on a large pre-trained visual model for restoring the image style; and the cross-fusion module is configured to repeatedly aggregate the output information of the detail supplementation module and the style restoration module; Processing the sample foggy image using the pre-trained style restoration module to obtain a sample style restored image; Processing the sample foggy image by the detail supplementation module to be trained to obtain detail supplementation image features; Inputting the sample style restoration image and the detail supplement image features into the cross fusion module to be trained to obtain a sample defogging image; Based on the sample defogging image and the sample clean image corresponding to the sample foggy image, updating the model parameters of the detail supplement module to be trained and the cross fusion module to be trained until a trained image defogging network is obtained; Inputting the image to be defogged into the trained image defogging network to obtain a defogged image output by the trained image defogging network; The detail supplement module includes at least: a first convolution, a second convolution, a third convolution and a fourth convolution, wherein the first convolution, the second convolution, the third convolution and the fourth convolution are respectively responsible for detail supplementation in four dimensions: point level, horizontal level, vertical level and plane level; The style restoration module includes at least: an improved MAE encoder and a MAE decoder; the improved MAE encoder includes: multiple dehazing MAE coding layer structures connected in sequence, each dehazing MAE coding layer structure includes: a first adapter, a MAE encoder layer, and a second adapter connected in sequence; the first adapter includes a linear layer and an activation layer, responsible for feature conversion across multiple blocks; the second adapter involves a single linear transformation.

2. The single image dehazing method based on style restoration and detail supplementation according to claim 1, characterized in that: The sample foggy image is processed by the detail supplementation module to be trained to obtain detail supplementation image features, including: Inputting the sample foggy image into the first convolution, the second convolution, the third convolution, and the fourth convolution respectively for processing to obtain a first image feature, a second image feature, a third image feature, and a fourth image feature; Connecting the sample foggy image, the first image feature, the second image feature, the third image feature, and the fourth image feature to obtain a fifth image feature; The spatial resolution of the fifth image feature is adjusted to obtain the detail supplementary image feature.

3. The single image dehazing method based on style restoration and detail supplementation according to claim 1, characterized in that: Inputting the sample style restoration image and the detail supplement image features into the cross fusion module to be trained to obtain a sample defogging image, including: In the cross-fusion module to be trained, an upsampling operation and a first spatial resolution adjustment operation are performed on the sample style restoration image to obtain a sixth image feature, Connecting the sixth image feature and the detail supplementary image feature to obtain a seventh image feature; Processing the seventh image feature through multiple dimensionality reduction modules to obtain an intermediate feature map; each of the multiple dimensionality reduction modules includes: dimensionality reduction convolution and activation function; A second spatial resolution adjustment operation is performed on the intermediate feature map to obtain the sample defogging image.

4. The single image dehazing method based on style restoration and detail supplementation according to claim 1, characterized in that: Processing the sample foggy image by the pre-trained style restoration module to obtain a sample style restored image includes: The sample foggy image is processed sequentially through the improved MAE encoder and the MAE decoder to obtain the sample style restored image.

5. The single image dehazing method based on style restoration and detail supplementation according to claim 4, characterized in that: The pre-trained style restoration module is obtained by training the style restoration module to be trained, and the style restoration module to be trained includes: a pre-trained MAE encoder layer, a pre-trained MAE decoder, a first adapter to be trained, and a second adapter to be trained; the training steps of the style restoration module to be trained include at least: Inputting a sample clean image corresponding to the sample foggy image into the style restoration module to be trained to obtain a first sample image; Obtaining a first loss based on the sample clean image and the first sample image; Inputting the sample foggy image into the style restoration module to be trained to obtain a second sample image; Obtaining a second loss based on the second sample image and the first sample image; Obtaining an image reconstruction loss based on the first loss and the second loss; Obtaining a style loss based on the second sample image and the sample clean image; Based at least on the image reconstruction loss and the style loss, network parameters of the first adapter to be trained and the second adapter to be trained are updated until the pre-trained style restoration module is obtained.

6. The single image dehazing method based on style restoration and detail supplementation according to claim 5, characterized in that: Updating network parameters of the first adapter to be trained and the second adapter to be trained based at least on the image reconstruction loss and the style loss until the pre-trained style restoration module is obtained, comprising: Calculating a total loss based on the image reconstruction loss and the image reconstruction loss weight, and the style loss and the style loss weight; updating network parameters of the first adapter to be trained and the second adapter to be trained based on the total loss until the pre-trained style restoration module is obtained; During the first convergence of the total loss, the image reconstruction loss weight is set to 1 and the style loss weight is set to 0, so that the style restoration module has image reconstruction capability; After the total loss converges for the first time, the image reconstruction loss weight is set to 1 and the style loss weight is set to 1000 to enhance the style restoration capability of the style restoration module.

7. The single image dehazing method based on style restoration and detail supplementation according to any one of claims 1 to 6, characterized in that: Based on the sample defogging image and the sample clean image corresponding to the sample foggy image, updating the model parameters of the detail supplement module to be trained and the cross fusion module to be trained until a trained image defogging network is obtained, including: Calculating a first loss function based on the sample defogging image and the sample clean image corresponding to the sample foggy image to obtain a third loss, and updating model parameters of the detail supplement module to be trained and the cross fusion module to be trained based on the third loss; After the third loss converges, a second loss function is calculated based on the sample defogging image and the sample clean image corresponding to the sample foggy image to obtain a fourth loss, and model parameters of the detail supplement module to be trained and the cross fusion module to be trained are continuously updated based on the fourth loss until the fourth loss converges, thereby obtaining a trained detail supplement module and a trained cross fusion module; The trained image defogging network is obtained based on the pre-trained style restoration module, the trained detail supplement module and the trained cross-fusion module.

8. A single image defogging device based on style restoration and detail supplementation, characterized in that: The device comprises: An image input module is configured to input a sample foggy image into an image defogging network to be trained. The image defogging network to be trained includes at least: a pre-trained style restoration module, a detail supplementation module to be trained, and a cross-fusion module to be trained; the detail supplementation module is composed of multi-directional and multi-dimensional convolutions to extract and supplement image details; the style restoration module is implemented based on a large pre-trained visual model to restore the image style; and the cross-fusion module is configured to repeatedly aggregate the output information of the detail supplementation module and the style restoration module. a style processing module, configured to process the sample foggy image using the pre-trained style restoration module to obtain a sample style restored image; a detail processing module, configured to process the sample foggy image through the detail supplementation module to be trained to obtain detail supplementation image features; a feature fusion module, configured to input the features of the sample style restored image and the detail supplemented image into the cross fusion module to be trained, to obtain a sample defogging image; a model training module, configured to update model parameters of the detail supplement module to be trained and the cross-fusion module to be trained based on the sample defogging image and the sample clean image corresponding to the sample foggy image, until a trained image defogging network is obtained; An image defogging module is configured to input the image to be defogged into the trained image defogging network to obtain a defogged image output by the trained image defogging network; The detail supplement module includes at least: a first convolution, a second convolution, a third convolution and a fourth convolution, wherein the first convolution, the second convolution, the third convolution and the fourth convolution are respectively responsible for detail supplementation in four dimensions: point level, horizontal level, vertical level and plane level; The style restoration module includes at least: an improved MAE encoder and a MAE decoder; the improved MAE encoder includes: multiple dehazing MAE coding layer structures connected in sequence, each dehazing MAE coding layer structure includes: a first adapter, a MAE encoder layer, and a second adapter connected in sequence; the first adapter includes a linear layer and an activation layer, responsible for feature conversion across multiple blocks; the second adapter involves a single linear transformation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the single image defogging method based on style restoration and detail supplementation is implemented as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the single image defogging method based on style restoration and detail supplementation is implemented as claimed in any one of claims 1 to 7.

Images