Image defogging method based on dual-task collaborative network

Abstract

The invention relates to the technical field of image defogging, in particular to an image defogging method based on a dual-task collaborative network, and the method comprises the steps: S1, collecting a foggy RGB image and a sparse depth map; s2, constructing a depth completion network, and outputting an initial prediction depth map; and S3, constructing a defogging network, and outputting a predicted defogged image. And S4, inputting the predicted defogged image and the sparse depth map into the depth completion network again, and outputting a final predicted depth map. And S5, obtaining a defogging difference mapping matrix and a depth difference mapping matrix. And S6, constructing a defogging loss function and a depth loss function. And S7, obtaining a defogging model and a depth completion model. And S8, inputting the foggy RGB image and the corresponding sparse depth map into the defogging model and the depth completion model, and outputting the defogged RGB image and the corresponding depth map. The method has the advantages that depth completion and image defogging tasks complement and cooperate, the depth completion and the image defogging tasks are synchronously optimized in a unified framework, and a clear image in a foggy day scene is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image dehazing technology, and in particular to an image dehazing method based on a dual-task collaborative network. Background Technology

[0002] Autonomous vehicles are typically equipped with cameras and LiDAR as sensors. Cameras are used to capture RGB images, while LiDAR is used to acquire 3D point cloud information. In foggy conditions, the image quality captured by the camera degrades, so image dehazing is necessary to improve its applicability in downstream applications.

[0003] Depth completion methods aim to densify sparse point clouds acquired by LiDAR. Because the original point cloud data is sparse and difficult to use directly, depth completion algorithms are needed to upsample the sparse depth map into a dense depth map, guided by RGB images. However, in foggy conditions, image quality degradation leads to a decrease in the performance of depth completion algorithms.

[0004] Currently, the mainstream dehazing algorithms are mainly divided into two types: Non-deep learning methods: These methods are based on traditional image processing techniques, and their processing flow is clearly interpretable. However, they have limitations in complex real-world scenes; for example, dark channel prior methods struggle to effectively process images containing sky regions.

[0005] Deep learning methods: This method utilizes deep learning techniques to dehaze single-frame images, mainly divided into three implementation methods: supervised, semi-supervised, and unsupervised. Although it can achieve high-quality dehazing results by learning image features, existing methods generally suffer from insufficient utilization of prior knowledge and fail to fully integrate the potential guiding role of real physical models and geometric constraints on dehazing results. Summary of the Invention

[0006] Based on this, this invention proposes an image dehazing method based on a dual-task collaborative network, integrating the dehazing network and the deep completion network into a unified learning framework. By introducing a dual-task interaction mechanism based on difference mapping, collaborative optimization of deep completion and image dehazing is achieved.

[0007] To achieve the above objectives, the technical solution of this invention is implemented as follows: an image dehazing method based on a dual-task cooperative network, comprising: S1: Acquire a foggy RGB image and its corresponding sparse depth map; S2: Construct a depth completion network. Input the foggy RGB image and sparse depth map into the depth completion network, and output the initial predicted depth map. S3: Construct a dehazing network by taking the hazy RGB image and the initial predicted depth map as input, and outputting the predicted dehazed image; S4: Input the predicted dehazed image and sparse depth map into the depth completion network again to output the final predicted depth map; S5: Based on the dehazing error between the real dehazing image and the predicted dehazing image, obtain the dehazing difference mapping matrix; Based on the depth error between the final predicted depth map and the true depth map, a depth difference mapping matrix is ​​obtained; S6: Construct a dehazing loss function based on the dehazing difference mapping matrix and depth error; A depth loss function is constructed based on the depth difference mapping matrix and the dehazing error; S7: By minimizing the dehazing loss function and the depth loss function, the dehazing model and the depth completion model are finally trained. S8: Input the hazy RGB image to be dehazed and the corresponding sparse depth map into the dehazing model and the depth completion model, and output the dehazed RGB image and the corresponding depth map.

[0008] Furthermore, in step S1, a foggy RGB image is acquired by a camera; a point cloud is acquired by a LiDAR, and the point cloud is converted into a corresponding sparse depth image.

[0009] Furthermore, the dehazing network adopts the U-Net structure, which includes an encoder and a decoder; both the encoder and the decoder include multiple dehazing module groups, each of which consists of multiple stacked dehazing modules; the dehazing module groups perform residual processing and attention processing on the input features.

[0010] Furthermore, step S3 includes the following steps: S31: In the encoder, the foggy RGB image is processed by the first dehazing module group and then concatenated with the initial predicted depth map. The concatenated features are then passed through the subsequent dehazing module groups to complete feature encoding and obtain the encoded features. S32: In the decoder, multiple dehazing module groups combine the output features of the corresponding level of the dehazing module group in the encoder to complete feature decoding and obtain the decoded features; S33: Add the decoded features to the corresponding elements of the hazy RGB image to obtain the predicted dehazed image.

[0011] Furthermore, step S31 includes: alternating downsampling and dehazing module group operations on the concatenated features to obtain encoded features; Step S32 includes: upsampling the encoded features and concatenating them with the output features of the corresponding level of the dehazing module group in the encoder, inputting them into the first dehazing module group of the decoder for decoding, and then alternating between upsampling and dehazing module group operations to finally obtain the decoded features.

[0012] Furthermore, the dehazing error between the actual dehazed image and the predicted dehazed image is: ; The final depth error between the predicted depth map and the true depth map is: ; in, R j This represents the dehazing error between the actual dehazed image and the predicted dehazed image. This represents the predicted dehazed image; J gt Represents a true dehazed image; R d The final depth error between the predicted depth map and the true depth map; This represents the final predicted depth map; D gt This represents the true depth map.

[0013] Furthermore, the dehazing loss function is: ; The depth loss function is: ; in, This represents the dehazing loss function. A e Represents the depth difference mapping matrix; Represents the depth loss function. A d This represents the dehazing difference mapping matrix.

[0014] Furthermore, in step S5, the dehazing error and depth error are respectively processed by the difference mapping module to obtain the dehazing difference mapping matrix and the depth difference mapping matrix.

[0015] Furthermore, the difference mapping module includes depthwise separable convolution, multilayer perceptron, and sigmoid activation function; the input features are extracted locally through depthwise separable convolution, and the extracted features are sequentially passed through multilayer perceptron and sigmoid activation function to generate the difference mapping matrix.

[0016] The present invention can achieve the following beneficial effects: 1) Depth completion and image dehazing tasks complement and collaborate with each other. They are optimized synchronously within a unified framework, which significantly improves the accuracy of clear image reconstruction and depth estimation in foggy scenes.

[0017] 2) By explicitly marking insufficient dehazing areas using a dehazing difference mapping matrix, the depth completion network is guided to improve the depth accuracy of key areas. At the same time, the depth difference mapping matrix is ​​used to focus on areas with large depth errors, providing reverse assistance to the dehazing network. Depth information enhances the structure perception of the dehazing network, while accurate depth provides reliable physical priors for dehazing, forming a two-way closed-loop optimization.

[0018] 3) Introduce LiDAR interaction to enhance the multimodal comprehensive perception capability of the environment in foggy conditions. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of an image dehazing method based on a dual-task cooperative network provided according to an embodiment of the present invention; Figure 2 This is a diagram of a dual-task collaborative network framework provided according to an embodiment of the present invention; Figure 3 This is a deep completion network framework diagram provided according to an embodiment of the present invention; Figure 4 This is a structural diagram of a defogging module provided according to an embodiment of the present invention; Figure 5 This is a structural diagram of the difference mapping module provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] The invention will now be described in detail with reference to specific embodiments.

[0025] In traditional atmospheric scattering models, the mathematical expression of the atmospheric scattering model is as follows: ; ; in I x For in pixels x The pixel values ​​of the observed image at that location, This represents the ideally restored image (dehazed target image) under fog-free conditions. This represents the atmospheric light value (considered a constant in defogging scenarios). This represents the scene transmittance. β represents the attenuation factor caused by fog particle scattering. J ( x ) t ( x This is called the direct transmission component. (1- t ( x )) is the atmospheric light component.

[0026] Methods based on atmospheric scattering models heavily rely ont ( x )and J ( x Accurate estimation of ) is needed, but the uneven distribution of smog in reality leads to variations in local areas of the image. t ( x The difficulty in accurately estimating this limits the improvement of defogging performance.

[0027] End-to-end dehazing methods do not require atmospheric scattering models; they directly recover clear images from hazy maps, thus eliminating the need for atmospheric scattering models. t ( x )and J ( x The image dehazing relies heavily on prior information. However, the lack of a physical model for guidance poses a challenge to clear image restoration. To address this, dehazing methods based on prior information (such as the dark channel method) have been proposed, but a solution that effectively utilizes atmospheric scattering models to guide image dehazing is still lacking.

[0028] This invention notes the complementary nature of depth completion and image dehazing in autonomous driving and robot vision systems: depth information provides crucial physical guidance for the image dehazing process, while high-quality dehazed images significantly improve the accuracy of depth completion. However, most existing deep learning methods fail to fully explore the correlation between depth information in fogged images and their inherent characteristics.

[0029] Given that modern mobile devices are equipped with cameras and lidar as standard, this invention is the first to introduce sparse depth images acquired by lidar into fog-free image reconstruction.

[0030] This invention presents an image dehazing method based on a dual-task collaborative network, integrating image dehazing and depth completion into a unified learning framework. This network achieves collaborative optimization of depth completion and image dehazing by introducing a difference-map-based dual-task interaction mechanism, ultimately constructing a dual-task-driven intelligent dehazing method. This innovative design not only solves the problem of isolated processing of the two tasks in traditional methods but also significantly improves overall performance through mutual promotion between tasks.

[0031] like Figures 1 to 5 As shown, an image dehazing method based on a dual-task cooperative network provided by an embodiment of the present invention includes: S1: Acquire a foggy RGB image and its corresponding sparse depth map.

[0032] Specifically, a camera captures foggy RGB images. A LiDAR scanner captures point clouds, which are then converted into corresponding sparse depth images.

[0033] S2: Construct a Depth Completion Network. Input the hazy RGB image and a sparse depth map into the Depth Completion Network, and output an initial predicted depth map. The initial predicted depth map is used to guide the dehazing of the hazy RGB image.

[0034] In this embodiment, the deep completion network is selected as follows: Figure 3 The example shown is LRRU (Long-short Range Recurrent Updating Networks). Figure 3 In the middle, C represents feature concatenation, and TDU represents target update module.

[0035] It should be noted that LRRU is an existing technology. It first densifies the sparse depth map using a simple non-learning method. Then, based on a long short-range iteration strategy, it iteratively optimizes the initial depth map using target-dependent update units to generate a high-precision dense depth map. This invention, through LRRU, achieves fast and accurate depth completion while maintaining a relatively small computational and parameter count, and still guarantees good generation results despite reducing the number of parameters and computational costs.

[0036] S3: Construct a dehazing network by inputting the hazy RGB image and the initial predicted depth map into the dehazing network, and outputting the predicted dehazed image.

[0037] Specifically, the dehazing network adopts a five-level U-Net (U-shaped network) structure, which includes an encoder and a decoder. Both the encoder and decoder include multiple dehazing module groups (DhBlocks). Each dehazing module group consists of multiple stacked dehazing modules (DhBlocks). The dehazing module groups perform residual processing and attention processing on the input features.

[0038] In this embodiment, the encoder includes three dehazing module groups, with the number of dehazing modules in each group being [2, 2, 4]. The decoder includes two dehazing module groups, with the number of dehazing modules in each group being [2, 2].

[0039] like Figure 4 As shown, the residual processing of the dehazing module is as follows: the input features are processed by two 3×3 convolutions (Conv3×3), and then added to the input features to output the residual.

[0040] Attention processing in the dehazing module: The residual is sequentially processed by 3×3 convolution, channel attention, and spatial attention. Finally, the residual is added to the 3×3 convolution and the final feature is output.

[0041] Furthermore, S31: In the encoder, it consists of successive down-sampling and dehazing module groups. Down-sampling is used to reduce the feature resolution to extract large-scale semantic information. At this time, the dehazing module in the dehazing module group further refines the features through convolution and attention mechanisms. That is, in the encoder, the hazy RGB image is processed by convolution and the first dehazing module group, and then concatenated with the initial predicted depth map. The concatenated features are then passed through subsequent dehazing module groups to complete feature encoding and obtain the encoded features.

[0042] In this invention, firstly, low-level features of the foggy image are extracted using convolution, and the feature representation capability is enhanced by DhBlock. Then, the output features are concatenated and fused with sparse depth information from the initial predicted depth map. The fused features are then input into a U-Net-based network structure.

[0043] S32: In the decoder, multiple dehazing module groups combine the output features of the corresponding level of the dehazing module group in the encoder to complete feature decoding and obtain the decoded features.

[0044] Specifically, the decoder consists of successive up-sampling and dehazing module groups. Up-sampling is used to progressively restore the spatial resolution of the features. Meanwhile, the dehazing module in the dehazing module group enhances the feature recovery, ultimately restoring the output to the same spatial size as the input. That is, in the decoder, the encoded features obtained from the encoder are upsampled and concatenated with the output features of the corresponding level of the dehazing module group in the encoder. This concatenation is then input into the first dehazing module group of the decoder for decoding. Afterward, upsampling and dehazing module group operations are performed alternately to finally obtain the decoded features.

[0045] By using skip connections to fuse features at the same level in the encoder and decoder, information loss is effectively prevented and the ability to recover details is improved.

[0046] S33: The decoded features are added to the corresponding elements of the hazy RGB image to obtain the predicted dehazed image. In other words, the final dehazed image is obtained by adding the repair term (decoded features) output by the decoder to the initial input.

[0047] S4: Input the predicted dehazed image and sparse depth map back into the depth completion network to output the final predicted depth map. Compared to step S2 (directly using the hazy image), this step uses the dehazed image as input, which has clearer texture and more accurate edge information, thereby improving the accuracy of the depth completion network.

[0048] S5: Based on the dehazing error between the real dehazed image and the predicted dehazed image, obtain the dehazing difference mapping matrix. Based on the depth error between the final predicted depth map and the real depth map, obtain the depth difference mapping matrix.

[0049] Specifically, the dehazing error and depth error are processed by the difference mapping module (DM) to obtain the dehazing difference mapping matrix and the depth difference mapping matrix, respectively.

[0050] like Figure 5 As shown, the difference mapping module includes depthwise separable convolution (DWConv), multilayer perceptron (MLP), and sigmoid activation function. Specific steps include: The input features are extracted locally using DWConv, a step that efficiently captures spatial features while reducing computation. The extracted features are then processed through an MLP and a sigmoid activation function to generate a difference mapping matrix.

[0051] Furthermore, the dehazing error between the actual dehazed image and the predicted dehazed image is: ; in, R j This represents the dehazing error between the actual dehazed image and the predicted dehazed image. This represents the predicted dehazed image; J gt This represents a true dehazed image.

[0052] R j This reflects the difference between the predicted dehazed image and the actual dehazed image, indirectly indicating the locations of areas where the dehazing effect is unsatisfactory. If the depth completion network can focus on these areas and output accurate depth estimates, it can provide more precise depth information for image dehazing during the dual-task interaction process, thereby assisting the dehazing network in restoring a clear and ideal image.

[0053] The final depth error between the predicted depth map and the true depth map is: ; in, R d The final depth error between the predicted depth map and the true depth map; This represents the final predicted depth map; D gt This represents the true depth map.

[0054] R dThis reflects the difference between the final predicted depth map and the true depth map, indirectly indicating the location of areas where depth completion is not ideal. If image dehazing can focus on these areas and output more accurate and clear image information within them, it can provide more precise image guidance for depth completion during the dual-task interaction process, assisting the depth completion network in obtaining accurate depth information.

[0055] S6: Construct a dehazing loss function based on the dehazing difference mapping matrix and the depth error.

[0056] Specifically, the dehazing loss function is: ; in, This represents the dehazing loss function. A e This represents the depth difference mapping matrix.

[0057] Differences R j The input to the difference mapping module yields the dehazing difference mapping matrix between the predicted dehazed image and the clear image. Its physical meaning is to indicate areas where dehazing is insufficient (such as residual haze or loss of detail). By applying the dehazing difference mapping matrix to the depth loss, the deep network is forced to improve the estimation accuracy in these areas, thereby providing a more accurate depth prior for the dehazing network.

[0058] The depth loss function is: ; in, Represents the depth loss function. A d This represents the dehazing difference mapping matrix.

[0059] Differences R d The input to the difference mapping module yields a depth difference mapping matrix, which physically indicates areas with large depth prediction errors. By applying the depth difference mapping matrix to the dehazing loss, the dehazing network is forced to improve its dehazing capabilities in these areas, thereby providing more accurate image information for depth completion.

[0060] S7: By minimizing the dehazing loss function and the depth loss function, and using the backpropagation algorithm to update the parameters of the dehazing network and the depth completion network, the dehazing network and the depth completion network are iteratively optimized, and finally the dehazing model and the depth completion model are trained.

[0061] S8: Input the hazy RGB image to be dehazed and the corresponding sparse depth map into the dehazing model and the depth completion model, and output the dehazed RGB image and the corresponding depth map.

[0062] Specifically, first, the hazy RGB image and its corresponding sparse depth map are input into the depth completion model to obtain the initial predicted depth map; then, the hazy RGB image and the initial predicted depth map are input into the dehazing model to obtain the predicted dehazed image. Finally, the predicted dehazed image and the sparse depth map are input into the depth completion model again to obtain the final predicted depth map.

[0063] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image dehazing method based on a dual-task cooperative network, characterized in that, include: S1: Acquire a foggy RGB image and its corresponding sparse depth map; S2: Construct a depth completion network, input the fogged RGB image and the sparse depth map into the depth completion network, and output the initial predicted depth map; S3: Construct a dehazing network by inputting the hazy RGB image and the initial predicted depth map, and outputting the predicted dehazed image; S4: Input the predicted dehazed image and the sparse depth map back into the depth completion network to output the final predicted depth map; S5: Based on the dehazing error between the real dehazing image and the predicted dehazing image, obtain the dehazing difference mapping matrix; Based on the depth error between the final predicted depth map and the true depth map, a depth difference mapping matrix is ​​obtained; S6: Construct a dehazing loss function based on the dehazing difference mapping matrix and the depth error; A depth loss function is constructed based on the depth difference mapping matrix and the dehazing error; S7: By minimizing the dehazing loss function and the depth loss function, the dehazing model and the depth completion model are finally trained. S8: Input the hazy RGB image to be dehazed and the corresponding sparse depth map into the dehazing model and the depth completion model, and output the dehazed RGB image and the corresponding depth map.

2. The image dehazing method based on a dual-task cooperative network according to claim 1, characterized in that, In step S1, a foggy RGB image is acquired by a camera; a point cloud is acquired by a lidar and converted into a corresponding sparse depth image.

3. The image dehazing method based on a dual-task cooperative network according to claim 1, characterized in that, The dehazing network adopts a U-Net structure, including an encoder and a decoder; both the encoder and the decoder include multiple dehazing module groups, and each dehazing module group is composed of multiple stacked dehazing modules; the dehazing module group performs residual processing and attention processing on the input features.

4. The image dehazing method based on a dual-task cooperative network according to claim 3, characterized in that, Step S3 includes the following steps: S31: In the encoder, the foggy RGB image is processed by the first dehazing module group and then concatenated with the initial predicted depth map. The concatenated features are then passed through subsequent dehazing module groups to complete feature encoding and obtain coded features. S32: In the decoder, multiple dehazing module groups combine the output features of the corresponding level of the dehazing module group in the encoder to complete feature decoding and obtain decoded features; S33: The decoded features are added to the corresponding elements of the hazy RGB image to obtain the predicted dehazed image.

5. The image dehazing method based on a dual-task cooperative network according to claim 4, characterized in that, Step S31 includes: alternating downsampling and dehazing module group operations on the concatenated features to obtain encoded features; Step S32 includes: upsampling the encoded features and concatenating them with the output features of the dehazing module group at the corresponding level in the encoder, inputting them into the first dehazing module group of the decoder for decoding, and then alternating between upsampling and the dehazing module group operation to finally obtain the decoded features.

6. The image dehazing method based on a dual-task cooperative network according to claim 1, characterized in that, The dehazing error between the actual dehazed image and the predicted dehazed image is: ; The depth error between the final predicted depth map and the true depth map is: ； in, R j This represents the dehazing error between the actual dehazed image and the predicted dehazed image; This represents the predicted dehazed image; J gt Represents a true dehazed image; R d The depth error between the final predicted depth map and the true depth map; This represents the final predicted depth map; D gt This represents the true depth map.

7. The image dehazing method based on a dual-task cooperative network according to claim 6, characterized in that, The defogging loss function is: ； The depth loss function is: ； in, Represents the dehazing loss function. A e Represents the depth difference mapping matrix; Represents the depth loss function. A d This represents the dehazing difference mapping matrix.

8. The image dehazing method based on a dual-task cooperative network according to claim 1, characterized in that, In step S5, the dehazing error and the depth error are respectively processed by the difference mapping module to obtain the dehazing difference mapping matrix and the depth difference mapping matrix.

9. The image dehazing method based on a dual-task cooperative network according to claim 8, characterized in that, The difference mapping module includes a depthwise separable convolution, a multilayer perceptron, and a sigmoid activation function; the input features are extracted locally through the depthwise separable convolution, and the extracted features are sequentially passed through the multilayer perceptron and the sigmoid activation function to generate a difference mapping matrix.

Images