Image defogging method and system for target detection driving joint optimization learning

Abstract

The invention discloses an image defogging method and system for target detection driving joint optimization learning, and relates to the technical field of artificial intelligence image processing and machine vision. According to the method, the detection guide information is integrated into the defogging network architecture, so that the defogging network architecture generates features beneficial to detection of clearness and further improves the target detection performance, meanwhile, the target detection network can provide valuable information and guide the defogging learning process in turn, and the defogging efficiency is improved. The low-level tasks and the high-level tasks can jointly promote each other and achieve mutual gain, in addition, the model utilizes priori knowledge based on physics to enhance feature extraction and representation capabilities of a defogging network architecture, and a detection guide integration module fully utilizes detection information, so that the detection efficiency is improved. Finally, experiments prove that the 2DNet not only can be satisfactory in the visual quality of the defogging result, but also shows the superiority in improving the target detection performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and machine vision in artificial intelligence, and specifically to an image dehazing method and system driven by object detection and jointly optimized for learning. Background Art

[0002] Haze is a common atmospheric condition caused by light scattering from various suspended particles such as dust, smoke, and moisture. This scattering effect significantly reduces the quality of images captured under such conditions, leading to problems such as reduced contrast, color fading, and loss of sharpness. This phenomenon not only impairs the aesthetic value of photos but also poses a great challenge to vision processing systems, especially in tasks involving precision and accuracy, such as image recognition and analysis. Therefore, the field of image dehazing has become an important research area in computer vision and imaging science.

[0003] Although general object detection algorithms have become mature on standard datasets, the detection tasks under adverse weather conditions, especially in foggy environments, still face severe challenges. Existing research mainly focuses on three technical routes: some methods directly train detection models on degraded images, but limited by the feature degradation caused by haze occlusion and light attenuation, it is difficult for the model to capture the discriminative contours and texture information of the target, and problems such as missed detection and localization deviation are likely to occur in actual scenarios.

[0004] To alleviate the above problems, a cascaded processing flow of "restore first and then detect" is adopted, and the quality of the input image is improved through traditional or deep learning dehazing algorithms. However, a large number of experiments show that the correlation between image quality metrics such as PSNR and SSIM and the detection accuracy (mAP) is weak, and some restoration algorithms even interfere with the feature extraction process of the detection network due to over-enhancing high-frequency noise.

[0005] To address the bottleneck problem of the disconnection between image restoration and object detection tasks in the traditional cascade architecture, the joint optimization method based on task collaboration has gradually become the research focus. Such methods achieve cross-task interaction through sharing network structures and joint training mechanisms. For example, Liu et al. proposed IA-YOLO, which designed a dynamic enhancement unit for fog concentration perception to adaptively adjust the feature enhancement intensity according to the input image; Huang et al. further constructed DSNet with a dual-branch architecture, embedding a differentiable physical scattering model in the visibility enhancement branch and sharing multi-scale feature maps with the detection branch. At the level of the task interaction mechanism, Wang et al. proposed TogetherNet, which achieved feature disentanglement and weight allocation for the restoration and detection tasks through a cross-layer attention mechanism; Lee et al. further proposed a detection confidence feedback mechanism to reversely guide the parameter update direction of the restoration network in real time through the performance of the downstream task. In addition, Kalwar et al. proposed GDIP, which innovatively proposed a gated differentiable image processing block to fuse multiple preprocessing techniques through gated weights and designed a regularized training variant to improve the model's generalization ability for synthetic data while maintaining the efficiency in the inference stage, providing a lightweight solution for task collaboration in complex scenarios. Although the above methods have made certain progress, however, there are inherent conflicts between the optimization objectives of image restoration and object detection. For example, the restoration task focuses on global texture reconstruction while the detection task relies on local semantic discrimination, resulting in the joint training being prone to falling into suboptimal solutions.

[0006] Therefore, a new solution needs to be proposed for the above problems. Summary of the Invention

[0007] The purpose of the present invention is to provide a fog removal method and system for object detection-driven joint optimization learning, which can quickly and effectively remove haze from hazy images and complete object detection after haze removal by designing and training a model through deep learning methods, so as to solve the technical problems proposed in the background art.

[0008] To achieve the above purpose, the present invention provides the following technical solution: A fog removal method for object detection-driven joint optimization learning, at least including the following steps:

[0009] S1: Feed the images and corresponding detection labels in the publicly available haze detection dataset into a publicly available object detector to generate initial detection guidance information, and use VOC2012-FOG and Foggy Driving as the synthetic haze datasets required in the training stage and test stage of the present invention;

[0010] S2: Design an image dehazing model with joint multi-task optimization, namely 2DNet. The 2DNet includes a dehazing network architecture, a detection network architecture, and a detection-guided integration module. The 2DNet aims to jointly optimize the dehazing task and the detection task by leveraging the guidance information of the detection prediction results. The input of the 2DNet is a hazy image.

[0011] S3: Design a corresponding total loss function for learning.

[0012] S4: Feed the hazy detection data training set images for model training and the corresponding detection guidance information generated by S1 into this model for feature fusion, and detect the results through an object detector.

[0013] S5: After the model training is completed, use the RGB images and detection labels of the publicly available hazy detection dataset Foggy Driving and adopt publicly available evaluation metrics to verify the effectiveness of this model.

[0014] S6: After verifying the effectiveness of the dehazing model and the detection model, deploy the two pre-trained models to the RV1126 development board to realize the development of the system.

[0015] Furthermore, the specific method for generating the detection guidance information in S1 is as follows: Adopt an open-source Yolov7-tiny model, feed all the RGB images and detection labels in the publicly available hazy dataset into this network to obtain the detection regions and categories, and calculate the detection guidance information through a formula.

[0016] Furthermore, the dehazing network architecture establishes a U-Net architecture with residual connections. The dehazing network architecture consists of three downsamplings and three upsamplings, so as to effectively extract multi-scale feature information.

[0017] To effectively improve the dehazing ability of the dehazing network architecture while maintaining light weight, a hierarchical physical enhancement module HPEM is designed.

[0018] In the first layer, first, the input features are normalized through a BatchNorm layer to enhance the consistency of the feature distribution.

[0019] Then the feature stream enters two processing branches with different structures in parallel: The first branch samples the cascaded structure of a pointwise convolutional layer and a depthwise separable convolutional layer to extract spatial context features while ensuring parameter efficiency. Since the gating mechanism helps to improve the expression ability of the network, the other branch uses a pointwise convolutional layer combined with a Sigmoid function as the gating signal for the features. The outputs of the two branches are calculated through a dot product and projected in dimension through another pointwise convolutional layer.

[0020] The final output is connected with the input features through a residual connection, which enhances the feature expression ability while ensuring the stability of gradient propagation.

[0021] In the second layer, first, the input features still pass through a BatchNorm layer. To more effectively guide the learning direction of the dehazing model and deepen its understanding of the haze characteristics, a physical perception double-branch module PDB is introduced in the feature space. Based on the physical constraints of the atmospheric scattering model, it is re-derived as a feature space mapping relationship, as shown in the following formula:

[0022]

[0023] where and respectively represent the estimated atmospheric light and transmittance in the feature space, F in corresponds to the input features, and F out corresponds to the output features;

[0024] Different from the existing methods that directly estimate physical parameters in the image space, PDB innovatively realizes feature-level physical modeling through a double-branch structure: for the global uniformity characteristic of atmospheric light, the atmospheric light estimation branch uses global average pooling to compress the spatial dimension and generates a channel attention-style global representation through pointwise convolution; for the spatial non-uniform characteristic of transmittance, the transmittance estimation branch adopts a channel expansion strategy, uses 1×1 convolution to increase the number of channels by 4 times, and connects with 3×3 depthwise separable convolution, and uses the Gaussian error linear unit GELU activation to capture local degradation features;

[0025] After obtaining the estimated and the output features are calculated according to the atmospheric scattering model formula; through the PDB module, F in generates interpretable features from the input features for restoring haze images, which can enhance the interpretability of the dehazing network architecture and at the same time align with the generation process of haze images, thus contributing to haze removal.

[0026] Furthermore, as a typical single-stage detection framework based on regression, the detection network architecture preferentially uses YOLOv7-tiny as the baseline detector under the requirement of balancing computational efficiency and model performance, and realizes lightweight design by streamlining the network structure and channel dimension;

[0027] Compared with the computationally intensive large-scale detection models, the lightweight architecture is more conducive to meeting the engineering requirements of real-time inference while ensuring the detection accuracy;

[0028] At the algorithm implementation level, the object detection network in the detection network architecture adopts a two-stage training strategy:

[0029] Before training for the image dehazing task, the object detection network YOLOv7-tiny has been pre-trained on a clear image dataset, enabling it to have accurate object classification and bounding box regression capabilities. When the original image is input, it can output detection results containing the target category and its spatial coordinates.

[0030] During the subsequent training process of the image dehazing model, the parameters of the object detection network are always kept frozen. This design not only effectively avoids the gradient conflict caused by joint multi-task training but also ensures the stability of the dehazing module training.

[0031] Furthermore, the goal of the detection-guided integration module is to restore an image that has both good visual quality and is beneficial for detection. The key of the detection-guided integration module lies in narrowing the target gap between the dehazing task and the detection task, comprehensively using the pre-trained detector to provide valuable guidance for 2DNet, and combining the network structure and learning objectives.

[0032] To achieve detection guidance, the detection-guided integration module mainly performs three key operations, namely, building a detection-guided fusion network for the application of detection-guided fusion, building a detection-guided attention module for the application of detection-guided attention, and building a detection loss. Under the comprehensive consideration of the network structure and learning objectives, they jointly promote the integration of the dehazing effect that is beneficial for detection; the detection-guided attention module and the detection-guided fusion network are used to integrate detection information into the network structure, while the detection loss L det helps with the optimization process.

[0033] Recognizing the limitations of the initial detection results generated by YOLOv7-tiny, the potential valuable information contained therein was explored. This information can serve as effective prior knowledge, indicating more effective patterns and helping the dehazing process to pay more attention to the regions containing the objects of interest. The aim is to enrich the high-level understanding ability of the scene during the training process of the dehazing model. To implement this strategy, a detection-guided image, denoted as g, was first generated as an auxiliary input to the dehazing network architecture, similar to a mask image.

[0034] Initially, g was set as a completely zero image, and the value of g is determined by the following formula:

[0035]

[0036] where (x, y) are pixel coordinates; b is the predicted bounding box, and in the region within the bounding box, the value is set to the product of the class ID plus 1 and the score; this assignment method distinguishes the regions with and without objects and the differences between different objects, thus providing valuable guidance information for the dehazing network architecture during the training process.

[0037] To make full use of the guiding information of detection prediction, a detection-guidance fusion network (DGFM) and a detection-guidance attention module (DGAM) are constructed. DGFM is used for shallow feature maps, and DGAM is used for deep feature maps, ensuring guidance from coarse-grained to fine-grained respectively;

[0038] For DGFM, multi-scale convolutional kernels are used to process the input features in parallel to extract potential feature information at different scales. To maintain the resolution consistency between the original features and the guiding features, after parallel convolution processing, the features are fused, and then subsequent convolution and non-linear activation operations are performed. Through this multi-scale convolution design, different-scale information in the shallow features can be effectively captured, enhancing the directivity and propagation efficiency of the shallow features;

[0039] For DGAM, the core is to explicitly model the mutual relationship between two feature maps using the cross-attention mechanism; through the action of cross-attention, complementary information between different feature maps is extracted and further optimized through convolution and pooling operations; in DGAM, feature maps f and g are mutually fused through the attention maps generated by pointwise convolution and pooling operations respectively, thereby improving the fine-grainedness and accuracy of feature expression; finally, by applying this attention information to the deep features, high-precision feature extraction at the pixel level is optimized;

[0040] In addition to providing guidance for the network, a detection loss L det is introduced to help optimize the image dehazing process. The imposition of the detection loss endows the dehazing network architecture with connectivity, encouraging it to optimize in a direction more conducive to detection.

[0041] Furthermore, the total loss function in S3 is defined as follows:

[0042] L total = L res + λL det

[0043] where L res represents the loss function of the image restoration task; L det represents the loss function of the object detection task, and λ is a trade-off hyperparameter used to balance the importance of these two tasks. According to the ablation study of the loss weights, λ is set to 0.4 to obtain the best performance;

[0044] In the image dehazing task, the mean absolute error (MAE) loss is used to obtain the estimated clean image;

[0045] In contrast, the mean square error (MSE) loss tends to weaken the semantic gradient information, resulting in performance degradation;

[0046] The loss function L for the image restoration task res is expressed as:

[0047]

[0048] where N represents the total number of training samples, J and represent the clear image and the estimated restored image respectively;

[0049] To promote the dehazing optimization in a direction more conducive to detection, the detection loss of the original YOLOv7-tiny is adopted as L det , and its expression is:

[0050] L det = λ box L box + λ obj L obj + λ cls L cls

[0051] where L box , L obj and L cls represent the localization loss, confidence loss, and classification loss respectively, and the corresponding weight coefficients are kept consistent with the original configuration;

[0052] This design of the loss function not only optimizes the image restoration task but also ensures that the dehazing network can be optimized in a direction conducive to object detection, thus achieving a dual improvement in image quality and detection ability.

[0053] An object detection system in a haze environment, for an image dehazing system for object detection-driven joint optimization learning according to any one of claims 1-6 above, includes a system hardware architecture and system software functions;

[0054] The system hardware architecture includes an embedded edge computing server and a remote client collaborative design module. Among them, the embedded edge computing server serves as an edge computing unit to form the core hardware body. The embedded edge computing server includes four parts: the RV1126 embedded platform, an Ethernet interface, a vision acquisition module, and a power supply. The remote client collaborative design module is configured with a standard networked PC terminal to achieve remote access;

[0055] The system software function visualization module displays the dehazed image and the detected target information. Users can view the dehazing effect of the image and the target detection result through the interface.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] By integrating detection guidance information into the dehazing network architecture, the present invention enables the dehazing network architecture to generate clear features beneficial for detection, thereby improving the performance of object detection. At the same time, the object detection network can provide valuable information and in turn guide the learning process of dehazing, indicating that low-level tasks and high-level tasks can promote each other and achieve mutual enhancement. In addition, the model utilizes physics-based prior knowledge to enhance the feature extraction and representation capabilities of the dehazing network architecture, and the detection guidance integration module makes full use of detection information. Finally, experiments prove that 2DNet not only has a satisfactory visual quality of dehazing results, but also demonstrates its superiority in improving object detection performance. At the same time, RKNN is used to optimize the dehazing and detection models and complete the construction of a post-dehazing detection system based on an embedded platform. Experimental verification shows that the system can simply implement functions such as the acquisition of foggy video images, image dehazing, and object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0059] Figure 1 The 2DNet framework for detection task-driven image dehazing designed by the present invention;

[0060] Figure 2 The structural diagram of the dehazing network architecture provided by the present invention;

[0061] Figure 3 The structural diagram of the hierarchical physical enhancement module provided by the present invention;

[0062] Figure 4 The physical perception double-branch module diagram provided by the present invention;

[0063] Figure 5 The schematic diagram of the DGFM network structure provided by the present invention;

[0064] Figure 6 The schematic diagram of the DGAM network structure provided by the present invention;

[0065] Figure 7 The comparison experiment results of dehazing effect and object detection on VOCFog provided by the present invention;

[0066] Figure 8 The comparison experiment results of dehazing effect and object detection on Foggy Driving provided by the present invention;

[0067] Figure 9 The ablation experiment results provided by the present invention;

[0068] Figure 10 The overall functional architecture diagram of the system provided by the present invention

[0069] Figure 11 The wiring diagram of RV1126 provided by the present invention;

[0070] Figure 12 The system functional flowchart provided by the present invention;

[0071] Figure 13 The schematic diagram of the system main interface provided by the present invention;

[0072] Figure 14 The schematic diagram of the selected model file provided by the present invention;

[0073] Figure 15 The schematic diagram of the system displaying the input image provided by the present invention;

[0074] Figure 16 The schematic diagram of the system displaying the detection result provided by the present invention;

[0075] Figure 17 The schematic diagram of the system saving the defogging result provided by the present invention. Detailed implementation manners

[0076] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0077] The method of the present invention uses detection information and task-driven learning to better guide the defogging process, thereby improving the accuracy of downstream detection tasks. Specifically, on the one hand, an attention mask is generated through the detection result to guide the defogging network architecture to focus on the key areas sensitive to the detection task; on the other hand, the detection loss is backpropagated to the defogging network architecture to achieve task-driven parameter optimization. At the same time, in order to make full use of the detection information, a detection-guided attention module DGAM and a detection-guided fusion network DGFM are designed to provide guiding information for the defogging network architecture at the coarse-grained and fine-grained levels.

[0078] An image defogging method for target detection-driven joint optimization learning includes at least the following steps:

[0079] S1: Input the images and corresponding detection labels in the publicly available haze detection dataset into a publicly available target detector to generate initial detection guidance information, and use VOC2012-FOG and Foggy Driving as the synthetic haze datasets required in the training stage and testing stage of the present invention;

[0080] S2: Design an image dehazing model with joint optimization of multiple tasks, namely 2DNet (refer to Figure 1 ), 2DNet includes a dehazing network architecture, a detection network architecture, and a detection-guided integration module. 2DNet aims to jointly optimize the dehazing task and the detection task by using the guidance information of the detection prediction results. The input of 2DNet is a hazy image

[0081] S3: Design a corresponding total loss function for learning;

[0082] S4: Feed the hazy detection data training set images for model training and the detection guidance information generated by S1 corresponding to the training set into this model, perform feature fusion, and detect the results through the object detector. In this embodiment, the required hazy training set contains 12,336 samples of VOC-Fog;

[0083] S5: After the model training is completed, use the RGB images and detection labels of the public hazy detection dataset Foggy Driving, and adopt the public evaluation metrics to verify the effectiveness of this model;

[0084] The specific method for verifying the effectiveness of the model in S5 is as follows: Feed the hazy RGB images and their detection labels in the public dataset into the model. The haze-free image generated by the feature recovery encoder is the first prediction result of the model. Then, send the prediction result into the object detection network for the detection task. Use three public evaluation metrics to verify the effectiveness of the model prediction results. The following are the evaluation metrics that need to be used:

[0085] Peak signal-to-noise ratio (PSNR) is used to measure the difference between the reconstructed image and the original image. The higher its value, the better the reconstruction quality. Its mathematical expression is:

[0086]

[0087] Structural similarity index (SSIM): SSIM evaluates the brightness, contrast, and structural similarity of an image. Its value ranges between 0 and 1, and the value closer to 1 indicates higher similarity. Its expression is:

[0088]

[0089] Mean average precision (mAP): It is used to comprehensively consider the accuracy and recall rate of the model in different categories. The higher its value, the higher the detection accuracy. Its expression is:

[0090]

[0091] S6: After verifying the effectiveness of the defogging model and the detection model, the two pre-trained models are deployed on the RV1126 development board to realize the development of the system.

[0092] The specific method for generating detection guidance information in S1 is as follows: Using an open-source Yolov7-tiny model, all RGB images and detection labels in the public haze dataset are fed into the network to obtain the detection regions and categories, and the detection guidance information is calculated through formulas.

[0093] The defogging network architecture builds a U-Net architecture with residual connections (refer to Figure 2 ), and the defogging network architecture consists of three downsamplings and three upsamplings, so as to effectively extract multi-scale feature information;

[0094] To effectively improve the defogging ability of the defogging network architecture while maintaining light weight, a hierarchical physical enhancement module HPEM is designed, as shown in Figure 3 ;

[0095] In the first layer, first, the input features are normalized by a BatchNorm layer to enhance the consistency of the feature distribution;

[0096] Then the feature stream enters two processing branches with different structures in parallel: the first branch samples the cascade structure of the pointwise convolution layer and the depthwise separable convolution layer to extract spatial context features on the premise of ensuring parameter efficiency. Since the gating mechanism helps to improve the expression ability of the network, the other branch uses a pointwise convolution layer combined with the Sigmoid function as the gating signal of the feature. The outputs of the two branches are calculated by dot product and projected in dimension through another pointwise convolution layer;

[0097] Finally, the output is connected with the input features by residual connection to ensure the stability of gradient propagation while enhancing the feature expression ability;

[0098] In the second layer, first, the input features still pass through a BatchNorm layer. To more effectively guide the learning direction of the defogging model and deepen its understanding of the haze characteristics, a physical perception double-branch module PDB is introduced in the feature space, as shown in Figure 4 ; Based on the physical constraints of the atmospheric scattering model, it is re-derived as the feature space mapping relationship, refer to the following formula:

[0099]

[0100] Among them, and respectively represent the atmospheric light and transmittance estimated in the feature space, F in corresponds to the input features, and F out corresponds to the output features;

[0101] Different from directly estimating physical parameters in the image space by existing methods, PDB innovatively realizes feature-level physical modeling through a two-branch architecture: for the global uniformity characteristic of atmospheric light, the atmospheric light estimation branch adopts global average pooling to compress the spatial dimension and generates a channel attention-based global representation through pointwise convolution; for the spatial non-uniform characteristic of transmittance, the transmittance estimation branch adopts a channel expansion strategy, uses 1×1 convolution to increase the number of channels by 4 times, and connects 3×3 depthwise separable convolution, and uses the Gaussian error linear unit GELU activation to capture local degradation features;

[0102] After obtaining the estimated and output features are calculated according to the atmospheric scattering model formula; through the PDB module, F in generates interpretable features from the input features for restoring hazy images, which can enhance the interpretability of the dehazing network architecture and at the same time align with the generation process of hazy images, thus contributing to haze removal.

[0103] As a typical single-stage detection framework based on regression, the YOLO series shows significant advantages in many visual scenarios with its excellent detection performance and engineering ease of use. Under the requirement of balancing computational efficiency and model performance, YOLOv7-tiny is preferentially used as the baseline detector, and lightweight design is achieved by streamlining the network structure and channel dimension;

[0104] Compared with computationally intensive large-scale detection models, lightweight architectures are more conducive to meeting the engineering requirements of real-time inference while ensuring detection accuracy;

[0105] At the algorithm implementation level, the object detection network in the detection network architecture adopts a two-stage training strategy:

[0106] First, before the training of the image dehazing task, the object detection network YOLOv7-tiny has been pre-trained on a clear image dataset, enabling it to have accurate object classification and bounding box regression capabilities, and the input of the original image can output detection results including the target category and its spatial coordinates;

[0107] During the subsequent training process of the image dehazing model, the parameters of the object detection network always remain frozen. This design not only effectively avoids the gradient conflict caused by joint multi-task training but also ensures the stability of the dehazing module training.

[0108] The goal of the detection-guided integration module is to recover an image that not only has good visual quality but also is beneficial for detection. The key of the detection-guided integration module lies in narrowing the objective gap between the dehazing task and the detection task, comprehensively using the pre-trained detector to provide valuable guidance for 2DNet, and combining the network structure and learning objective;

[0109] To achieve detection guidance, the detection-guided integration module mainly performs three key operations, namely, building a detection-guided fusion network for the application of detection-guided fusion, building a detection-guided attention module for the application of detection-guided attention, and building a detection loss. Under the comprehensive consideration of the network structure and learning objective, they jointly promote the integration of the dehazing effect that is beneficial for detection; The detection-guided attention module and the detection-guided fusion network are used to integrate detection information into the network structure, while the detection loss L det helps to optimize the process;

[0110] Recognizing the limitations of the initial detection results generated by YOLOv7-tiny, the potential valuable information contained therein was explored. This information can serve as effective prior knowledge to indicate more effective patterns and help the dehazing process pay more attention to the regions containing the objects of interest. The purpose is to enrich the high-level understanding ability of the scene during the training process of the dehazing model. To implement this strategy, a detection-guided image, denoted as g, was first generated as an auxiliary input to the dehazing network architecture, similar to a mask image;

[0111] Initially, g was set as a all-zero image, and the value of g is determined by the following formula:

[0112]

[0113] where (x, y) is the pixel coordinate; b is the predicted bounding box, and in the region within the bounding box, the value is set to the product of the class ID plus 1 and the score (the class ID starts from 0); This assignment method distinguishes the regions with and without objects and the differences between different objects, thus providing valuable guiding information for the dehazing network architecture during the training process;

[0114] To make full use of the guiding information of the detection prediction, a detection-guided fusion network and a detection-guided attention module were built. The detection-guided fusion network is DGFM, and the detection-guided attention module is DGAM. DGFM is used for the shallow feature map, while DGAM is used for the deep feature map, ensuring guidance from coarse-grained to fine-grained respectively;

[0115] For DGFM, as Figure 5As shown, multi-scale convolutional kernels are used to process the input features in parallel to extract potential feature information at different scales. To maintain the resolution consistency between the original features and the guiding features, feature fusion is performed after parallel convolution processing, followed by subsequent convolution and non-linear activation operations. Through this multi-scale convolution design, different scale information in the shallow features can be effectively captured, enhancing the directionality and propagation efficiency of the shallow features;

[0116] For DGAM, as Figure 6 shown, the core is to use the cross-attention mechanism to explicitly model the mutual relationship between two feature maps; through the action of cross-attention, complementary information between different feature maps is extracted and further optimized through convolution and pooling operations; in DGAM, the feature maps f and g are mutually fused through the attention maps generated by pointwise convolution and pooling operations respectively, thereby improving the fine-grainedness and accuracy of feature expression; finally, by applying this attention information to the deep features, high-precision feature extraction at the pixel level is optimized;

[0117] In addition to providing guidance for the network, a detection loss L det is introduced to help optimize the image dehazing process. The imposition of the detection loss endows the dehazing network architecture with connectivity, encouraging its optimization towards a direction more conducive to detection.

[0118] The total loss function in S3 is defined as follows:

[0119] L total = L res + λL det

[0120] where L res represents the loss function of the image restoration task; L det represents the loss function of the object detection task, and λ is a trade-off hyperparameter used to balance the importance of these two tasks. According to the ablation study of the loss weights, λ is set to 0.4 to obtain the best performance;

[0121] In the image dehazing task, the mean absolute error (MAE) loss is used to obtain the estimated clean image;

[0122] In contrast, the mean squared error (MSE) loss tends to weaken the semantic gradient information, resulting in performance degradation, which has been verified in our experiments;

[0123] The loss function L res of the image restoration task is expressed as:

[0124]

[0125] where N represents the total number of training samples, J and respectively represent the clear image and the estimated restored image;

[0126] To promote the dehazing optimization in a direction more favorable for detection, the detection loss of the original YOLOv7-tiny is adopted as L det , and its expression is:

[0127] L det = λ box L box + λ obj L obj + λ cls L cls

[0128] where L box , L obj and L cls respectively represent the localization loss, the confidence loss, and the classification loss, and the corresponding weight coefficients are kept consistent with the original configuration;

[0129] This design of the loss function not only optimizes the image restoration task but also ensures that the dehazing network can be optimized in a direction favorable for object detection, thus achieving a double improvement in image quality and detection ability.

[0130] Example 2:

[0131] Based on the proposed image dehazing system with object detection-driven joint optimization learning.

[0132] The 2DNet model lays the foundation for achieving efficient object detection in complex haze environments. In practical application scenarios such as outdoor security, haze interference will significantly reduce the visual quality captured by monitoring devices and affect the monitoring targets. It is necessary to improve the clarity of images through image dehazing technology to meet the requirements of the monitoring system for high-visual-quality images. For this reason, this chapter constructs a complete embedded object detection system in a haze environment, which mainly adopts an embedded server-side and client-side architecture. The former mainly undertakes functions such as image acquisition and storage, image haze removal, and object detection, and the latter mainly realizes the remote viewing of the monitoring screen. The overall architecture of the system is as Figure 10 shown.

[0133] (a) System hardware architecture

[0134] The hardware architecture of this system mainly consists of a collaborative design mode of an embedded edge computing server and a remote client. Among them, the embedded server side serves as the edge computing unit to form the core hardware body, which is composed of four parts: the RV1126 embedded platform, the Ethernet interface, the vision acquisition module, and the power supply. The client is configured with a standard networked PC terminal to achieve remote access. The system operation process is as follows: After the server starts, it first performs a self-diagnosis of the status, checks whether the peripherals and the network are connected normally, and enters the working ready state after the self-check is completed. Then, the camera is turned on, and the pre-trained neural network model is loaded. After an image and video are input, the system will perform defogging processing on them and output the display, and perform object detection on the displayed image and synchronously overlay and present the detection frame. The client software will establish a TCP connection with the embedded server to obtain the monitoring screen of the current embedded end. The specific wiring is as Figure 11 shown. The power cord is at the upper right corner, the USB Type-C interface is at the upper left corner, which is connected to the PC for ADB debugging. The 3Pin row pin on the right is the UART debugging serial port, and the USB-TTL signal interaction is realized through the CH340G chip.

[0135] (b) System software functions

[0136] To facilitate users to view the system operation results in real time, a visualization module is designed to display the defogged image and the detected target information. Users can view the defogging effect of the image and the object detection results through the interface. The system flow chart is as Figure 12 shown.

[0137] PyQt is a Python library for creating graphical user interfaces (GUIs). It is developed by Riverbank Computing and is a Python encapsulation of the Qt framework. Qt is a cross-platform application development framework with rich graphical controls and functions. Therefore, in this chapter, a target detection system for haze environment based on PyQt is developed through the Python language. The main interface of the system is as Figure 13 shown.

[0138] The main interface of this system is constructed using the QMainWindow class, and the left-right split layout is realized through the QHBoxLayout class. The left control area is carried by a QWidget class container, and the QVBoxLayout class is used for vertical layering inside, including three groups of core function modules: The model management module uses QPushButton class controls to implement the selection and initialization of the defogging model and the detection model. The input source selection module uses four QPushButton class controls to correspond to four media input methods: pictures, videos, folders, and cameras respectively. The export function group uses the same class controls to realize the diversified output of the processing results. The specific function diagram is shown as Figures 14 - 17 shown.

[0139] The following verification is further proposed:

[0140] Table 1 Quantitative comparison of different algorithms for the VOC-FOG dataset

[0141]

[0142]

[0143] In Table 1, different dehazing methods and different detection results after dehazing are compared. Refer to Figure 7 , due to the data synthesis scheme of the present invention, the synthesized dataset is closer to the real distribution, and the concentration of haze is high and not evenly distributed. Therefore, for some independently optimized "dehaze first and then detect" methods, due to their overly simple structures, the dehazing effect is very limited, such as DehazeNet and AOD-Net. For other networks with certain dehazing capabilities, such as FFANet and DehazeFuse, the image quality is significantly improved, but the detection index of DehazeFuse only improves by 0.01 compared to the foggy image baseline, which indicates that there is a lack of a strong causal relationship between the improvement of image visual quality and detection accuracy. And other joint methods have all improved the accuracy of object detection to a certain extent. It is worth noting that 2DNet is weaker than the DehazeFuse network in terms of dehazing effect due to its lightweight design and lack of depth information assistance, but it shows the best performance in object detection tasks.

[0144] Table 2 Quantitative comparison of different algorithms for the Foggy Driving dataset

[0145]

[0146]

[0147] Refer to Figure 8 and Table 2, the comparison of the detection performance for the real haze dataset Foggy Driving shows that the necessity of task co-design in real scenarios is more significant. Among the separate methods, although the detection of DehazeFuze is slightly higher than the foggy image baseline, its performance is significantly limited in key categories such as Motorbike and Bus, which verifies the challenge of the domain difference between synthetic data and real scenarios to the generalization of traditional dehazing algorithms. And some separate methods such as DehazeNet and AOD-Net even lead to a significant deterioration in detection performance, highlighting the destructive impact of the complex distribution of real haze on independently optimized models. Among the joint methods, 2DNet significantly leads with an mAP of 38.72, especially achieving 2-4 times the detection accuracy of other methods in the Motorbike category, proving the ability of the lightweight collaborative framework to capture long-tail targets.

[0148] Table 3 Ablation Experiment Results

[0149]

[0150] To further verify the effectiveness of the method proposed in the present invention, extensive ablation studies were conducted. As shown in Table 3 and Figure 9 as follows, the roles of different components were analyzed. The key components of 2DNet were systematically removed, and several variants were established, including: (1) w / o DGFM: removing the Detection-Guided Fusion Module (DGFM); (2) w / o DGAM: removing the Detection-Guided Attention Module (DGAM); (3) MAE→MSE: replacing the recovery loss from Mean Absolute Error (MAE) with Mean Squared Error (MSE); (4) w / oL det ; removing the object detection loss L det .

[0151] In summary, the present invention addresses the problem that the improvement of the visual quality of the dehazed image may not necessarily lead to the enhancement of the detection performance, and proposes an image dehazing method and system 2DNet based on object detection-driven joint optimization learning. By integrating detection-guided information into the dehazing network architecture, this method enables the dehazing network architecture to generate clear features conducive to detection, thereby improving the object detection performance. At the same time, the object detection network can provide key important object information and, in turn, guide the dehazing learning process. It is proved that low-level tasks and high-level tasks can promote each other and achieve mutual benefits. In addition, the model uses physics-based prior knowledge to enhance the feature extraction and representation ability of the dehazing network architecture, and the detection-guided integration module makes full use of detection information. Finally, through comparative experiments, it is proved that 2DNet can not only achieve satisfactory visual quality in the dehazing results, but also demonstrate its superiority in improving object detection performance. At the same time, the lightweight design of 2DNet enables it to be efficiently deployed to the edge computing platform, realizing the engineering implementation of the object detection system in the haze environment, and achieving good dehazing and detection effects in the actual haze scene.

[0152] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed by the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. An image dehazing method driven by object detection and jointly optimized learning, characterized in that: At least include the following steps: S1: Feed the images and corresponding detection labels in the publicly available haze detection dataset into a publicly available object detector to generate initial detection guidance information, and use VOC2012-FOG and Foggy Driving as the synthetic haze datasets required in the training and testing phases of the present invention; S2: Design an image dehazing model with joint optimization of multiple tasks, namely 2DNet. The 2DNet includes a dehazing network architecture, a detection network architecture, and a detection-guided integration module. The 2DNet aims to jointly optimize the dehazing task and the detection task by using the guidance information of the detection prediction results. The input of the 2DNet is a hazy image S3: Design a corresponding total loss function for learning; S4: Feed the haze detection data training set images for model training and the detection guidance information generated by S1 corresponding to the training set into this model, perform feature fusion, and detect the results through the object detector; S5: After the model training is completed, use the RGB images and detection labels of the publicly available haze detection dataset Foggy Driving, and adopt publicly available evaluation metrics to verify the effectiveness of this model; S6: After verifying the effectiveness of the dehazing model and the detection model, deploy the two pre-trained models to the RV1126 development board to realize the development of the system.

2. The image dehazing method for target detection-driven joint optimization learning according to claim 1, characterized in that: The specific method for generating the detection guidance information in S1 is as follows: Adopt an open-source Yolov7-tiny model, feed all the RGB images and detection labels in the publicly available haze dataset into this network to obtain the detection regions and categories, and calculate the detection guidance information through formulas.

3. An image dehazing method for target detection-driven joint optimization learning according to claim 1, characterized in that: The dehazing network architecture establishes a U-Net architecture with residual connections, and the dehazing network architecture is composed of three downsamplings and three upsamplings, so as to effectively extract multi-scale feature information; In order to effectively improve the dehazing ability of the dehazing network architecture while maintaining light weight, a hierarchical physical enhancement module HPEM is designed; In the first layer, first, the input features are normalized through a BatchNorm layer to enhance the consistency of the feature distribution; Then the feature stream enters two processing branches with different structures in parallel: The first branch samples the cascaded structure of the pointwise convolution layer and the depthwise separable convolution layer to extract spatial context features on the premise of ensuring parameter efficiency. Since the gating mechanism helps to improve the expression ability of the network, the other branch uses the pointwise convolution layer combined with the Sigmoid function as the gating signal of the feature. The outputs of the two branches are calculated through dot product and projected in dimension through another pointwise convolution layer; Finally, the output is connected with the input features in a residual manner to ensure the stability of gradient propagation while enhancing the feature expression ability; In the second layer, first, the input features still pass through a BatchNorm layer. In order to more effectively guide the learning direction of the dehazing model and deepen its understanding of the haze characteristics, a physical perception double-branch module PDB is introduced in the feature space; Based on the physical constraints of the atmospheric scattering model, it is re-derived as the feature space mapping relationship, see the following formula: Among them, and respectively represent the atmospheric light and transmittance estimated by the feature space. F in corresponds to the input feature, and F out corresponds to the output feature. Different from directly estimating physical parameters in the image space by existing methods, PDB innovatively realizes feature-level physical modeling through a two-branch architecture: for the global uniformity characteristic of the atmospheric light, the atmospheric light estimation branch uses global average pooling to compress the spatial dimension and generates a channel attention-style global representation through pointwise convolution; for the spatial non-uniformity characteristic of the transmittance, the transmittance estimation branch adopts a channel expansion strategy, uses 1×1 convolution to increase the number of channels by 4 times, and connects 3×3 depthwise separable convolution, and uses the Gaussian error linear unit GELU activation to capture local degradation features; After obtaining the estimated and the output features are calculated according to the atmospheric scattering model formula; through the PDB module, F in generates interpretable features from the input features for restoring hazy images, which can enhance the interpretability of the dehazing network architecture and align with the generation process of hazy images, thus contributing to haze removal.

4. The image dehazing method for target detection-driven joint optimization learning according to claim 3, characterized in that: As a typical single-stage detection framework based on regression, the detection network architecture preferentially uses YOLOv7-tiny as the baseline detector under the requirement of balancing computational efficiency and model performance, and realizes lightweight design by streamlining the network structure and channel dimension; Compared with large-scale detection models that are computationally intensive, lightweight architectures are more conducive to meeting the engineering requirements of real-time inference while ensuring detection accuracy; At the algorithm implementation level, the object detection network in the detection network architecture adopts a two-stage training strategy: First, before the training of the image defogging task, the object detection network YOLOv7-tiny has been pre-trained on a clear image dataset, enabling it to have accurate object classification and bounding box regression capabilities, and the input of the original image can output detection results including the target category and its spatial coordinates; During the subsequent training process of the image defogging model, the parameters of the object detection network always remain frozen. This design not only effectively avoids gradient conflicts caused by joint multi-task training, but also ensures the stability of the defogging module training.

5. An image dehazing method for target detection-driven joint optimization learning according to claim 4, characterized in that: The goal of the detection-guided integration module is to restore an image that has both good visual quality and is conducive to detection. The key of the detection-guided integration module lies in narrowing the target gap between the defogging task and the detection task, and comprehensively using the pre-trained detector to provide valuable guidance for 2DNet, combining the network structure and learning objectives; To achieve detection guidance, the detection-guided integration module mainly uses three key operations, namely building a detection-guided fusion network for the application of detection-guided fusion, building a detection-guided attention module for the application of detection-guided attention, and building a detection loss. Under the comprehensive consideration of the network structure and learning objectives, it jointly promotes the integration of the defogging effect conducive to detection; The detection-guided attention module and the detection-guided fusion network are used to integrate detection information into the network structure, and the detection loss L det helps to optimize the process; Recognizing the limitations of the initial detection results generated by YOLOv7-tiny, the potential valuable information contained in them is explored. These information can be used as effective prior knowledge to indicate more effective patterns and help the defogging process pay more attention to the regions containing the objects of interest. The purpose is to enrich the high-level understanding ability of the scene during the training process of the defogging model. To achieve this strategy, a detection-guided image, denoted as g, is first generated as an auxiliary input to the defogging network architecture, similar to a mask image; Initially, g is set as a completely zero image, and the value of g is determined by the following formula: Among them, (x, y) are pixel coordinates; b is the predicted bounding box, and in the area within the bounding box, the value is set to the product of the class ID plus 1 and the score; this assignment method distinguishes the areas with and without objects and the differences between different objects, thus providing valuable guiding information for the dehazing network architecture during the training process; To make full use of the guiding information detected and predicted, a detection-guided fusion network (DGFM) and a detection-guided attention module (DGAM) are built. The detection-guided fusion network is DGFM, and the detection-guided attention module is DGAM. DGFM is used for shallow feature maps, while DGAM is used for deep feature maps, respectively ensuring guidance from coarse-grained to fine-grained; For DGFM, multi-scale convolutional kernels are used to process the input features in parallel to extract potential feature information at different scales. To maintain the resolution consistency between the original features and the guiding features, after parallel convolution processing, the features are fused, and then subsequent convolution and non-linear activation operations are performed. Through this multi-scale convolution design, different-scale information in the shallow features can be effectively captured, enhancing the directionality and propagation efficiency of the shallow features; For DGAM, the core is to explicitly model the mutual relationship between two feature maps using the cross-attention mechanism; through the action of cross-attention, complementary information between different feature maps is extracted and further optimized through convolution and pooling operations; in DGAM, the feature maps f and g are mutually fused through the attention maps generated by pointwise convolution and pooling operations respectively, thereby improving the fine-grainedness and accuracy of feature expression; finally, by applying this attention information to the deep features, the high-precision feature extraction at the pixel level is optimized; In addition to providing guidance for the network, a detection loss L is introduced det , to help optimize the image dehazing process. The application of the detection loss endows the dehazing network architecture with connectivity and encourages its optimization towards a more detection-friendly direction.

6. The image dehazing method for target detection-driven joint optimization learning according to claim 5, wherein: The total loss function in S3 is defined as follows: L total = L res + λL det Among them, L res represents the loss function of the image restoration task; L det represents the loss function of the object detection task. λ is a trade-off hyperparameter used to balance the importance of these two tasks. According to the ablation study of the loss weights, λ is set to 0.4 to obtain the best performance; In the image dehazing task, the mean absolute error (MAE) loss is used to obtain the estimated clean image; In contrast, the mean squared error (MSE) loss tends to weaken the semantic gradient information, resulting in performance degradation; The loss function L for the image restoration task res is expressed as: where N represents the total number of training samples, J and represent a clear image and an estimated restored image, respectively; To promote the dehazing optimization in a direction more conducive to detection, the detection loss of the original YOLOv7-tiny is adopted as L det , and its expression is: L det = λ box L box + λ obj L obj + λ cls L cls Among them, L box , L obj and L cls respectively represent the localization loss, confidence loss, and classification loss, and the corresponding weight coefficients remain consistent with the original configuration; This loss function design not only optimizes the image restoration task but also ensures that the dehazing network can be optimized in a direction beneficial to object detection, thus achieving a double improvement in image quality and detection ability.

7. A haze environment target detection system for an image haze removal system for target detection-driven joint optimization learning according to any one of the above claims 1-6, characterized in that: It includes the system hardware architecture and the system software functions; The system hardware architecture includes an embedded edge computing server and a remote client collaborative design module. Among them, the embedded edge computing server serves as the edge computing unit to form the core hardware body. The embedded edge computing server includes four parts: the RV1126 embedded platform, the Ethernet interface, the vision acquisition module, and the power supply. The remote client collaborative design module is configured with a standard networked PC terminal to achieve remote access; The system software function visualization module displays the dehazed image and the detected target information, and users can view the dehazing effect of the image and the target detection result through the interface.

Images