Multi-weather target detection method based on degraded category perception image restoration

Abstract

The invention discloses a multi-weather target detection method based on degraded category perception image restoration, and relates to the field of computer vision and artificial intelligence. The method comprises the following steps: (1) adopting a training normal form of restoration and detection combined learning, simultaneously carrying out restoration and detection decoupling by utilizing features extracted by a single feature encoder, and extracting clear image features from images of various weather types when a model is trained by using various different types of degraded images, therefore, different weather conditions can be adapted; (2) image degradation information is introduced, coding features of the clear image and the degradation image are distinguished, only the degradation image is subjected to image restoration, and mutual influence between the clear image and a restoration module is avoided; and (3) introducing a degradation category predictor to predict the degradation category of the image, and making a detector have the degradation category perception capability through degradation classification loss.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-weather target detection method based on degradation category-aware image restoration, belonging to the field of computer vision and intelligent information processing. BACKGROUND

[0002] Target detection technology can locate targets from images and give target categories, which is one of the basic tasks in computer vision. In recent years, with the proposal of many high-quality data sets and the proposal of advanced target detection algorithms, the real-time and accuracy of target detection technology on standard data sets has made significant progress. However, under adverse weather conditions, the data captured by the sensor often leads to a decrease in image quality, low visibility, and blurred target details, which brings challenges to the detection model in extracting target features. This limits the generalization and robustness of the model, resulting in a serious decline in detection performance under adverse weather conditions such as fog, haze, rain, and snow.

[0003] Existing methods use image pre-training restoration techniques such as de-fogging, de-raining, and de-snowing as preprocessing to mitigate the impact of adverse weather factors, and then send them to the detection network for target detection. Although these techniques can generally improve the visual quality of images, the generated restoration output may not be completely suitable for machine perception. Therefore, the performance of the two-stage restoration-detection framework trained separately is not ideal. In addition, some researchers use a cascading approach to optimize the restoration and detection networks simultaneously to ensure that the restoration results are more suitable for the perception of the detection network. Although these methods do improve detection accuracy under adverse weather conditions, they inevitably introduce additional model parameters, increasing model complexity, which is not conducive to real-time target detection. Recently, some research has focused on enhancing object detection under adverse weather conditions through transfer learning, especially domain adaptation methods. These methods mainly reduce the domain gap between the source domain (clear images) and the target domain (degraded images) to improve the perception of degraded images learned from clear images. These methods can extract the intrinsic features of the input image rather than the features of the restored image, but the transfer learning method ignores the fact that degradation factors always make it more difficult to extract target features, and it relies too much on source domain information, which limits their performance. It should be noted that most existing methods are optimized for single environmental degradation, and cannot handle multiple degradation scenarios with a single model. SUMMARY

[0004] In order to solve the deficiencies of the prior art, the present application proposes a multi-weather scene target detection method based on degradation category-aware image restoration, the purpose of which is to introduce degradation category information on the basis of the existing restoration-detection joint learning framework, thereby improving the detection performance of degraded images while ensuring the detection performance of clear images, and enabling the detector to have degradation category perception ability, thereby improving the stability of a single model under different weather conditions.

[0005] The application adopts the following technical scheme: a multi-weather target detection method based on degraded category-aware image restoration, which comprises the following steps:

[0006] (1) introducing a target detection network YOLOXs as a basic network, which comprises a backbone network and a detection decoder, using the backbone network as a shared feature encoder (SFE) for encoding image features, and using clear images and images of multiple different degradation types as inputs, and using the encoded image features for subsequent restoration, degradation prediction and target detection decoupling;

[0007] (2) introducing a restoration decoder based on the network in (1), using the multi-scale features extracted by the shared feature encoder to restore the input image;

[0008] (3) introducing a degradation discrimination module between the shared feature encoder and the restoration decoder in the framework in (2) to form a degradation discrimination image restoration decoder (DDIR), using the degradation category information to distinguish the degraded images from the clear images in the input, so as to restore only the degraded images, and the restored degraded images and the input clear images are used as the restoration results together;

[0009] (4) introducing a degradation category predictor (DCP) in the framework in (3) to distinguish the degradation information of the image, so that the encoder has degradation category awareness.

[0010] (5) in the inference process, the degradation discrimination image restoration decoder and the degradation category predictor in the framework in (4) can be selectively removed as needed to ensure real-time performance.

[0011] Compared with the prior art, the application has the following advantages:

[0012] 1. The degradation discrimination image restoration decoder proposed by the method can distinguish between degraded images and clear images when training a single model with images under multiple weather conditions, thereby avoiding the influence of clear images on the restoration module, which improves the detection performance of degraded images while ensuring the detection ability of clear images, and avoids the additional computational burden caused by restoring clear images during training;

[0013] 2. The application avoids the interference of clear images on the restoration module, and introduces clear images for model training to help the model learn more accurate target features;

[0014] 3. The restoration decoder of the application uses more low-level image features, which helps to restore image details and helps the shared feature encoder to learn detailed information that is easily lost in a degraded environment;

[0015] 4、The present application uses the proposed degradation class predictor, so that the shared feature encoder can learn the degradation class information of the image during the training process, so that the model has degradation class perception ability, which helps to improve the robustness of the detector under different degradation types;

[0016] 5、In the inference process, the degradation discriminative image restoration decoder and the degradation class predictor proposed by the present application are removable, and the corresponding decoder can be retained according to the needs in the application process, so as to avoid redundant calculation overhead on the premise of ensuring the corresponding function. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The overall network block diagram of the degradation class perception restoration multi-weather scene target detection method of the present application;

[0018] Figure 2 The degradation discriminative image restoration process of the present application;

[0019] Figure 3 The target detection result schematic diagram of the present application on clear images and fog, rain and snow degradation images; DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0021] Figure 1 The network structure and principle of the degradation class perception restoration multi-weather scene target detection method are shown, including a target detection network composed of a shared feature encoder and a detection decoder, and a degradation discriminative image restoration decoder and a degradation class predictor. In the training process, the target detection, image restoration and degradation classification are optimized jointly to make the shared feature encoder extract clear image features from the degradation image and make it have degradation class perception ability, so as to improve the detection performance of the model on the degradation image and enhance the stability of the model under multi-weather conditions on the premise of ensuring the detection accuracy of the clear image. In the inference process, the degradation discriminative image restoration decoder and the degradation class predictor can be removed according to the needs, and the additional calculation burden can be avoided on the premise of providing the corresponding information. The specific process is as follows:

[0022] (1) Target detection network

[0023] As Figure 1As shown, the object detection network of the present application includes a shared feature encoder and an object detection decoder. Among them, the shared feature encoder is composed of Focal, convolution, cross-stage local network (CSP), spatial pyramid pooling (SPP) and dynamic Transformer feature enhancement module (DTFE). For the input image X, Focal first extracts shallow features from the low-quality image and scales down the shallow features to obtain the first shallow image features. Then a series of cascaded convolution and CSP are used to extract multi-scale features. In this process, SPP and DTFE are used to enhance the feature representation capability of the entire encoder. The shared feature encoder extracts four scale features, which can be represented as:

[0024] f ×4 ,f ×8 ,f ×16 ,f ×32 =E SFE (X) (1)

[0025] Where E(·) represents the feature extraction process of the input image, f ×4 ,f ×8 ,f ×16 ,f ×32 represents image features of different scales, and subscript xn represents the down-sampling multiple of the feature size relative to the input image. These multi-scale features will be used for decoupling of object detection, image restoration and degradation class prediction. Therefore, for the object detection network, the shared feature encoder model optimized through back propagation ensures that these features contain information from clear images and perception of degradation categories. This helps to improve feature extraction capability and enhance the generalization ability of the model under various weather conditions.

[0026] The object detection decoder uses the widely used feature pyramid network and path aggregation network, and introduces self-correcting convolution (SC Conv) in the path aggregation network to expand the feature receptive field. Specifically, the feature pyramid network first uses a 1×1 convolution to adjust the number of channels and then up-samples the high-level features f ×32 , and then performs a concatenation operation on the up-sampled features and the corresponding scale features from the output of the shared feature encoder. This operation is repeated twice to produce two fused features of different scales, and the specific process is as follows:

[0027]

[0028] Where Conv k,s represents a convolution operation with a kernel size of k×k and a stride of s. Up(·) represents a two-fold up-sampling. Cat[·,·] cThis indicates a concatenation operation along the channel dimension. CSP(·) and SC(·) represent the functions of the CSP and SC modules, respectively. In the subsequent path aggregation network, features... Convolution operations are performed to further extract features, and the convolution stride is set to achieve feature downsampling, thereby generating features with smaller dimensions. and This process can be represented as:

[0029]

[0030] Finally, three detection heads were applied to each... Decoupling helps in detecting targets of different sizes.

[0031] (2) Restore the network

[0032] Since both sharp and degraded images are used for model training to enhance the model's adaptability in different environments, restoring sharp images can interfere with the model's training process, resulting in poor restoration performance for degraded images. Therefore, this invention introduces a degrade discriminator between the shared feature encoder and the restoration decoder. This discriminator uses degrade information to distinguish between sharp and degraded images, restoring only the degraded images, thus avoiding mutual interference between the sharp image and the restoration module.

[0033] The overall restoration process is as follows Figure 2 As shown, for a batch of input images, regardless of whether they are clear or degraded, multi-scale image features are first extracted using a shared feature encoder. Then, the image features are distinguished based on the image degradation information. If it is a degraded image, the image features are sent to the restoration decoder to obtain the restored image. For clear images, no image restoration is performed; the input image and the degraded image restoration result are directly concatenated as the final restored output. It is important to note that the degradation discriminator discriminates each image in the batch of inputs one by one, ensuring that the image order of the batch of inputs and the corresponding restoration results is consistent, in order to facilitate the calculation of the final restoration loss. The specific process is as follows: Figure 1 As shown, for degraded images, the restoration decoder of this invention utilizes multi-scale features f extracted by a shared feature encoder. ×4 ,f ×8 ,f ×16 ,f ×32 Generate the restored image. Deep features f ×32 The image is restored to the same size as the input image through five cascaded upsampling and convolution operations. During this process, the shallower features f... ×16 ,f ×8 ,f ×4 They were gradually integrated into it, and the overall process is as follows:

[0034]

[0035] in Refers to the degraded image features extracted by a shared feature encoder, f R This represents the image features generated during the restoration process by the restoration decoder. This represents the output of the restoration decoder, where H and W are the spatial dimensions of the input images. For a batch of sharp images from the input, it is directly compared with R. D The final restored result is obtained by piecing together the parts.

[0036] R = Cat[R] D ,X C (11)

[0037] Here X C R represents the sharp image in the input, and R represents the final image restoration result. During inference, this restoration decoder can be removed to avoid additional computational burden.

[0038] (3) Degradation Category Decoder

[0039] like Figure 2 As shown, this invention extends the joint learning framework for detection and restoration by adding a degradation category decoder, which uses multi-scale features extracted by a shared feature encoder to predict the image degradation category. Specifically, the number of feature channels is first adjusted by a 1×1 convolution, then global features are extracted through a pooling layer, and finally the final prediction result is generated through a fully connected layer. The overall process is as follows:

[0040] O D =FC(Pool(Conv) 1,1 (f ×32 (12)

[0041] Where FC(·) represents a fully connected layer, and Pool(·) represents a pooling layer. D This represents the model's predicted probability of the type of image degradation. Degradation category predictors can be selectively removed depending on whether the model needs to provide image degradation information, thus balancing functional requirements with detection speed requirements.

[0042] (4) Loss Function

[0043] In order to optimize detection, restoration and degradation classification respectively, this invention introduces detection loss, restoration loss and degradation classification loss respectively.

[0044] The detection loss consists of three parts: L box L represents the positional difference between the predicted bounding box and the actual bounding box. obj The confidence loss of the predicted bounding box, L cls The target category prediction loss can be represented as:

[0045] L Det =λL box +L cls +L obj (13)

[0046] Where λ is set to 5. box and L cls Using IOU loss and cross-entropy loss functions respectively, L obj Focal loss is then used to balance the imbalance between the number of positive and negative samples in the detection task.

[0047] The restoration loss is calculated using the MSE function to determine the difference between the sharp image and the restored image. It's important to note that since this invention trains the model using both sharp and degraded images, the restoration result includes not only the restored degraded image but also the sharp image from the input. Since the sharp image serves as the label, the restoration loss for the input sharp image is always 0, thus not affecting the model parameters and avoiding interference from the sharp image in the restoration module. Therefore, the restoration loss can be expressed as:

[0048]

[0049] Where N represents the number of images in a batch of input, N D This refers to the number of degraded images in the input. R D and These represent the restored degraded image and its corresponding clear image label, respectively.

[0050] The degradation classification loss uses the cross-entropy function to calculate the difference between the degradation category prediction result and the true degradation label, expressed as:

[0051]

[0052] Where K represents the type of deterioration weather, set to 4, representing four image weather conditions: clear image, fog image, rain image, and snow image. t represents the actual deterioration category of the image. This represents the probability that the model predicts the image to be of the i-th type of degradation.

[0053] The total loss used in this invention is the sum of the three losses mentioned above, as detailed below:

[0054] L=αL Det +βL Res +γL dc (16)

[0055] Where α, β, and γ are the weighting coefficients of the detection loss, restoration loss, and degradation category loss, respectively. Jointly optimizing the model using these three losses enables the detection model to extract clearer image features and acquire degradation category awareness, which helps improve the detector's performance under various weather conditions.

[0056] This invention aims to improve the detection performance of a single detection model under different weather conditions. Therefore, 11,707 images from the VOC dataset were selected as the clear image dataset VOC-Clean, with 9,578 images used as the training set and 2,129 images as the test set. Based on VOC-Clean, fog image dataset VOC-FOG, rain image dataset VOC-Rain, and snow image dataset VOC-Snow were synthesized. Furthermore, the real-world fog image dataset RTTS and Foggy Driving Dataset (FDD) were used as test sets to evaluate the effectiveness of this invention in real-world degraded scenes. All experiments were conducted on a single GeForce RTX 3090 graphics card. During the training phase, stochastic gradient descent (SGD) was used to optimize the model, with an initial learning rate of 0.01, momentum of 0.937, weight decay of 0.0005, batch size of 8, and input image size fixed at 640*640.

[0057] Four detection methods were selected as comparison methods, specifically:

[0058] Method 1: The method proposed by Ge et al., see reference "YOLOX: Exceeding yolo series in 2021."

[0059] Method 2: The method proposed by Li et al., see reference "AOD-Net: All-in-one dehazing network. 2017".

[0060] Method 3: The method proposed by Huang et al., see reference "DSNet: Joint semantic learning for object detection in inclement weather conditions, 2020".

[0061] Method 4: The method proposed by Wang et al., see reference "Togethernet: Bridging image restoration and object detection together via dynamic enhancement learning. 2022".

[0062] To comprehensively evaluate the improvement effect of this invention, the method was evaluated under both single foggy weather degradation and multi-weather conditions. First, under single foggy weather degradation, the basic detection method YOLOXs, the two-stage restoration-then-detection method AODNet-YOLOXs*, and the target detection methods TogetherNet and DS-Net under adverse weather conditions were selected for comparison. Since this invention utilizes multiple degradation information, it needs to be trained on datasets with different degradation types. Therefore, models trained for 50 epochs using VOC-Clean and VOC-FOG were used to evaluate the detection performance on synthetic and real fog images. In the comparative experiments, YOLOXs and TogetherNet were trained for 100 epochs on the VOC-FOG training set, while YOLOXs* was trained for 100 epochs on the VOC-Clean training set using the YOLOXs model. The results are shown in Table 1.

[0063] In multi-weather scenarios, since both the two-stage method and adverse weather detection methods only optimize for single-level degradation, YOLOX and TogetherNet were chosen for comparison. The detection results on different weather test sets are as follows: Figure 2 As shown. It should be noted that YOLOXs* is also trained for 100 epochs on the VOC-Clean training set, while the other three methods are trained for 25 epochs on a mixed dataset of VOC-Clean, VOC-FOG, VOC-Rain, and VOC-Snow, and tested on each of the four datasets separately.

[0064] As can be seen, under the single foggy weather degradation condition, this invention significantly improves upon existing methods on both synthetic and real datasets. Under multiple weather conditions, this invention achieves the highest accuracy in all four weather scenarios and can accurately assess the type of image degradation, with a Top1 accuracy of 99.27%. Figure 3 The detection performance of YOLOXs, TogetherNet, and the method of this invention on clear, foggy, rainy, and snowy images was compared.

[0065] Table 1 compares the detection performance on synthetic fog map datasets and real fog map datasets.

[0066]

[0067] Table 2 compares the detection performance on clear images and composite images of fog, rain, and snow.

[0068]

Claims

1. A multi-weather target detection method based on degraded class-aware image restoration, characterized in that, The method comprises the following steps: (1) using a recovery and detection combined learning training paradigm, using single feature encoder extracted features for decoupling recovery and detection, when using multiple different types of degraded images to train the model, clear image features can be extracted from various weather types of images, thereby adapting to different weather conditions; (2) introducing image degradation information, distinguishing the encoding features of clear images and degraded images, only recovering the degraded images, avoiding the mutual influence between clear images and recovery modules, and inputting the recovery results of degraded images and clear images together as the recovery output and clear image label to calculate the loss, so that the model can improve the detection effect of degraded images while ensuring the detection effect of clear images; (3) introducing a degradation category predictor, using the features extracted by the recovery and detection feature encoder for degradation category prediction, and introducing a classification loss to optimize the feature encoder and the degradation category predictor, so that the detector has degradation category perception ability.

2. The method of claim 1, wherein, The combined learning framework and training paradigm in step (1) are as follows: First, a shared feature encoding network is used to extract features from the input image. Through the Focus operation, the shallow features of the input image are extracted and scaled down to obtain the first shallow image features. Through the backbone network, the first shallow image features are extracted by a set of modules composed of multiple convolutions and cross-stage local modules (CSP) for feature transformation, and a multi-core convolution with a preset step size is used for downsampling operation to obtain multiple image features with different scales. The dynamic Transformer feature enhancement module (DTFE) is used to extract features from the output image features, and the extracted multi-scale features can be represented as: f ×4 ,f ×8 ,f ×16 ,f ×32 =E SFE (X) (1) wherein wherein E(·) represents a feature extraction process of the input image, f ×4 ,f ×8 ,f ×16 ,f ×32 represent image features at different scales; For the object detection decoder, a feature pyramid network and a path aggregation network are adopted, and a self-correcting convolution (SC Conv) is introduced in the path aggregation network to expand the feature receptive field. Through the network, three different scale features f ×8 ×16 ×32 are fused to finally generate three fused features with high-level semantic information and low-level detailed information, and the specific process is as follows:​​ where Conv k,s denotes the convolution operation with kernel size k x k and stride s, Up(·) denotes the up-sampling by a factor of 2, Cat[·,·] c denotes the concatenation operation along the channel dimension, CSP(·) and SC(·) denote the functions of CSP and SC modules, respectively; and are sent into the detection head to decouple target information, and through IOU, cross-entropy function and focal loss as the prediction box regression loss, classification loss and prediction box confidence loss, which together constitute the detection loss to optimize the shared feature encoder and the target detection decoder, so that the shared feature encoder has the ability to extract target information from images. For image restoration, the image restoration decoder is composed of 5 cascaded upsampling and convolution operations, which utilize four different scale features f ×4 ×8 ×16 ×32 The smallest features f ×32 are recovered to the same size as the input image by the restoration decoder, which gradually incorporates larger size features f ×16 ×8 ×4 and the restoration loss is calculated as the mean square error between the restored image and the clear image label to optimize the restoration decoder and the shared feature encoder.​​​​​ 3. The method of claim 1, wherein, In step (2), a degradation discriminator is applied, and the specific steps are as follows: A degradation discriminator is introduced between the image feature encoder and the recovery decoder, and the degradation discriminator and the recovery decoder together constitute a degradation discrimination image recovery decoder. After a batch of input images are extracted by the shared feature encoder, the features are distinguished according to the input image degradation category information, the degraded image features are sent to the recovery decoder for image recovery, and the clear image features are not subjected to image recovery, but directly use the input image as the recovery result. The clear input image and the recovered degraded image are spliced according to the input order as the final recovery result, and the overall recovery process is as follows: R = Cat[R D , C ] (11) wherein denotes the degraded image features extracted by the shared feature encoder, f R denotes the image features generated in the restoration decoder restoration process, denotes the output of the restoration decoder, H, W are the spatial dimensions of the input image, X C denotes the clear image in the input, R denotes the final image restoration result; in the model optimization process, the clear image is directly taken as the restoration result, so the restoration loss between the clear image and the label is 0, therefore, the restoration loss can be represented as: where N denotes the number of input images in a batch, N D R denotes the number of degraded images in the input D and respectively represent the restored degraded image and its corresponding clean image label; can make the shared feature encode its ability to extract clean image information from degraded images, while avoiding interference between clean images and restored decoders; The above degradation discrimination image recovery decoder is used to assist the training of the detection model, and in the inference process, the recovery decoder can be removed to avoid additional computational burden and speed up the inference speed.

4. The method of claim 1, wherein, In step (3), a degradation category predictor is applied, and the specific steps are as follows: On the basis of the recovery and detection combined learning paradigm, a degradation category decoder is introduced, which is composed of convolution layers, pooling layers and fully connected layers. It uses the last layer of features output by the shared feature encoder to decode the image degradation category and output the predicted degradation category. The specific process is as follows: O D = FC(Pool(Conv 1,1 (f ×32 ))) (13) where FC(·) denotes a fully connected layer, Pool(·) represents a pooling layer, O D denotes the predicted probability of the model for the input image degradation type; in the training process, the classification loss between the predicted degradation class and the real degradation label is calculated using the cross-entropy function, as follows: where K denotes the type of the weather type in the training data; t denotes the true degradation class of the image, O i D denotes the probability that the model predicts the image to be the i-th degradation; the shared feature encoder and the degradation class predictor are optimized by the loss, so that the shared feature encoder has the degradation class perception ability; The degradation class predictor can accurately determine the image degradation information. In the inference process, the degradation class predictor can be removed to reduce the computational burden, or can be retained to provide additional image degradation information.

Images