Robust target detection method suitable for severe weather conditions

Abstract

The invention provides a robust target detection method suitable for severe weather conditions. The method comprises the following steps: acquiring a severe weather image data set; an SR-YOLO target detection network is constructed, on the basis of a YOLOv5s model, a CDCN cascade deformable convolution module is introduced into a backbone network, a neck network is replaced by a cascade FF feature fusion module, and label distribution is realized after a detection head through an OTA-Snow snowy day optimal transmission distribution module; training and testing the SR-YOLO target detection network through a severe weather image data set to obtain a trained SR-YOLO target detection network; and inputting a to-be-detected severe weather image into the trained SR-YOLO target detection network to obtain a target detection result. According to the method, the calculation overhead is reduced, and the reasoning speed, the feature fusion efficiency and the label distribution quality are improved.

Description

Technical Field

[0001] This invention relates to the technical field of target detection, and more particularly to a robust target detection method suitable for adverse weather conditions. Background Technology

[0002] With the continuous development of deep learning technology, object detection based on convolutional neural networks (CNNs) has been widely used in autonomous driving. While these detectors exhibit high performance under normal weather conditions, their effectiveness decreases significantly under adverse weather conditions. Neural network detectors are highly dependent on the integrity of the target image, and severe weather conditions such as snowfall, rain, and fog can severely degrade image quality. Especially in snowy weather, targets are often partially or completely obscured, leading to the loss of crucial information and even detection failures.

[0003] To address the problem of target detection under adverse weather conditions, some researchers have developed specific detection methods for these scenarios. The most common approach is to preprocess the image using existing image restoration algorithms, such as snow removal methods, as described in the literature [C. Guo, C. Li, J. Guo, CCLoy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deepcurve estimation for low-light image enhancement,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 1780–1789.]. The preprocessed image is then input into the detection model to obtain the final result. While these preprocessing techniques can improve visual quality, they often lead to the loss of image details and reduce processing speed. In existing methods, a common approach is to first preprocess the snowy image (e.g., snow removal, enhancement) and then input the processed image into the detection model, such as... This "serial two-stage processing" increases computational overhead and significantly reduces inference speed, making it difficult to meet real-time requirements. When object detection is performed directly on the raw image, even the best-performing detectors will experience significant performance degradation under adverse weather conditions due to their high reliance on the detector's own feature extraction capabilities.

[0004] The image-adaptive yolo proposed by Liu et al. (reference number: [W.Liu,G.Ren,R.Yu,S.Guo,J.Zhu,andL.Zhang,“Image-adaptive yolo for object detection in adverse weather conditions,”in Proceedings of the AAAI Conference on Artificial Intelligence,vol.36,2022,pp.1792–1800]) improves detection performance by adaptively enhancing each image. However, the enhancement of this method mainly stays at the image level and lacks global optimization allocation of anchor points and ground truth labels. It is prone to misallocation under low contrast and background noise conditions in snowy weather, which affects the quality of the supervision signal. The R-YOLO framework proposed by Wang et al. (reference number: [L.Wang,H.Qin,X.Zhou,X.Lu,andF.Zhang,“R-yolo:a robust object detector inadverse weather,”IEEE Transactions on Instrumentation and Measurement,vol.72,pp.1–11,2022.]) utilizes domain adaptation and adversarial learning to reduce inter-domain differences. Although it can alleviate the domain shift problem under severe weather conditions to some extent, its label allocation ability is insufficient in complex snowy backgrounds, making it difficult to guarantee the accuracy of the supervision signal, leading to missed detections and false detections. The CF-YOLO framework proposed by Ding et al. (document number: [Q Ding, P.Li, X.Yan, D.Shi, L.Liang, W.Wang, H.Xie, J.Li, and M.Wei, “Cf-yolo: cross fusion yolo for object detection in adverse weather with a high-quality real snow dataset,” IEEE Transactions on Intelligent Transportation Systems, 2023.]) fuses features from different stages of the network to improve detection performance. However, it still uses a sequential feature fusion method, which has low information transmission efficiency and is prone to losing low-level detail information during the fusion process, making it difficult to effectively recover subtle features in snowy scenes.The first snow removal network, Snownet (reference number: [Y.-F.Liu, D.-W.Jaw, S.-C.Huang, and J.-N.Hwang, “Desnownet: Context-aware deep network for snow removal,” IEEE Transactions on Image Processing, vol.27, no.6, pp.3064–3073, 2018.]), employs a multi-stage progressive snow removal model. While improving image clarity, it also introduces additional computational overhead and increases inference time. Furthermore, a domain bias exists between its image restoration and object detection, resulting in limited improvement in detection accuracy. Wang et al. proposed RDMNet (document number [X.Wang,X.Liu,H.Yang,Z.Wang,X.Wen,X.He,L.Qing,and H.Chen,“Degradation modeling for restoration-enhanced object detection in adverse weather scenes,”IEEE Trans.Intell.Veh.,2024.]), which improves target detection performance in adverse weather conditions through a dual-branch network of degradation modeling and restoration. However, this method has a complex structure and slow inference speed, making it difficult to meet real-time requirements. Summary of the Invention

[0005] To address the technical problems of existing methods, such as high computational cost, slow inference speed, low feature fusion efficiency, and label mismatch caused by severe weather conditions, this invention proposes a robust target detection method (SnowRobust-YOLO, SR-YOLO) suitable for severe weather conditions. This method reduces computational cost and improves inference speed, feature fusion efficiency, and label assignment quality.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] A robust target detection method suitable for severe weather conditions, comprising the following steps:

[0008] S1: Obtain a dataset of severe weather images;

[0009] S2: Construct the SR-YOLO target detection network. Based on the YOLOv5s model, introduce the CDCN cascaded deformable convolution module into the backbone network, replace the neck network with the cascaded FF feature fusion module, and implement label allocation through the OTA-Snow snow-day optimal transmission allocation module after the detection head.

[0010] S3: Train and test the SR-YOLO target detection network using a dataset of severe weather images to obtain a trained SR-YOLO target detection network;

[0011] S4: Input the severe weather image to be detected into the trained SR-YOLO object detection network to obtain the object detection results.

[0012] Furthermore, the backbone network includes a Focus module, a first Conv module, a first C3 module, a second Conv module, a second C3 module, a third Conv module, a third C3 module, a fourth Conv module, an SPP module, and a CDCN module connected in sequence; the CDCN module adaptively adjusts the shape of the convolution kernel and the sampling position through deformable convolution;

[0013] The neck network includes a first FF module and a second FF module connected to each other. The first FF module is connected to the second C3 module, the third C3 module and the CDCN module respectively. The first FF module and the second FF module recover the low-level information lost in the high-level feature extraction process by simultaneously fusing features from different stages of the network.

[0014] Furthermore, the CDCN module includes a parallel fifth Conv module and a sixth Conv module. The inputs of both the fifth and sixth Conv modules are connected to the SPP module, and the output of the fifth Conv module is connected to the DCN module. The output of the sixth Conv module is concatenated with the output of the DCN module and then output to the first FF module through the seventh Conv module. For the input feature map from the fifth Conv module, the DCN module uses deformable convolution to adaptively adjust the convolution sampling position and weights to extract a geometrically adaptive local feature map. For the local feature map obtained by deformable convolution, the DCN module uses deformable pooling to adaptively adjust the pooling sampling position and weights to obtain an enhanced geometrically aware feature map.

[0015] Furthermore, the first FF module includes three parallel processing stages. Each stage processes features from the second C3 module, the third C3 module, and the CDCN module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage.

[0016] The first processing stage includes three parallel gOctCBR layers: a first gOctCBR layer, a second gOctCBR layer, and a third gOctCBR layer. The input of the first gOctCBR layer is connected to the output of the second C3 module, the input of the second gOctCBR layer is connected to the output of the third C3 module, and the input of the third gOctCBR layer is connected to the output of the seventh Conv module in the CDCN module. The outputs of the first, second, and third gOctCBR layers are all connected to the input of the first CSP module.

[0017] The second processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the second CSP module.

[0018] The third processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the third CSP module.

[0019] Furthermore, the second FF module includes three parallel processing stages corresponding to the first FF module. Each stage processes the output features from the first CSP module, the second CSP module, and the third CSP module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage.

[0020] The first processing stage includes parallel fourth, fifth, and sixth gOctCBR layers. The input of the fourth gOctCBR layer is connected to the output of the first CSP module, the input of the fifth gOctCBR layer is connected to the output of the second CSP module, and the input of the sixth gOctCBR layer is connected to the output of the third CSP module. The outputs of the fourth, fifth, and sixth gOctCBR layers are all connected to the input of the fourth CSP module.

[0021] The second processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the fifth CSP module.

[0022] The third processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the sixth CSP module.

[0023] The output of the fourth CSP module is connected to the first detection head, the output of the fifth CSP module is connected to the second detection head, and the output of the sixth CSP module is connected to the third detection head.

[0024] Furthermore, each gOctCBR layer includes a gOctConv layer, a batch normalization layer, and a PReLU activation function connected in sequence;

[0025] Each CSP module includes a first CBS layer and a second CBS layer in parallel. The output of the second CBS layer is connected to the input of the first Bottleneck layer, and the output of the first Bottleneck layer is connected to the input of the second Bottleneck layer. The outputs of the first CBS layer and the second Bottleneck layer are concatenated and then output through a third CBS layer.

[0026] Each Bottleneck layer consists of a fourth CBS layer and a fifth CBS layer connected in sequence. The input of the fourth CBS layer is skipped to the output of the fifth CBS layer. Each CBS layer consists of a convolutional layer, a batch normalization layer, and a Silu activation function connected in sequence.

[0027] Furthermore, the method for tag allocation via the OTA-Snow module's optimal snow-day transmission allocation module after the detection head is as follows:

[0028] (1) Dynamically allocate estimates for each ground truth box to obtain the number of candidate boxes that each ground truth box needs to match;

[0029] (2) Define the requirement vector based on the number of candidate boxes. Supply vectors are generated based on the estimated number of candidate boxes through dynamic allocation. Update: n is the number of candidate boxes, and m is the number of ground truth boxes;

[0030] (3) Construct a multi-dimensional cost matrix, including a classification cost matrix, a regression cost matrix, a center prior cost matrix, a snow influence cost matrix, and a background classification cost matrix;

[0031] (4) Aggregate the classification cost matrix, regression cost matrix, center prior cost matrix and snow influence cost matrix to obtain the foreground cost matrix, and concatenate the background classification cost matrix as the last row with the foreground cost matrix to obtain the global cost matrix c.

[0032] (4) Calculate the optimal allocation strategy based on the Sinkhorn-Knopp algorithm.

[0033] Furthermore, the method for calculating the optimal allocation strategy based on the Sinkhorn-Knopp algorithm is as follows:

[0034] Step 1: Initialize the Sinkhorn scaling coefficients u0 and v0, and perform Sinkhorn-Knopp iterative calculations based on the global cost matrix c, supply vector s, and demand vector d;

[0035] Step 2: Iterate until the required number of iterations or the scaling factor v is reached. t,j and scaling factor u t+1,i When the change is less than the threshold, the final scaling coefficient u is obtained. last,i and the final scaling factor v last,j ;

[0036] Step 3: Based on the final scaling coefficient u after iteration last,i and the final scaling factor v last,j Calculate the optimal allocation strategy:

[0037]

[0038] in, Let be the probability that candidate box j matches the true box i, which represents the optimal allocation strategy.

[0039] Furthermore, the method for dynamically assigning estimates to each ground truth bounding box is as follows:

[0040] Calculate the matching quality metric (IoU) between each candidate box and its corresponding ground truth box, such as IoU or classification confidence. Sort the candidate boxes according to their matching quality with the target ground truth box from highest to lowest. Select the top-ranked candidate boxes and sum their matching quality metrics to obtain the summed matching quality metric for that ground truth box. Round the summed matching quality metric to the nearest integer as the number of matches, k, for that ground truth box. i And set a lower limit constraint:

[0041]

[0042] Where max(1,·) represents the lower bound constraint of assigning at least one candidate box; Indicates rounding down; ∑ j Q ij Q represents the cumulative matching quality index. ij Let be the quality metric for matching candidate box j with the corresponding ground truth box i;

[0043] Supply vectors are generated based on the estimated number of candidate boxes through dynamic allocation. The update method is as follows:

[0044]

[0045] The classification cost matrix is ​​calculated using Focal Loss, the regression cost matrix is ​​calculated using IoU Loss, the center prior cost matrix is ​​calculated using center point loss, the snow influence cost matrix is ​​calculated using snow influence loss, and the background classification cost matrix is ​​calculated using Focal Loss.

[0046] Furthermore, after obtaining the optimal allocation strategy, the total loss function is calculated as follows:

[0047]

[0048] Among them, L cls Focal Loss, used as the classification loss, L reg For IoU Loss, used as the regression loss, λ cls , λ reg These represent the weighting coefficients for classification loss and regression loss, respectively.

[0049] The Sinkhorn-Knopp iterative calculation formula is:

[0050]

[0051] Among them, s i For the elements in the supply vector, d j Let t be the element in the demand vector, γ be the iteration number, and v be the regularization parameter. t,j The scaling factor for the j-th candidate box is used to adjust the total transmission amount of all ground truth boxes i to candidate box j, so that candidate box j matches the preset requirement d. j u t+1,i The scaling factor is the value corresponding to the i-th ground truth box, used to adjust the total amount of transmission from ground truth box i to all candidate boxes j, so that ground truth box i matches the preset supply.

[0052] The beneficial effects of this invention are as follows:

[0053] In snowy conditions, due to snowflake obscuring or noise interference, some anchor points resemble neither clear foreground targets nor pure background, thus becoming ambiguous anchor points. This invention proposes an Optimal Transport Assignment for Snowy Conditions (OTA-Snow) module. Through a global optimization strategy, foreground and background cost matrices are constructed during the training phase. Combined with the constraints of the supply vector, the optimal transport method is used for global matching to reasonably assign these ambiguous anchor points to the background or the most suitable target category. This improves the accuracy of the supervision signal, effectively avoids label mismatch problems caused by snow noise, thereby improving detection accuracy and enhancing detection performance in snowy scenarios.

[0054] The Cascade Deformable Convolution Network (CDCN) module is designed. Through the adaptive sampling mechanism of deformable convolution and pooling, it directly adapts to snow occlusion and geometric deformation during the feature extraction stage. This not only improves the feature representation capability but also maintains a high inference speed, which is more in line with the needs of real-time detection.

[0055] A feature fusion module (FF) was developed, which restores low-level information lost during high-level feature extraction by simultaneously fusing features from different stages of the network, thereby enhancing feature representation and improving feature fusion efficiency.

[0056] The SR-YOLO framework proposed in this application adopts an end-to-end integrated design, which deeply couples robustness enhancement in snowless environments with object detection, avoiding additional independent preprocessing steps and thus reducing inference overhead. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a diagram of the overall architecture of the SR-YOLO model of the present invention.

[0059] Figure 2 This is a schematic diagram of the deformable convolution operation in the DCN module of the present invention.

[0060] Figure 3 This is a structural diagram of the CDCN module of the present invention.

[0061] Figure 4 This is the FF module structure of the present invention.

[0062] Figure 5 The figure shows a comparison of the performance of this invention on the RSOD dataset with YOLOv8s.

[0063] Figure 6 This is a comparison chart of the visualization results of the SR-YOLO feature map based on the RSOD dataset of this invention with other methods.

[0064] Figure 7 This is a comparison chart of the visualization detection results of the present invention on the RTTS dataset with other methods. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] A robust target detection method suitable for severe weather conditions, such as Figure 1 As shown, the steps are as follows:

[0067] S1: Obtain a dataset of severe weather images;

[0068] The dataset used in this invention includes:

[0069] The RSOD dataset (Real Snow Object Detection) contains 2,100 images of real-world traffic scenes in snowy conditions, with precise bounding boxes for each object. The training set contains 1,701 images, the validation set contains 189 images, and the test set contains 210 images. This dataset annotates 12 categories of traffic-related objects, including pedestrians, bicycles, cars, buses, trucks, traffic lights, fire hydrants, benches, dogs, backpacks, umbrellas, and handbags. The RSOD dataset covers a variety of urban traffic scenes and environments, helping to improve the model's generalization ability under snowy conditions. This invention utilizes this dataset for training and validation to enhance the model's detection performance in snowy environments.

[0070] The RTTS dataset (Real-world Task-driven Testing Set) is a subset of the RESIDE dataset, containing 4322 real-world traffic scene images captured under foggy conditions, along with bounding boxes for each target. The training set contains 3457 images, and the validation set contains 865. This dataset primarily annotates five categories of traffic-related targets: motorcycles, bicycles, pedestrians, buses, and cars, covering diverse traffic and driving scenarios. This invention utilizes this dataset for training and validation to improve the model's robustness under foggy conditions.

[0071] The rRain dataset contains 1900 real-world rainy traffic scene images collected at different locations and times, covering five categories of traffic targets: pedestrians, bicycles, motorcycles, cars, and buses. Unlike the RSOD and RTTS datasets, the rRain dataset does not have a fixed division into training, validation, and test sets; it is used as the test set. This invention utilizes this dataset to evaluate and validate the model's detection performance and robustness in rainy conditions.

[0072] Preprocessing was performed on the RSOD, RTTS, and rRain datasets. First, all images were uniformly adjusted to 640×640 resolution to fit the network input. Data augmentation methods were applied to all training sets, including random cropping, flipping, and brightness / contrast adjustment, to enhance the model's generalization ability under complex weather conditions. The datasets were converted into the bounding box coordinates + class label format required for object detection, for subsequent modules to use. The number of ground truth bounding boxes m = size(H) and the number of candidate bounding boxes n = size(S) were initialized, where H represents the set of ground truth bounding boxes and S represents the set of anchor points.

[0073] S2: Construct the SR-YOLO object detection network. Based on the YOLOv5s model, introduce a cascaded deformable convolutional module (CDCN) into the backbone network, replace the neck network with a cascaded feature fusion module (FF), and implement label allocation through the snow-optimal transmission allocation module (OTA-Snow) after the detection head.

[0074] The backbone network includes a Focus module, a first Conv module, a first C3 module, a second Conv module, a second C3 module, a third Conv module, a third C3 module, a fourth Conv module, an SPP module, and a CDCN module connected in sequence. The CDCN module adaptively adjusts the shape and sampling position of the convolution kernel through deformable convolution, enabling the network to better capture the geometric deformation of targets covered by snow, and further improves the feature representation ability of targets occluded by snow.

[0075] like Figure 3 As shown, the CDCN module includes a fifth Conv module and a sixth Conv module in parallel. The inputs of the fifth Conv module and the sixth Conv module are both connected to the SPP module. The output of the fifth Conv module is connected to the DCN module. The output of the sixth Conv module is concatenated with the output of the DCN module and then output through the seventh Conv module.

[0076] The processing method of the DCN module is as follows:

[0077] First, for the input feature map x1 from the fifth Conv module, deformable convolution adaptively adjusts the convolution sampling position and weights to extract geometrically adaptive local feature maps. Deformable convolution adaptively adjusts the sampling position and weights to make the sampling points conform to the target geometry, while enhancing salient regions and suppressing the background, thus achieving accurate extraction of effective local features.

[0078] Specifically, the method for implementing deformable convolution is as follows:

[0079] like Figure 2As shown, for a certain position e on the output feature map y1 of deformable convolution, N standard convolution sampling points e are preset. n In this embodiment, there are 9 grid points in a 3×3 grid. The first offset Δe is predicted by the first offset prediction sub-network (an independent 1×1 convolutional layer I). n The first modulation factor Δm is predicted through the first modulation factor prediction sub-network (an independent 1×1 convolutional layer II). n On the input feature map x1 from the fifth Conv module, find each actual sampling position e+e n +Δe n eigenvalues ​​x1(e+e) n +Δe n For each eigenvalue x1(e+e) n +Δe n ) By fixing the weight d n and the first modulation factor Δm n We perform weighted summation on the N weighted eigenvalues ​​to obtain the eigenvalue y1(e) of the output feature map y1 at position e; we then iterate through all positions on the output feature map y1 to obtain the output feature map y1.

[0080]

[0081] Furthermore, the local feature map obtained by deformable convolution is augmented with a geometrically perceptual feature map by adaptively adjusting the pooling sampling position and weights through deformable pooling. Deformable pooling, by adaptively adjusting the pooling sampling position and weights, reduces dimensionality and parameters while prioritizing the focus on the effective target region, avoiding the loss of geometric information and preserving the target's key morphological features.

[0082] Specifically, the method for implementing deformable pooling is as follows:

[0083] For a pooling unit j on the feature map y2 output by deformable pooling, K standard sampling points d are preset. jk d jk This represents the k-th sampling point in the j-th pooling unit, which in this embodiment is 4 grid points in a 2×3 matrix. The second offset Δd is predicted by the second offset prediction sub-network (an independent 1×1 convolutional layer III). jk The second modulation factor Δm is predicted by the second modulation factor prediction subnetwork (an independent 1×1 convolutional layer IV). jk On the output feature map y1 from the deformable convolution, find each actual sampling location d jk +Δd jk eigenvalues ​​y1(d jk +Δd jk For each eigenvalue y1(d) jk +Δd jk) through the second modulation factor Δm jk Weighted summation is performed on the K weighted eigenvalues ​​to obtain the eigenvalue y2(j) of the output feature map y2 in pooling unit j, and then averaged by the number of samples K; the output feature map y2 is obtained by traversing all pooling units on the output feature map y2.

[0084]

[0085] like Figure 1 As shown, the neck network includes a first FF module and a second FF module connected to each other. The first FF module is connected to the second C3 module, the third C3 module and the CDCN module respectively. The FF module restores the low-level information lost in the high-level feature extraction process by simultaneously fusing features from different stages of the network, thereby enhancing the overall feature representation capability.

[0086] like Figure 4 As shown, the first FF module includes three parallel processing stages. Each stage processes features from the second C3 module, the third C3 module, and the CDCN module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage.

[0087] The first processing stage includes three parallel gOctCBR layers: a first gOctCBR layer, a second gOctCBR layer, and a third gOctCBR layer. The input of the first gOctCBR layer is connected to the output of the second C3 module, the input of the second gOctCBR layer is connected to the output of the third C3 module, and the input of the third gOctCBR layer is connected to the output of the seventh Conv module in the CDCN module. The outputs of the first, second, and third gOctCBR layers are all connected to the input of the first CSP module.

[0088] Second processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the second CSP module.

[0089] The third processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the third CSP module.

[0090] The first FF module is used to fuse features from the second C3 module, the third C3 module, and the CDCN module. The processing is similar for each stage in the FF module. Here, we take the processing of the first processing stage in the first FF module as an example. The method is as follows:

[0091] Y1 = f CSP1(Conv 11 (X1)⊕Conv 12 (Resize(X2))

[0092] ⊕Conv 13 (Resize(X3)))

[0093] Where Y1 is the output of the first CSP module, f CSP1 For the first CSP module, Conv 11 For the first gOctCBR layer, Conv 12 For the second gOctCBR layer, Conv 13 For the third gOctCBR layer, the Resize(·) operation is used to adjust the feature maps of different stages to the size suitable for the corresponding CSP module. X1 is the output feature map from the second C3 module, X2 is the output feature map from the third C3 module, and X3 = y2 is the output feature map from the CDCN module.

[0094] The second FF module includes three parallel processing stages corresponding to the first FF module. Each stage processes the output features from the first CSP module, the second CSP module, and the third CSP module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage.

[0095] The first processing stage includes parallel fourth, fifth, and sixth gOctCBR layers. The input of the fourth gOctCBR layer is connected to the output of the first CSP module, the input of the fifth gOctCBR layer is connected to the output of the second CSP module, and the input of the sixth gOctCBR layer is connected to the output of the third CSP module. The outputs of the fourth, fifth, and sixth gOctCBR layers are all connected to the input of the fourth CSP module.

[0096] Second processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the fifth CSP module.

[0097] Third processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the sixth CSP module.

[0098] The output of the fourth CSP module is connected to the first detection head, the output of the fifth CSP module is connected to the second detection head, and the output of the sixth CSP module is connected to the third detection head.

[0099] Among them, the first, second, and third gOctCBR layers do not share weights, and the fourth, fifth, and sixth gOctCBR layers do not share weights.

[0100] Each gOctCBR layer consists of a gOctConv layer, a batch normalization layer, and a PReLU activation function connected in sequence.

[0101] Each CSP module includes a first CBS layer and a second CBS layer in parallel. The output of the second CBS layer is connected to the input of the first Bottleneck layer, and the output of the first Bottleneck layer is connected to the input of the second Bottleneck layer. The outputs of the first and second Bottleneck layers are concatenated and then output through a third CBS layer. Each Bottleneck layer includes a fourth and a fifth CBS layer connected in sequence. The input of the fourth CBS layer is connected to the output of the fifth CBS layer in a skip connection. Each CBS layer includes a convolutional layer, a batch normalization layer, and a Silu activation function connected in sequence.

[0102] Unlike the sequential fusion strategies in traditional FPN and PANet modules, the FF module directly connects low-level and high-level features to achieve multi-scale fusion, preserving spatial details and minimizing information loss, thus improving feature transfer efficiency and robustness in snowy conditions. A key feature of the FF module is the integration of features enhanced by the CDCN module. These features possess spatial adaptability and deformation awareness, guiding the fusion process and enabling the network to highlight information-rich regions while suppressing irrelevant background noise, thereby improving feature discrimination capabilities in snowy scenes. The FF module uses the gOctConv layer for multi-frequency processing and reduces spatial redundancy. The fused features are then refined using the CSP module to ensure minimal computational overhead.

[0103] A key feature of the FF module is the integration of CDCN-enhanced features. These features are spatially adaptive and deformation-aware, guiding the fusion process and enabling the network to highlight information-rich regions and suppress irrelevant background noise, thereby improving feature discrimination capabilities in snowy scenes.

[0104] Target detection in snowy conditions is susceptible to the effects of background homogenization and low target contrast, significantly reducing detection performance. To address these challenges and improve the accuracy of SR-YOLO in snowy conditions, this invention proposes the OTA-Snow module. Specifically, this invention redesigns the YOLO anchor box mechanism, replacing the IoU-based local anchor box matching mechanism with a globally optimized allocation method based on optimal transmission (OTA-Snow) to better capture the appearance changes of targets obscured by snow. Subsequently, this invention constructs a cost matrix that fuses classification and regression losses and reweights it to adapt to the detection requirements of low-contrast targets. Furthermore, we introduce a snow impact modeling term to further enhance the expressive power of foreground features.

[0105] The method for tag allocation after the detection head, via the OTA-Snow optimal transmission allocation module, is as follows:

[0106] First, dynamically allocate estimates for each ground truth bounding box to obtain the number of candidate boxes k that each ground truth bounding box needs to match. i .

[0107] Specifically, the matching quality metric, such as IoU or classification confidence, is calculated for each candidate box and its corresponding ground truth box. Candidate boxes are then sorted from highest to lowest matching quality with the target ground truth box. The top-ranked candidate boxes are selected, and their matching quality metrics are summed to obtain the summed matching quality metric for that ground truth box. The summed matching quality metric is then rounded down to obtain the number of matches, k, for that ground truth box. i And set a lower limit constraint:

[0108]

[0109] Where max(1,·) means that at least one candidate box is assigned; Indicates rounding down; ∑ j Q ij Q represents the cumulative matching quality index. ij Let be the matching quality metric between candidate box j and its corresponding ground truth box i.

[0110] A fixed k-value cannot adapt to the diversity of targets in snowy weather, which may lead to insufficient anchor frames for large targets or redundant anchor frames for small targets. A dynamic k-value can adjust the number of anchor frames allocated according to the actual situation of the target, improve the flexibility of anchor frame allocation, ensure that each real frame has enough anchor frames for matching without excessive use, and avoid resource waste or insufficient matching.

[0111] Furthermore, based on the estimated number of candidate boxes k i Perform supply vector Update: For each ground truth bounding box, k will be selected from the candidate bounding boxes. iThe best-matching candidate bounding box is used as a positive sample of the ground truth bounding box, and the remaining candidate bounding boxes are all used as negative samples of the background. The required vector is then... Defined as d j =1, j=1,2,…,n, indicating that each candidate box can only select one true box. The supply vector update formula is:

[0112]

[0113] Furthermore, a multi-dimensional cost matrix is ​​constructed, including a classification cost matrix, a regression cost matrix, a center prior cost matrix, a snow influence cost matrix, and a background classification cost matrix.

[0114] Wherein, the classification cost matrix c cls The calculation method is as follows:

[0115]

[0116] in, For each element in the classification cost matrix, L cls To represent the classification loss, Focal Loss is used here. The classification score for the j-th candidate box is given. Let be the category label of the i-th real bounding box.

[0117] Regression cost matrix c reg The calculation formula is as follows:

[0118]

[0119] in, L represents an element in the regression cost matrix. reg For regression loss, IoU Loss is used. Let j be the position parameter of the j-th candidate box. represents the position parameter of the i-th ground truth bounding box.

[0120] Central prior cost matrix c cp The calculation method is as follows:

[0121]

[0122] in, For the elements in the central prior cost matrix, L cp For the center point loss, S j This is the j-th candidate box.

[0123] L cpThis represents the center point loss, used to measure the distance error between the center point of the candidate bounding box and the center point of the ground truth bounding box. Its main purpose is to guide the network's final output predicted bounding box to more accurately cover the target center, thereby improving the offset problem that easily occurs in snowy object detection. The calculation method is as follows:

[0124]

[0125] in, It is the candidate box S j The center coordinates, These are the center coordinates of the real bounding box i.

[0126] Snow impact cost matrix c snow The calculation method is as follows:

[0127]

[0128] in, L is an element in the snow impact cost matrix. snow Snow impact loss, where M represents the snow mask. The snow impact loss measures the degree of occlusion experienced by the ground truth bounding box in a snowy environment. The formula is:

[0129]

[0130] Where M(p) represents whether pixel p within the candidate box is covered by snow. This represents the set of pixels within the candidate bounding box area.

[0131] The background classification cost matrix is ​​calculated as follows:

[0132]

[0133] in, Let Ω be an element in the background classification cost matrix, and L be the background label. cls For classification loss, Focal Loss is used here;

[0134] Furthermore, the classification cost matrix, regression cost matrix, center prior cost matrix, and snow influence cost matrix are aggregated to obtain the foreground cost matrix c. fg And the background classification cost matrix As the last row and the foreground cost matrix c fg The global cost matrix c is obtained by concatenating the matrices.

[0135] Foreground cost matrix c fg for:

[0136] c fg =c cls +αc reg +c cp+c snow

[0137] Where α is the regression cost weight coefficient.

[0138] Furthermore, the optimal allocation strategy is calculated based on the Sinkhorn-Knopp algorithm:

[0139] Step 1: Initialize the scaling coefficients u0 and v0 in the Sinkhorn scaling vector to all be 1. Perform Sinkhorn-Knopp iterative calculation based on the global cost matrix c, supply vector s, and demand vector d. The calculation formula is as follows:

[0140]

[0141]

[0142] Where t is the number of iterations, γ is the regularization parameter, and v t,j The scaling factor for the j-th candidate box is used to adjust the total transmission amount of all ground truth boxes i to candidate box j, so that candidate box j matches the preset requirement d. j u t+1,i The scale factor corresponding to the i-th ground truth bounding box (real target or background) is used to adjust the total transmission amount of ground truth bounding box i to all candidate bounding boxes j, so that ground truth bounding box i matches the preset supply amount s. i .

[0143] Step 2: Iterate until the required number of iterations or the scaling factor v is reached. t,j and scaling factor u t+1,i When the change is less than the threshold, the final scaling coefficient u is obtained. last,i and the final scaling factor v last,j :

[0144] ||v t+1,j -v t,j ||<ε and||u t+1,i -u t,i ||<ε.

[0145] Step 3: Based on the final scaling coefficient u after iteration last,i and the final scaling factor v last,j Calculate the optimal allocation strategy:

[0146]

[0147] in, The probability that candidate box j matches the ground truth box i is called the predicted box. The candidate box after label assignment is called the predicted box.

[0148] After obtaining the optimal allocation strategy π *Each predicted bounding box is then assigned a positive / negative sample identity, and the loss function can be calculated as follows:

[0149]

[0150] Among them, L cls The classification loss is represented by Focal Loss, calculated for all predicted bounding boxes (positive and negative samples); L reg The regression loss uses IoU Loss, calculated only for positive samples; λ cls λ reg This represents the loss weighting coefficient.

[0151] S3: Train and test the SR-YOLO target detection network using a dataset of severe weather images to obtain a trained SR-YOLO target detection network;

[0152] S4: Input the severe weather image to be detected into the trained SR-YOLO object detection network to obtain the object detection results.

[0153] Experimental implementation and details:

[0154] To verify the effectiveness of the proposed method, this invention implements the SR-YOLO model using CUDA 11.1 based on the PyTorch 1.8.1 deep learning framework. During training, a stochastic gradient descent (SGD) optimizer was used, with an initial learning rate of 0.01, a momentum parameter of 0.937, and a weight decay of 0.005. The batch size was set to 32, and for ease of training and validation, both training and testing images were adjusted to 640×640 pixels. All experiments were performed on a single NVIDIA GeForce RTX 3090 GPU equipped with 24GB of video memory.

[0155] The dataset is as described in step S1. The evaluation metrics use mean average precision (mAP) to assess detection accuracy and frames per second (FPS) to measure inference speed.

[0156] Performance on the RSOD dataset:

[0157] To evaluate the SR-YOLO model, this invention conducted qualitative and quantitative analyses. The comparison objects included several lightweight detectors optimized for mobile devices, such as YOLOv5s, YOLOv7-tiny, SSD300, YOLOv8s, YOLO11s, YOLOX-s, CFYOLO, R-YOLO, and the RDMNet model.

[0158] Table 1 shows the mAP results on the RSOD dataset. The SR-YOLO model performs best among all target categories, with an mAP of 0.801 for traffic lights, demonstrating strong detection capabilities.

[0159] Table 1 Comparison of mAP metrics for different methods on the RSOD dataset

[0160]

[0161] Table 2 Comparison of SR-YOLO and state-of-the-art methods on the RTTS dataset.

[0162]

[0163] like Figure 5 As shown, the SR-YOLO model achieves higher confidence levels compared to YOLOv8s, effectively reducing false negatives and false positives. The FF module recovers crucial high-level details, the CDCN module enhances feature capture capabilities in complex scenes, and OTA-Snow refines the detection of snow-covered targets. These modules work synergistically to improve overall mAP performance. Figure 6 The feature map visualization on the RSOD dataset is shown, demonstrating that the SR-YOLO model can capture stronger color cues and more accurate target localization, supporting real-time and accurate detection.

[0164] Performance in other severe weather conditions:

[0165] To further evaluate the robustness of the model, experiments were conducted on the RTTS dataset. As shown in Table 2, the SR-YOLO model achieved the highest overall mAP of 0.793, demonstrating excellent performance in the detection of bicycle (0.750) and motorcycle (0.813) categories, thus validating its effectiveness under hazy conditions. Figure 7 This visually demonstrates the superior detection accuracy of the SR-YOLO model compared to other methods.

[0166] Table 3 presents the results on the rRain dataset. SR-YOLO leads with a highest mAP of 69.1%, particularly excelling in the car (0.788) and bicycle (0.771) categories. These results highlight the model's robustness and accuracy in rainy environments, surpassing existing methods.

[0167] Table 3 Quantitative Comparison of rRain Datasets

[0168]

[0169] Ablation experiment analysis:

[0170] The effectiveness of each module was verified through ablation experiments on the RSOD dataset, as shown in Tables 4 and 5. The introduction of the FF module improved mAP by 0.3%, while combining OTA-Snow with the FF module resulted in a 0.9% improvement. The complete integration of all three modules significantly improved mAP by 6.5%.

[0171] As shown in Table 4, compared with the DCNv2 module, the CDCN module further improves the mAP from 66.3% to 68.9%. This improvement is mainly attributed to the adaptive deformable convolution of the CDCN module, which can reduce background noise and enhance detection capabilities under snowy conditions.

[0172] These three modules work together: the OTA-Snow strategy refines label assignment in complex environments, the CDCN captures geometric changes caused by snow cover and enhances feature representation, and the FF module recovers fine-grained visual information lost in high-level feature extraction. Together, they form a complementary framework that significantly enhances the model's robustness and detection accuracy in snowy environments, validating its overall effectiveness.

[0173] Table 4 Comparison of DCNv2 and CDCN modules on the RSOD dataset.

[0174]

[0175] Table 5 Ablation Experiment Results (RSOD Dataset)

[0176]

[0177] Efficiency Analysis:

[0178] Table 6 compares the efficiency of different methods. Our method has a processing time of 0.0234 seconds per frame, achieving a frame rate of 42.7 FPS. While not the fastest in pure processing speed, it maintains a good balance between efficiency and accuracy. Notably, while ensuring competitive detection performance, our method keeps the number of parameters and computational cost at a reasonable level, demonstrating its high practicality and reliability in real-world applications. It is particularly suitable for scenarios with high requirements for both processing speed and detection accuracy, providing strong support for real-time and effective decision-making.

[0179] In summary, this invention proposes the SR-YOLO model, integrating the OTA-Snow, CDCN, and FF modules into a unified architecture, enabling the model to effectively address the challenges of visual degradation and geometric deformation caused by snow accumulation. Specifically, OTA-Snow improves supervision quality through optimized label assignment under complex snowy conditions; guided by this optimized assignment, the CDCN module enhances feature extraction through adaptive sampling, better capturing snow-induced deformation and occlusion; based on the feature representation generated by the CDCN module, the FF module recovers low-level visual information and strengthens semantic consistency through efficient multi-scale feature fusion. The collaborative design of these three modules constitutes a unified detection process, enabling label assignment, deformation modeling, and feature integration to work together to improve detection accuracy and robustness. Experimental results on the RSOD, RTTS, and rRain datasets validate the effectiveness of the SR-YOLO model, highlighting its potential as a lightweight and practical real-time target detection solution in severe weather.

[0180] Table 6 Efficiency Analysis

[0181]

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robust target detection method suitable for severe weather conditions, characterized in that, The steps are as follows: S1: Obtain a dataset of severe weather images; S2: Construct the SR-YOLO target detection network. Based on the YOLOv5s model, introduce the CDCN cascaded deformable convolution module into the backbone network, replace the neck network with the cascaded FF feature fusion module, and implement label allocation through the OTA-Snow snow-day optimal transmission allocation module after the detection head. S3: Train and test the SR-YOLO target detection network using a dataset of severe weather images to obtain a trained SR-YOLO target detection network; S4: Input the severe weather image to be detected into the trained SR-YOLO object detection network to obtain the object detection results.

2. The robust target detection method for adverse weather conditions according to claim 1, characterized in that, The backbone network includes a Focus module, a first Conv module, a first C3 module, a second Conv module, a second C3 module, a third Conv module, a third C3 module, a fourth Conv module, an SPP module, and a CDCN module connected in sequence. The CDCN module adaptively adjusts the shape of the convolution kernel and the sampling position through deformable convolution; The neck network includes a first FF module and a second FF module connected to each other. The first FF module is connected to the second C3 module, the third C3 module and the CDCN module respectively. The first FF module and the second FF module recover the low-level information lost in the high-level feature extraction process by simultaneously fusing features from different stages of the network.

3. The robust target detection method for adverse weather conditions according to claim 2, characterized in that, The CDCN module includes a parallel fifth Conv module and a sixth Conv module. The inputs of both the fifth and sixth Conv modules are connected to the SPP module, and the output of the fifth Conv module is connected to the DCN module. The output of the sixth Conv module is concatenated with the output of the DCN module and then output to the first FF module through the seventh Conv module. For the input feature map from the fifth Conv module, the DCN module uses deformable convolution to adaptively adjust the convolution sampling position and weights to extract geometrically adaptive local feature maps. For the local feature map obtained by deformable convolution, deformable pooling is used to adaptively adjust the pooling sampling position and weights to obtain an enhanced geometrically perceptual feature map.

4. The robust target detection method for adverse weather conditions according to claim 3, characterized in that, The first FF module includes three parallel processing stages. Each stage processes features from the second C3 module, the third C3 module, and the CDCN module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage. The first processing stage includes three parallel gOctCBR layers: a first gOctCBR layer, a second gOctCBR layer, and a third gOctCBR layer. The input of the first gOctCBR layer is connected to the output of the second C3 module, the input of the second gOctCBR layer is connected to the output of the third C3 module, and the input of the third gOctCBR layer is connected to the output of the seventh Conv module in the CDCN module. The outputs of the first, second, and third gOctCBR layers are all connected to the input of the first CSP module. The second processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the second CSP module. The third processing stage: The outputs of the first gOctCBR layer, the second gOctCBR layer, and the third gOctCBR layer are all connected to the input of the third CSP module.

5. The robust target detection method for adverse weather conditions according to claim 4, characterized in that, The second FF module includes three parallel processing stages corresponding to the first FF module. Each stage processes the output features from the first CSP module, the second CSP module, and the third CSP module and performs a resizing operation to adjust the input feature maps of different sizes to be suitable for the processing size of the corresponding stage. The first processing stage includes parallel fourth, fifth, and sixth gOctCBR layers. The input of the fourth gOctCBR layer is connected to the output of the first CSP module, the input of the fifth gOctCBR layer is connected to the output of the second CSP module, and the input of the sixth gOctCBR layer is connected to the output of the third CSP module. The outputs of the fourth, fifth, and sixth gOctCBR layers are all connected to the input of the fourth CSP module. The second processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the fifth CSP module. The third processing stage: The outputs of the fourth gOctCBR layer, the fifth gOctCBR layer, and the sixth gOctCBR layer are all connected to the input of the sixth CSP module. The output of the fourth CSP module is connected to the first detection head, the output of the fifth CSP module is connected to the second detection head, and the output of the sixth CSP module is connected to the third detection head.

6. The robust target detection method for adverse weather conditions according to claim 5, characterized in that, Each gOctCBR layer consists of a gOctConv layer, a batch normalization layer, and a PReLU activation function connected in sequence. Each CSP module includes a first CBS layer and a second CBS layer in parallel. The output of the second CBS layer is connected to the input of the first Bottleneck layer, and the output of the first Bottleneck layer is connected to the input of the second Bottleneck layer. The outputs of the first CBS layer and the second Bottleneck layer are concatenated and then output through a third CBS layer. Each Bottleneck layer consists of a fourth CBS layer and a fifth CBS layer connected in sequence. The input of the fourth CBS layer is skipped to the output of the fifth CBS layer. Each CBS layer consists of a convolutional layer, a batch normalization layer, and a Silu activation function connected in sequence.

7. The robust target detection method for adverse weather conditions according to any one of claims 1 to 6, characterized in that, The method for tag allocation after the detection head, via the OTA-Snow module's optimal snow weather transmission and allocation module, is as follows: (1) Dynamically allocate estimates for each ground truth box to obtain the number of candidate boxes that each ground truth box needs to match; (2) Define the requirement vector based on the number of candidate boxes. Supply vectors are generated based on the estimated number of candidate boxes through dynamic allocation. Update: n is the number of candidate boxes, and m is the number of ground truth boxes; (3) Construct a multi-dimensional cost matrix, including a classification cost matrix, a regression cost matrix, a center prior cost matrix, a snow influence cost matrix, and a background classification cost matrix; (4) Aggregate the classification cost matrix, regression cost matrix, center prior cost matrix and snow influence cost matrix to obtain the foreground cost matrix, and concatenate the background classification cost matrix as the last row with the foreground cost matrix to obtain the global cost matrix c. (4) Calculate the optimal allocation strategy based on the Sinkhorn-Knopp algorithm.

8. The robust target detection method for adverse weather conditions according to claim 7, characterized in that, The method for calculating the optimal allocation strategy based on the Sinkhorn-Knopp algorithm is as follows: Step 1: Initialize the Sinkhorn scaling coefficients u0 and v0, and perform Sinkhorn-Knopp iterative calculations based on the global cost matrix c, supply vector s, and demand vector d; Step 2: Iterate until the required number of iterations or the scaling factor v is reached. t,j and scaling factor u t+1,i When the change is less than the threshold, the final scaling coefficient u is obtained. last,i and the final scaling factor v last,j ; Step 3: Based on the final scaling coefficient u after iteration last,i and the final scaling factor v last,j Calculate the optimal allocation strategy: in, Let be the probability that candidate box j matches the true box i, which represents the optimal allocation strategy.

9. The robust target detection method for adverse weather conditions according to claim 8, characterized in that, The method for dynamically assigning estimates to each ground truth bounding box is as follows: Calculate the matching quality metric (IoU) between each candidate box and its corresponding ground truth box, such as IoU or classification confidence. Sort the candidate boxes according to their matching quality with the target ground truth box from highest to lowest. Select the top-ranked candidate boxes and sum their matching quality metrics to obtain the summed matching quality metric for that ground truth box. Round the summed matching quality metric to the nearest integer as the number of matches, k, for that ground truth box. i And set a lower limit constraint: Where max(1,·) represents the lower bound constraint of assigning at least one candidate box; Indicates rounding down; ∑ j Q ij Q represents the cumulative matching quality index. ij Let be the quality metric for matching candidate box j with the corresponding ground truth box i; Supply vectors are generated based on the estimated number of candidate boxes through dynamic allocation. The update method is as follows: The classification cost matrix is ​​calculated using Focal Loss, the regression cost matrix is ​​calculated using IoU Loss, the center prior cost matrix is ​​calculated using center point loss, the snow influence cost matrix is ​​calculated using snow influence loss, and the background classification cost matrix is ​​calculated using Focal Loss.

10. The robust target detection method for adverse weather conditions according to claim 8 or 9, characterized in that, After obtaining the optimal allocation strategy, the total loss function is calculated as follows: Among them, L cls Focal Loss, used as the classification loss, L reg For IoU Loss, used as the regression loss, λ cls , λ reg These represent the weighting coefficients for classification loss and regression loss, respectively. The Sinkhorn-Knopp iterative calculation formula is: Among them, s i For the elements in the supply vector, d j Let t be the element in the demand vector, γ be the iteration number, and v be the regularization parameter. t,j The scaling factor for the j-th candidate box is used to adjust the total transmission amount of all ground truth boxes i to candidate box j, so that candidate box j matches the preset requirement d. j u t+1,i The scaling factor is the value corresponding to the i-th ground truth box, used to adjust the total amount of transmission from ground truth box i to all candidate boxes j, so that ground truth box i matches the preset supply.

Images