Training methods, detection methods, and devices for target detection models in foggy roads

Abstract

This invention belongs to the field of autonomous driving environmental perception and aims to improve object detection performance under foggy and normal weather conditions. Specifically, it relates to a training method, detection method, and apparatus for a foggy road object detection model. The foggy road object detection model employs a knowledge distillation model and includes a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model). The object detection subnetwork is connected to the defogging feature selection subnetwork through the feature absorption subnetwork to form a unified framework, thereby improving object detection performance. This invention trains the teacher model in stages and uses an absorption checker to allow the student model to learn the F function in the defogging feature selection subnetwork. SC Features were introduced to improve the performance of the student model in feature extraction, optimize the training effect of the foggy road object detection model, and improve its performance. The training time for the foggy road object detection model was also reduced.

Description

Technical Field

[0001] This invention relates to the field of environmental perception for autonomous driving, specifically to a training method, detection method, and apparatus for a target detection model in foggy roads; the method uses a knowledge distillation-based network structure to detect pedestrians in foggy images with high real-time performance. Background Technology

[0002] In recent years, autonomous (assisted) driving technology has become a core trend in the automotive industry. To build reliable autonomous (assisted) driving technologies to assist human driving safely and reduce traffic accidents, many essential functions need to be equipped into these systems, including but not limited to object detection, target tracking, and behavior recognition. Object detection is one of the most critical and important tasks in autonomous (assisted) driving technology because fast and accurate detection not only supports precise navigation but also addresses potential hazards in complex driving environments. In foggy weather, driver visibility is impaired, increasing the risk of traffic accidents. End-to-end foggy image target detection models can help autonomous driving systems or assisted driving systems identify pedestrians and vehicles on the road in low-visibility conditions, improving driving safety.

[0003] Currently, mainstream object detection models for foggy images fall into two-stage approaches. First, the foggy dataset is defogging to generate a clear dataset. Then, an object detection network is used on the clear dataset to detect pedestrians and vehicles. This method suffers from poor real-time performance and cannot be used as an in-vehicle module for real-time detection in current foggy environments. Furthermore, the preprocessing step of restoring the foggy image to a clear one reduces the detection speed. Summary of the Invention

[0004] This invention aims to improve the performance of road target object detection under both foggy and normal weather conditions. It proposes a training method, detection method, and apparatus for a foggy road target detection model. The model is constructed by combining a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model), forming a unified framework to improve road target object detection performance. In this invention, the absorption checker in the feature absorption subnetwork absorbs beneficial features from the defogging feature selection subnetwork and shares them with the object detection subnetwork to better learn object classification and object localization.

[0005] In a first aspect, the present invention proposes a training method for a foggy road target detection model, wherein the foggy road target detection model includes a feature absorption subnetwork as a student model, an object detection subnetwork, and a defogging feature selection subnetwork as a teacher model, wherein the object detection subnetwork includes a neck network and a head network; the method includes:

[0006] Acquire a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box for the target region;

[0007] The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss.

[0008] F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AM Features, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss;

[0009] F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss;

[0010] F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

[0011] Furthermore, the dehazing feature selection subnet includes a set of differential convolutional layers with five parallel convolutional layers, a content-guided attention mechanism, and an improved Mamba mechanism; wherein, the set of differential convolutional layers with five parallel convolutional layers includes original convolution, central difference convolution, angular difference convolution, horizontal difference convolution, and vertical difference convolution; the content-guided attention mechanism includes a channel attention module and a spatial attention module.

[0012] Furthermore, the dehazing feature selection subnet also includes converting the differential convolution set of the five parallel convolutional layers into a single standard convolutional layer through the additive nature of the convolutional layers.

[0013] Furthermore, the neck network and the head network implement multi-scale object detection on three different layers.

[0014] In a second aspect, the present invention provides a method for detecting targets on foggy roads, comprising:

[0015] Acquire the image to be detected;

[0016] The image to be detected is input into the trained foggy road target detection model to detect the image;

[0017] The trained foggy road target detection model is trained based on a training method for a foggy road target detection model as described in the first aspect of the present invention. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models.

[0018] The image to be detected is input into the feature absorption subnetwork that serves as the student model to obtain the F-value of the image to be detected. M feature;

[0019] F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

[0020] Furthermore, the F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection results of the image to be detected, including:

[0021] F of the image to be detected M The features are input into the neck network of the object detection subnet to obtain the fused features of the image to be detected;

[0022] The fused features of the image to be detected are input into the head network of the object detection network to obtain the target detection result of the image to be detected.

[0023] In a third aspect, the present invention provides a training apparatus for a foggy road target detection model, wherein the foggy road target detection model includes a feature absorption subnetwork as a student model, an object detection subnetwork as a teacher model, and a defogging feature selection subnetwork as a teacher model, and the apparatus includes:

[0024] The first acquisition module is used to acquire a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box of the target region;

[0025] The training module is used to iteratively train the foggy road target detection model using the foggy image and the fog-free image until a fully trained foggy road target detection model is obtained; wherein:

[0026] The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss.

[0027] F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM feature;

[0028] F through a foggy image A Features and F of foggy images AM Features, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss;

[0029] F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss;

[0030] F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

[0031] In a fourth aspect, the present invention provides a target detection device for foggy roads, the device comprising:

[0032] The second acquisition module is used to acquire the image to be detected;

[0033] The trained foggy road target detection model is trained based on a training method for a foggy road target detection model as described in the second aspect of the present invention. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models.

[0034] The processing module is used to input the image to be detected into the feature absorption subnet serving as the student model, and obtain the F-value of the image to be detected. M Features; the F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

[0035] Furthermore, the processing module includes:

[0036] The first processing unit is used to process the F of the image to be detected. M The features are input into the neck network of the object detection subnet to obtain the fused features of the image to be detected;

[0037] The second processing unit is used to input the fused features of the image to be detected into the head network of the object detection network to obtain the target detection result of the image to be detected.

[0038] The beneficial effects of this invention are as follows:

[0039] This invention provides a training method, detection method, and apparatus for a foggy road target detection model. The foggy road target detection model adopts a knowledge distillation architecture, trains the teacher model in stages, and uses an absorption checker to improve the F values ​​in the student model. M Module learning of F in the feature selection subnet for dehazing SC Features help improve the performance of student models in feature extraction, optimize the training effect of foggy road object detection models, and improve their performance. It also reduces the training time of foggy road object detection models. In actual detection, only the trained feature absorption subnet and object detection subnet are used to detect pedestrians and vehicles in foggy images. This improves the real-time detection speed and performance of foggy road object detection models. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a target detection model structure for foggy roads provided in an embodiment of the present invention;

[0041] Figure 2 A flowchart illustrating a training method for a target detection model in foggy roads, provided in an embodiment of the present invention;

[0042] Figure 3 This is the internal structure of the dehazing feature selection subnet provided in the embodiments of the present invention;

[0043] Figure 4 This is the internal structure of the differential convolution set in the dehazing feature selection subnet provided in this embodiment of the invention;

[0044] Figure 5 This is the internal structure of feature fusion in the dehazing feature selection subnet provided in the embodiments of the present invention;

[0045] Figure 6 This is a flowchart illustrating a target detection method for foggy roads provided in an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of a training device for a target detection model in foggy roads, provided in an embodiment of the present invention.

[0047] Figure 8 This is a schematic diagram of a target detection device for foggy roads provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] This invention provides a training method, detection method, and apparatus for a target detection model in foggy roads. The process involves acquiring foggy and fog-free images; the foggy and fog-free images are a pair of target images from the same road scene, and the foggy image contains a target category and an initial bounding box for the target region; the foggy and fog-free images are then processed through a dehazing feature selection subnet to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnetwork is trained using the first loss; the F-value of the hazy image is then used to calculate the first loss. SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image.M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AM Features are used to calculate the first hinge loss. The parameters of the feature modification modules for the dehazing feature selection subnet and the feature absorption subnet are fixed. The absorption checker of the feature absorption subnet is trained using the first hinge loss. The F-value of the hazy image is then used to... AM Features are used to calculate the second hinge loss. The parameters of the absorption checker in the feature absorption subnet are fixed, and the feature modification module of the feature absorption subnet is trained using the second hinge loss. The F-value of the foggy image is then used to... M Features are processed through an object detection subnet to obtain candidate bounding boxes for target regions and target detection categories in the foggy image. The parameters of the feature modification module in the fixed feature absorption subnet are used to calculate a second loss based on the initial bounding boxes and candidate bounding boxes of the target regions in the foggy image, as well as the target type and target detection type. This second loss is then used to train the foggy road target detection model. Furthermore, the trained foggy road target detection model is used to detect pedestrians and vehicles in the image to be detected, thus obtaining the target detection results for the image.

[0050] This invention provides a training method, detection method, and apparatus for a foggy road target detection model. The foggy road target detection model adopts a knowledge distillation architecture, trains the teacher model in stages, and uses an absorption checker to improve the F values ​​in the student model. M Module learning of F in the feature selection subnet for dehazing SC Features help improve the performance of student models in feature extraction, optimize the training effect of foggy road object detection models, and improve their performance. It also reduces the training time of foggy road object detection models. In actual detection, only the trained feature absorption subnet and object detection subnet are used to detect pedestrians and vehicles in foggy images. This improves the real-time detection speed and performance of foggy road object detection models.

[0051] The training method, detection method, and apparatus for a foggy road target detection model provided in this embodiment of the invention will be described in detail below. The training method and detection method for a foggy road target detection model provided in this embodiment of the invention can be applied to electronic devices, such as server devices, smart terminal devices, etc. The training method and detection method for a foggy road target detection model provided in this disclosure can be applied to any foggy road scenario requiring the detection of pedestrians or vehicles.

[0052] See Figure 1 , Figure 1 This is a schematic diagram of a foggy road target detection model provided in an embodiment of the present invention. The foggy road target detection model adopts a knowledge distillation model and includes a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model). The object detection subnetwork is connected to the defogging feature selection subnetwork through the feature absorption subnetwork to form a unified framework to improve object detection performance. The object detection subnetwork includes a neck network and a head network. During the training phase, the detection dataset, which includes foggy and fog-free images, is used as input. The teacher model helps the student model extract features better. Both the teacher and student models can serve as the backbone of the entire network to extract feature information from the input images. The object detection subnetwork, acting as the neck and head of the entire network, is used for multi-scale fusion and feature detection of the extracted feature information. The absorption checker in the feature absorption subnetwork absorbs beneficial features from the defogging feature selection subnetwork and shares them with the object detection subnetwork to better learn object classification and object localization. During the detection phase, the trained student model and object detection subnetwork are directly used to complete the detection and classification of the input images.

[0053] See Figure 2 , Figure 2 A flowchart illustrating a training method for a foggy road target detection model provided in this embodiment of the invention includes the following steps:

[0054] 101. Obtain a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box for the target region;

[0055] The foggy image can be taken in foggy weather for a road scene, while the corresponding fog-free image can be taken in clear weather for the same road scene. Due to scattering, the colors in the foggy image may deviate from their original colors, details are blurred, object outlines are unclear, and the gray level distribution in the color histogram is relatively concentrated, with gray dominating the entire image. In contrast, the fog-free image has complete color information, rich and natural colors, complete and clear details, distinct object outlines, and a wide gray level distribution in the color histogram, capable of displaying more color variations.

[0056] In some embodiments, pedestrians and vehicles in foggy images are labeled to obtain labeled objects corresponding to each foggy image. The labeled images contain label information, which includes target category and initial bounding box of target region. The target category is used to indicate the category of pedestrians and vehicles in foggy images, and the initial bounding box of target region is used to indicate the position of pedestrians and vehicles in foggy images.

[0057] In some embodiments, the present invention may further perform cleaning and augmentation processing on the foggy images, and use the processed images as sample training image data for training the foggy road target detection model, so as to increase the robustness of the model.

[0058] 102. The foggy image and the defogging image are processed through a defogging feature selection sub-mesh to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss.

[0059] In some embodiments, since the foggy image and the fog-free image are image pairs from the same road scene, the fog-free image can optimize the feature extraction process of the foggy image in a manner similar to a reference image. After inputting the foggy image and the fog-free image into the dehazing feature selection subnetwork, the dehazing feature selection subnetwork can extract the F-values ​​of the foggy image and the fog-free image respectively. SC feature.

[0060] In some embodiments, the input hazy and hazy images can be preprocessed, such as resizing and normalizing, to adapt to the input requirements of the dehazing feature selection subnetwork. The dehazing feature selection subnetwork may include multiple convolutional layers, pooling layers, activation function layers, etc., and these hierarchical structures are used to extract features from the image. The last layer or several layers of the network may contain fully connected layers for processing the extracted features. SC Features are mapped onto feature vectors.

[0061] In some embodiments, the dehazing feature selection subnet has a more complex hierarchical structure, including more layers, a more complex structure, and regularization techniques, to improve the accuracy and robustness of feature extraction. Furthermore, F SC The dimensions and specific content of the features depend on the hierarchical design of the dehazing feature selection subnet and the training data.

[0062] For example, such as Figure 3As shown, the dehazing feature selection subnetwork includes a differential convolution set of five parallel convolutional layers, a content-guided attention mechanism, and an improved Mamba mechanism. In the dehazing feature selection subnetwork, the differential convolution set of these five parallel convolutional layers, the content-guided attention mechanism, and the improved Mamba mechanism work together to extract effective dehazing features.

[0063] For example, such as Figure 4 The differential convolution set of the five parallel convolutional layers includes: original convolution (VC), central difference convolution (CDC), angular difference convolution (ADC), horizontal difference convolution (HDC), and vertical difference convolution (VDC). The content-guided attention mechanism includes a channel attention module and a spatial attention module.

[0064] In some preferred embodiments, it is considered that deploying five parallel convolutional layers for feature extraction would unnecessarily increase parameters and inference time. Embodiments of the present invention attempt to utilize the additive nature of convolutional layers to simplify the parallel deployment of convolutions into a single standard convolution. The dehazing feature selection subnetwork further includes transforming the differential convolution set of the five parallel convolutional layers into a single standard convolutional layer through the additive nature of the convolutional layers. The formula is as follows:

[0065]

[0066] Among them, F out F represents the output feature. in k represents the input features. i＝1:5 These represent the kernels of VC, CDC, ADC, HDC, and VDC, respectively. * indicates the convolution operation, and K... cut This represents the kernel that combines parallel convolutions after the transformation.

[0067] For example, such as Figure 5 As shown, the content-guided attention mechanism module consists of channel attention and spatial attention, which are placed sequentially to calculate the attention weight in the channel and spatial dimensions. Channel attention calculates the channel vector (i.e., Wc∈R). c×1×1 To recalibrate features. Spatial attention computes the spatial importance map (SIM), i.e., Ws∈R. H×W It adaptively indicates the importance level of different areas. The content-guided attention mechanism module treats different channels and pixels unequally, thereby improving dehazing performance.

[0068] In this embodiment of the invention, the F-value of the foggy image is calculated. SC F features and fog-free images SCThe feature loss between features is the first loss. The training parameters of the dehazing feature selection subnet are adjusted according to the calculated feature loss until the preset termination condition is met, and the trained dehazing feature selection subnet is obtained.

[0069] The preset termination condition can be a pre-set number of iterations or the loss reaching a pre-set loss threshold.

[0070] For example, the loss used to train the dehazing feature selection subnet is designed by minimizing the pixel difference between the predicted haze-free image J and the corresponding ground truth GT. Embodiments of the invention may choose an L1 loss function (i.e., mean absolute error) to drive the training.

[0071]

[0072] 103. The F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM feature;

[0073] In some embodiments, the absorption inspector may be designed as follows:

[0074] Table 1 Internal structure of the absorption inspector

[0075]

[0076] In this embodiment of the invention, the feature absorption subnet has an F-value relative to the feature selection subnet. SC Feature f SC The feature absorption subnet is responsible for and shares data with the object detection subnet during training to improve detection performance under foggy weather conditions. To achieve this, the spectral normalization principle of GAN is adopted, and the feature absorption subnet includes an absorption checker and a feature modification module.

[0077] The absorption checker is constructed to absorb F. SC The beneficial features are identified and transferred to the feature modification module to enhance its functionality. In other words, the F-values ​​of the foggy image are modified. SC The features are trained, and the functional absorption subnet uses an absorption checker to understand which features are related to F. SC Features are similar. Then, the absorption checker provides feedback to the feature change module to create F features that cannot be distinguished by the absorption checker. MFeatures, the F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features that enable the absorption inspector to distinguish F M Features and F AM Features. In the proposed absorption checker, a weight-normalized spectral normalization technique is employed to facilitate efficient learning and stable training.

[0078] 104. F through a foggy image A Features and F of foggy images AM Features, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss;

[0079] In this embodiment of the invention, a first hinge loss is used to train the absorption checker of the feature absorption subnet. Hinge loss is a loss function commonly used in machine learning for the maximum margin classification problem, especially in Support Vector Machines (SVMs). M achine,SV M The first hinge loss is used to measure the difference between the model's predicted values ​​and the true values, and attempts to maximize the margin between correctly classified samples and incorrectly classified samples. The formula for the first hinge loss is as follows:

[0080]

[0081] Where A is the absorption checker, FM is the feature change module, and... Indicates feature change module F M The weight of is frozen during training of the absorption checker A; x represents the input foggy image, and A(x) represents the F of the foggy image. A Feature; z represents the input foggy image, This represents the F value of a foggy image. M Features indicate that the input data has been processed by the feature modification module. This represents the F value of a foggy image. AM The feature indicates that the input data is processed by the change module and then by the absorption checker; E x～qdata(x) Let x follow a distribution qdata(x), E z～pdata(z)This indicates that z follows the distribution of qdata(z); qdata(x) represents the set of foggy image samples applicable to the dehazing feature selection subnet, and qdata(z) represents the set of foggy image samples applicable to the feature modification module.

[0082] In this embodiment of the invention, the F-value of the foggy image is calculated. A Features and F of foggy images AM The hinge loss of the feature is the first hinge loss. The training parameters of the absorption checker of the feature absorption subnet are adjusted according to the calculated hinge loss until the preset termination condition is met, and the trained absorption checker is obtained.

[0083] The preset termination condition can be a pre-set number of iterations or the loss reaching a pre-set loss threshold.

[0084] 105. F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss;

[0085] In this embodiment of the invention, the purpose of the feature mutation module is to extract features from the input image and understand the features of the absorption checker to effectively improve the accuracy of object classification and object localization. The feature mutation module is equipped with certain layers of the basic detector, rather than being built separately. For example, the feature mutation module can be constructed using the first four residual blocks of Yolov8. To train the feature mutation module based on feedback from the absorption checker to improve its functional quality, all weights of all absorption checkers are fixed during training. A second hinge loss is used to train the feature mutation module of the feature absorption subnet, and the formula for the second hinge loss is as follows:

[0086]

[0087] Where FM is the characteristic mutation module and A is the absorption checker; This indicates that the weight of A is frozen during the training of the feature mutation module; z represents the input image of the FM module; This represents the F value of a foggy image. M feature, This represents the F value of a foggy image. AM feature.

[0088] In this embodiment of the invention, the F-value of the foggy image is calculated. AM The hinge loss of the feature is the second hinge loss. The training parameters of the feature modification module of the feature absorption subnet are adjusted according to the calculated hinge loss until the preset termination condition is met, and the trained feature modification module is obtained.

[0089] The preset termination condition can be a pre-set number of iterations or the loss reaching a pre-set loss threshold.

[0090] 106. The F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

[0091] In this embodiment of the invention, since the object detection subnetwork includes a neck network and a head network, it can perform multi-scale fusion on the features extracted in the above steps, and then classify the fused features. For example, multi-scale object detection can be implemented on three different layers with sizes of 13×13, 26×26, and 52×52. Finally, non-maximum suppression is used for the final object detection. The second loss function of the object detection subnetwork is shown below:

[0092] L od-net =L coord +L hw -L conf -L confno -L cls (4)

[0093] Among them, L od-net This indicates the second loss.

[0094] L coord The coordinate loss of the bounding box relative to the coordinate center (x, y) is:

[0095]

[0096] L hw This is a loss in both the width and height of the bounding box:

[0097]

[0098] L conf It is the confidence loss within the bounding box of the existing object:

[0099]

[0100] L confno The confidence loss is for bounding boxes without existing objects:

[0101]

[0102] L clsThe classification loss is the value of the cell containing the object.

[0103]

[0104] Among them, g 2 'b' represents the number of grid cells; 'b' represents the number of bounding boxes predicted for each cell.

[0105] In this embodiment of the invention, the prediction loss of the initial bounding box and the candidate bounding box of the target region in the foggy image, as well as the classification loss of the target type and the target detection type, are calculated. The training parameters of the object detection subnet are adjusted according to the calculated prediction loss and classification loss until the preset termination condition is met, and the trained object detection subnet is obtained.

[0106] The preset termination condition can be a pre-set number of iterations or the loss reaching a pre-set loss threshold.

[0107] Based on the training method of the above-mentioned target detection model for foggy roads, see [link to training method]. Figure 6 , Figure 6 A flowchart illustrating a target detection method for foggy roads provided in this embodiment of the invention includes the following steps:

[0108] 201. Obtain the image to be detected;

[0109] The image to be detected can be any foggy or fog-free image that requires road target detection.

[0110] 202. Input the image to be detected into the trained foggy road target detection model to detect the image;

[0111] The trained foggy road target detection model is trained based on a training method for a foggy road target detection model as described in the first aspect of the present invention. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models.

[0112] 203. Input the image to be detected into the feature absorption subnet serving as the student model to obtain the F-value of the image to be detected. M feature;

[0113] 204. The F-value of the image to be detected... M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

[0114] The target detection results include: category detection results and location detection results. The category detection results are used to indicate the category of detected objects such as pedestrians and vehicles in the image to be detected, and the location detection results are used to indicate the location of detected objects such as pedestrians and vehicles in the image to be detected.

[0115] In this embodiment of the invention, the speed of the trained foggy road target detection model is improved, thereby improving the detection efficiency and accuracy in foggy road scenes when using the trained foggy road target detection model to detect objects in the image to be detected.

[0116] This invention also provides a training device for a foggy road target detection model. The foggy road target detection model includes a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model). See [link to relevant documentation]. Figure 7 The device includes:

[0117] The first acquisition module 301 is used to acquire a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box of the target area;

[0118] Training module 302 is used to iteratively train the foggy road target detection model using the foggy image and the fog-free image until a trained foggy road target detection model is obtained; wherein:

[0119] The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss.

[0120] F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AMFeatures, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss;

[0121] F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss;

[0122] F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

[0123] In this embodiment of the invention, a foggy image and a fog-free image are acquired by a first acquisition module 301; the foggy road target detection model is iteratively trained using the foggy image and the fog-free image by a training module 302 until a trained foggy road target detection model is obtained; wherein:

[0124] The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnetwork is trained using the first loss; the F-value of the hazy image is then used to calculate the first loss. SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AM Features are used to calculate the first hinge loss. The parameters of the feature modification modules for the dehazing feature selection subnet and the feature absorption subnet are fixed. The absorption checker of the feature absorption subnet is trained using the first hinge loss. The F-value of the hazy image is then used to... AMFeatures are used to calculate the second hinge loss. The parameters of the absorption checker in the feature absorption subnet are fixed, and the feature modification module of the feature absorption subnet is trained using the second hinge loss. The F-value of the foggy image is then used to... M Features are processed through an object detection subnet to obtain candidate bounding boxes for target regions and target detection categories in the foggy image. With the parameters of the feature modification module in the fixed feature absorption subnet fixed, a second loss is calculated using the initial bounding boxes and candidate bounding boxes of the target regions in the foggy image, as well as the target type and target detection type. This second loss is then used to train the foggy road target detection model. This approach helps improve the feature extraction performance of student models, optimizes the training effect of the foggy road target detection model, and enhances its overall performance.

[0125] This invention also provides a target detection device for foggy roads, see [link to relevant documentation]. Figure 8 The device includes:

[0126] The second acquisition module 401 is used to acquire the image to be detected;

[0127] The trained foggy road target detection model is trained based on a training method for a foggy road target detection model as described in the second aspect of the present invention. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models.

[0128] Processing module 402 is used to input the image to be detected into the feature absorption subnet serving as the student model to obtain the F-value of the image to be detected. M Features; the F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

[0129] In some embodiments, the processing module 402 further includes:

[0130] The first processing unit 412 is used to process the F of the image to be detected. M The features are input into the neck network of the object detection subnet to obtain the fused features of the image to be detected;

[0131] The second processing unit 422 is used to input the fused features of the image to be detected into the head network of the object detection network to obtain the target detection result of the image to be detected.

[0132] In this embodiment of the invention, the image to be detected is acquired by the second acquisition module 401, and the image to be detected is input into the feature absorption subnet serving as the student model by the processing module 402 to obtain the F-value of the image to be detected. M Features; the F of the image to be detectedM The features are input to the object detection subnetwork, and the first processing unit 412 processes the F features of the image to be detected. M The features are input to the neck network of the object detection subnetwork to obtain the fused features of the image to be detected. The fused features of the image to be detected are then input to the head network of the object detection network through the second processing unit 422 to obtain the target detection result of the image to be detected. In this embodiment, a trained foggy road target detection model is used to detect pedestrians and vehicles in the image to be detected in order to obtain the target detection result of the image to be detected.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0136] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A training method for a target detection model on foggy roads, characterized in that, The foggy road target detection model includes a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model). The object detection subnetwork includes a neck network and a head network. The method includes: Acquire a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box for the target region; The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss. The dehazing feature selection subnetwork includes a set of differential convolutional layers with five parallel convolutional layers, a content-guided attention mechanism, and an improved Mamba mechanism; wherein, the set of differential convolutional layers with five parallel convolutional layers includes original convolution, central difference convolution, angular difference convolution, horizontal difference convolution, and vertical difference convolution; the content-guided attention mechanism includes a channel attention module and a spatial attention module; The dehazing feature selection subnet also includes converting the differential convolution set of the five parallel convolutional layers into a single standard convolutional layer through the additive nature of the convolutional layers; F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AM Features, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss; F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss; First hinge loss Second hinge loss The formula is as follows: Where A is the absorption checker and FM is the feature change module. Indicates feature change module F M The weight was frozen during training absorption checker A. This indicates that the weight of the absorption checker A is frozen during the training of the feature mutation module; x represents the input foggy image, and A(x) represents the F of the foggy image. A Feature; z represents the input foggy image, F represents a foggy image M feature, F represents a foggy image AM Features; E x～qdata(x) Let x follow a distribution qdata(x), E z～pdata(z) This indicates that z follows the distribution of qdata(z); qdata(x) represents the set of foggy image samples applicable to the dehazing feature selection subnet, and qdata(z) represents the set of foggy image samples applicable to the feature modification module; F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

2. The training method for a target detection model on a foggy road according to claim 1, characterized in that, The neck network and the head network implement multi-scale object detection on three different layers.

3. The training method for a target detection model on a foggy road according to claim 1, characterized in that, The second loss includes the coordinate loss of the bounding box relative to the coordinate center, the height and width loss of the bounding box, the confidence loss of the bounding box with objects, the confidence loss of the bounding box without objects, and the classification loss of the cell containing objects.

4. A method for detecting targets on foggy roads, characterized in that, include: Acquire the image to be detected; The image to be detected is input into the trained foggy road target detection model to detect the image; The trained foggy road target detection model is obtained by training a foggy road target detection model based on any one of the training methods described in claims 1-3. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models. The image to be detected is input into the feature absorption subnetwork that serves as the student model to obtain the F-value of the image to be detected. M feature; F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

5. The method for detecting targets on foggy roads according to claim 4, characterized in that, F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection results of the image to be detected, including: F of the image to be detected M The features are input into the neck network of the object detection subnet to obtain the fused features of the image to be detected; The fused features of the image to be detected are input into the head network of the object detection network to obtain the target detection result of the image to be detected.

6. A training device for a target detection model on a foggy road, characterized in that, The foggy road target detection model includes a feature absorption subnetwork (as a student model), an object detection subnetwork, and a defogging feature selection subnetwork (as a teacher model). The device includes: The first acquisition module is used to acquire a foggy image and a fog-free image; the foggy image and the fog-free image are a pair of target images in the same road scene, and the foggy image has a target category and an initial bounding box of the target region; The training module is used to iteratively train the foggy road target detection model using the foggy image and the fog-free image until a fully trained foggy road target detection model is obtained; wherein: The foggy image and the fog-free image are processed through a dehazing feature selection sub-network to obtain the F-value of the foggy image. SC F features and fog-free images SC Features; F through foggy images SC F features and fog-free images SC Features are used to calculate the first loss; the dehazing feature selection subnet is then trained using the first loss. The dehazing feature selection subnetwork includes a set of differential convolutional layers with five parallel convolutional layers, a content-guided attention mechanism, and an improved Mamba mechanism; wherein, the set of differential convolutional layers with five parallel convolutional layers includes original convolution, central difference convolution, angular difference convolution, horizontal difference convolution, and vertical difference convolution; the content-guided attention mechanism includes a channel attention module and a spatial attention module; The dehazing feature selection subnet also includes converting the differential convolution set of the five parallel convolutional layers into a single standard convolutional layer through the additive nature of the convolutional layers; F of the foggy image SC The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. A Features; The foggy image is passed through the feature transformation module of the feature absorption subnet to obtain the F of the foggy image. M Features; F of the foggy image M The feature is passed through the absorption checker of the feature absorption subnet to obtain the F of the foggy image. AM Features; F through foggy images A Features and F of foggy images AM Features, calculate the first hinge loss, fix the parameters of the feature change module of the dehazing feature selection subnet and the feature absorption subnet, and train the absorption checker of the feature absorption subnet through the first hinge loss; F through a foggy image AM Features are used to calculate the second hinge loss, fix the parameters of the absorption checker of the feature absorption subnet, and train the feature change module of the feature absorption subnet through the second hinge loss; First hinge loss Second hinge loss The formula is as follows: Where A is the absorption checker and FM is the feature change module. Indicates feature change module F M The weight was frozen during training absorption checker A. This indicates that the weight of the absorption checker A is frozen during the training of the feature mutation module; x represents the input foggy image, and A(x) represents the F of the foggy image. A Feature; z represents the input foggy image, F represents a foggy image M feature, F represents a foggy image AM Features; E x～qdata(x) Let x follow a distribution qdata(x), E z～pdata(z) This indicates that z follows the distribution of qdata(z); qdata(x) represents the set of foggy image samples applicable to the dehazing feature selection subnet, and qdata(z) represents the set of foggy image samples applicable to the feature modification module; F of the foggy image M The features are passed through the object detection subnet to obtain the target region candidate bounding box and target detection category of the foggy image; the parameters of the feature change module of the fixed feature absorption subnet are used to calculate the second loss through the initial bounding box and target region candidate bounding box of the foggy image, as well as the target type and target detection type, and the foggy road target detection model is trained through the second loss.

7. A target detection device for foggy roads, characterized in that, The device includes: The second acquisition module is used to acquire the image to be detected; The trained foggy road target detection model is trained based on the training method of a foggy road target detection model as described in any one of claims 1-3. The trained foggy road target detection model includes a feature absorption subnetwork and an object detection subnetwork as student models. The processing module is used to input the image to be detected into the feature absorption subnet serving as the student model, and obtain the F-value of the image to be detected. M Features; the F of the image to be detected M The features are input into the object detection subnetwork to obtain the target detection result of the image to be detected.

8. A target detection device for foggy roads according to claim 7, characterized in that, The processing module includes: The first processing unit is used to process the F of the image to be detected. M The features are input into the neck network of the object detection subnet to obtain the fused features of the image to be detected; The second processing unit is used to input the fused features of the image to be detected into the head network of the object detection network to obtain the target detection result of the image to be detected.

Images