A multitask detection method for comprehensive driving perception

Abstract

The present application relates to a kind of multi-task detection methods for comprehensive driving perception, constructs multi-task detection model;Multi-task detection model includes backbone network, neck network, detection head, segmentation head and depth living information extraction head;Wherein, backbone network is used to extract image feature, neck network is used to fuse the feature generated by backbone network;Detection head identifies multiple targets, including the detection frame information and confidence information of vehicle, pedestrian, and also gives the secondary label of the state information of representing other vehicle, whether to give way or follow the state information according to secondary label;Segmentation head divides multiple categories target, including drivable area, vehicle, pedestrian and background area, and according to classification information, give the label information whether the front drivable area is clean;Depth living information extraction head generates depth information;Time sequence integration target detection information obtains detection time sequence information, obtains more beneficial to subsequent task's perception information and pre-decision information.

Description

Technical Field

[0001] This invention relates to the field of driving perception technology, and in particular to a multi-task detection method for comprehensive driving perception. Background Technology

[0002] The rapid development of pure computer vision algorithms and end-to-end deep learning technologies has made autonomous driving a research hotspot in the field of computer vision. Nevertheless, vision-based tasks, such as object detection and image segmentation, still face considerable challenges in realizing low-cost autonomous driving systems. Given the cost-effectiveness of camera-based solutions, they dominate in practical object detection and segmentation tasks. Therefore, ensuring that vehicles can achieve effective environmental perception and provide more complete information downstream using only a single front-facing camera input has become a critical issue that urgently needs to be addressed. Traditionally, three key tasks of environmental perception are considered fundamental to guiding intelligent vehicles: traffic object detection, drivable area segmentation, and lane detection.

[0003] In recent years, with the rapid development of deep learning, many well-known object detection algorithms have emerged. Currently, mainstream object detection algorithms can be divided into two-stage methods and single-stage methods. Two-stage methods complete the detection task in two steps: region proposal and object classification. The most typical example is the R-CNN series of algorithms, which offer high detection accuracy and stability. However, their detection speed is usually slower than single-stage methods. Therefore, single-stage methods are more popular in real-time detection applications.

[0004] Existing object detection methods include single-frame detectors and the YOLO series, while semantic segmentation is handled by networks such as U-Net, SegNet, and ERNet. For lane detection, models such as LaneNet and Spatial Convolutional Neural Networks (SCNN) are used. However, using three separate networks to process the same image data stream can lead to unnecessary latency.

[0005] To address this issue, many existing technologies integrate these functionalities into a single encoder-decoder architecture, where the backbone and neck work together as encoders to generate context for three different tasks, as seen in MultiNet, DLT-Net, YOLOP, and HybridNet. Multi-task learning networks offer a computationally efficient solution by using encoder-decoder frameworks, where the encoder is effectively shared across different tasks, addressing the speed and performance limitations of existing models. Summary of the Invention

[0006] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a multi-task detection method for comprehensive driving perception.

[0007] This invention provides a multi-task detection method for comprehensive driving perception, comprising:

[0008] An end-to-end multi-task detection model is constructed to collaboratively process traffic object detection, segmentation, and deep generation tasks. The multi-task detection model includes a backbone network, a neck network, a detection head, a segmentation head, and a deep generation information extraction head. The backbone network and the neck network cooperate to form an encoder, wherein the backbone network is used to extract image features, and the neck network is used to fuse the features generated by the backbone network.

[0009] The detection head identifies multiple targets, including vehicle and pedestrian detection bounding box information and confidence information, and then provides secondary labels representing the status information of other vehicles. Based on the secondary labels, it determines whether to yield or follow the vehicle.

[0010] The segmentation head segments multiple categories of targets, including drivable areas, vehicles, pedestrians, and background areas, and provides label information on whether the drivable area ahead is clean based on the classification information.

[0011] The deep information extraction head generates depth information;

[0012] The obtained detection information, segmentation information, depth information, and state information are fused with the target detection information at time t-1 of the previous frame to obtain detection temporal information. Finally, the model integrates all the information to obtain the final result, which provides perception information and pre-decision information that are more conducive to subsequent tasks.

[0013] Furthermore, the backbone network is based on CSPDarknet53 to achieve multi-scale feature extraction. In order to enhance the information extraction capability, the C2F module connected to the output stage in each stage of CSPDarknet53 is optimized into a C2fA module.

[0014] Furthermore, the neck network uses a simple spatial pyramid fast pooling module to process the deepest feature maps from the backbone network. This module includes: an initial CBR module, which performs convolution operations on the input feature maps to extract features and accelerates the training process and improves model stability through batch normalization; a ReLU activation function introduces non-linearity to enhance the model's expressive power and reduces the number of channels to half of the original number; then, multiple cascaded max pooling operations are performed to concatenate the results of each max pooling operation; finally, a CBR module converts the fused feature map into the specified number of output channels.

[0015] Furthermore, during the feature fusion process, the neck network adopts a feature pyramid network structure, and the C2F module, which alternates with the CBS module, is optimized into a C2fA module to effectively combine features at different semantic levels.

[0016] Furthermore, any feature map input to the C2fA module is decomposed into two parts by the CBS module. The CBS module contains convolutional layers, batch normalization layers, and SiLU activation functions. One part is processed by a CBS module and then input into multiple convolutional layers with different kernel sizes. These multiple convolutional layers process the features in parallel. The results of the parallel processing of multiple convolutions are fused by the CBS module, processed by the CBR module, and then concatenated with the other part. The CBR module contains convolutional layers, batch normalization layers, and ReLU activation function layers.

[0017] Furthermore, the detection head adopts the anchor-based multi-scale detection head used in YOLOP, and integrates a path aggregation network and a feature pyramid network. A SimAttention mechanism is introduced before each detection head in the multi-scale detection head. The detection head uses three anchor points with different aspect ratios to predict the positional offset and scale change of the target in each grid cell on the multi-scale fused feature map. This process generates the probability, confidence score, and secondary label for each detection category. Specifically, the information from the three detection heads in the multi-scale detection head is obtained, with dimensions [20, 20, 10], [40, 40, 10], and [80, 80, 10]. The first two dimensions represent size information, and the last dimension represents the probability, confidence score, and secondary label for each detection category: the secondary label is a binary classification information for a stopped or moving vehicle. The state information of whether to yield or follow is determined based on the detection category of the secondary label.

[0018] Furthermore, the segmentation head and the deep information extraction head have the same structure, which includes the following sequentially combined modules: CBS module, upsampling using nearest neighbor interpolation, C2fA module, CBS module, upsampling, CBS module, C2fA module, upsampling, and CBS module.

[0019] Furthermore, the depth information extraction head uses a simple spatial pyramid fast pooling module to extract depth information; the depth head defines D discrete depth planes parallel to the image plane along the optical axis, and each depth plane together with the image feature plane constitutes the view frustum space; assuming the lowest depth range is a meters, the highest depth range is b meters, and the interval is c meters, a total of D = (ba) / c + 1 discrete depth planes are generated.

[0020] Each pixel (h, w) corresponds to D possible 3D points within the view frustum, and the depth of each point is a discrete value d;

[0021] The input feature map is copied and expanded along the depth dimension into a 4-dimensional tensor as a depth feature, representing the feature response of each pixel in different depth planes; the depth feature is combined with depth encoding;

[0022] For each pixel (h, w) and depth plane d, calculate its 3D coordinates in the camera coordinate system;

[0023] The model predicts the probability distribution of pixels in the depth plane.

[0024] The final depth information includes: 3D coordinates, depth features combined with depth encoding, and probability distribution.

[0025] Furthermore, the total loss function of the multi-task detection model is as follows:

[0026] ;

[0027] in, For detecting loss of traffic objects, To divide the loss, For depth information loss, Both are in equilibrium , and The balance factor.

[0028] Furthermore, traffic object detection loss Defined as a combination of the following components: penalized classification loss Confidence loss and bounding box regression loss , is represented as:

[0029] ;

[0030] in, They are in balance , and The balance factor, where the penalty classification loss is... and confidence loss Zoom loss and bounding box regression loss are used. Distance-IoU loss is used;

[0031] It includes the classification error between pixels and targets in the network output, the cross-entropy loss, and the IoU loss between the segmentation mask and the ground truth mask in the network output;

[0032] Cross-entropy loss is used, which is calculated by comparing the depth distance predicted by the model with the ground truth value, thereby guiding the model's learning process.

[0033] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0034] To improve the speed and performance of our model, we adopted CSPDarknet as the foundation of the backbone network and optimized it using the C2fA module to effectively combine features at different semantic levels. The backbone network achieves a balance between accuracy and computational cost. In the decoder, we retained YOLOP's anchor-based multi-scale detection scheme and combined it with a path aggregation network and a feature pyramid network. The integration of the feature pyramid network and the path aggregation network enhances feature fusion by facilitating the top-down propagation of semantic features and the bottom-up propagation of localization features. Furthermore, we introduced a SimAttention mechanism before the detection head to enhance overall performance. At an input resolution of 640x640, our network's convergence time is slightly longer than YOLOP's, but it significantly improves accuracy and recall.

[0035] This application simultaneously handles three driving perception tasks: traffic object detection, traffic object segmentation, and depth generation, and supports end-to-end training. In traffic object detection and segmentation, our model performs exceptionally well on the SDExpressway and BDD100K datasets, surpassing existing techniques in both accuracy and efficiency.

[0036] This application integrates all the information to obtain the final result, which provides more perceptual information and pre-decision information that are more conducive to subsequent tasks, thus providing effective support for intelligent driving decision-making. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the architecture of the multi-task detection model provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the architecture of a multi-task detection model represented by feature maps provided in an embodiment of the present invention;

[0041] Figure 3This is a schematic diagram of the focus in the backbone network provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the CBS and CBR modules provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the C2fA module provided in an embodiment of the present invention;

[0044] Figure 6 The image shows the effect of the multi-task detection model provided in the embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] Panoramic driving perception systems are crucial for autonomous driving as they provide key traffic-related information. This application constructs a multi-task detection model for comprehensive driving perception, which is an efficient and effective multi-task learning framework designed specifically for real-time panoramic driving perception. The multi-task detection model includes an encoder for feature extraction and three decoders that simultaneously perform traffic object detection, traffic object segmentation, and depth information generation. For feature extraction, a C2fA module is proposed to enhance the model's extraction capabilities. To enhance our dataset, we extend the SDExpressway dataset by adding nighttime and severe weather conditions.

[0048] Example 1

[0049] The multi-task detection method for comprehensive driving perception provided in this application includes:

[0050] Construct an end-to-end multi-task detection model that collaboratively handles traffic object detection, segmentation, and deep generation tasks. For example... Figure 1 and Figure 2 As shown, the multi-task detection model includes a backbone network, a neck network, a detection head, a segmentation head, and a depth information extraction head. The backbone network and the neck network cooperate to form an encoder, wherein the backbone network is used to extract image features, and the neck network is used to fuse the features generated by the backbone network; the detection head, segmentation head, and depth information extraction head are used for traffic object detection, traffic object segmentation, and depth information generation tasks, respectively.

[0051] Next, we will introduce in detail the design of the multi-task detection model architecture and the optimization strategies implemented to improve multi-task performance.

[0052] The backbone network aims to achieve multi-scale feature extraction, a crucial step in multi-task learning that directly impacts the performance of multi-task detection models across various tasks. This application uses CSPDarknet53 as the foundation of the backbone network for the multi-task detection model. Furthermore, to enhance information extraction capabilities, the C2F modules connected to the output stages of each stage of CSPDarknet53 are optimized into C2fA modules. The C2fA module is an improvement upon the C2f module, such as... Figure 5 As shown. Any feature map input to the C2fA module is decomposed into two parts by the CBS module. In this application, as... Figure 4 As shown, the CBS module contains convolutional layers, batch normalization layers, and the SiLU activation function. A portion of the data, after being processed by a single CBS module, is input into multiple convolutional layers with different kernel sizes. This parallel processing of features by multiple convolutional layers allows for broader integration of feature information, offering significant advantages. The results of the parallel convolutional processing are fused by the CBS module, processed by the CBR module, and then concatenated with another portion; for example... Figure 4 As shown, the CBR module contains convolutional layers, batch normalization layers, and ReLU activation function layers. The C2fA module, compared to C2f, includes additional convolutional layers, which helps extract more complex and deeper features. This design not only reduces gradient repetition during optimization but also improves feature propagation and feature reuse. Experimental results demonstrate that utilizing multiple parallel convolutional layers and multiple receptive layers significantly improves the model's ability to extract depth information from images, thereby improving the receptive field and feature representation capabilities. The CSPDarknet53 input first undergoes focus processing, as shown... Figure 3As shown, the focus layer is a spatial-channel reconstruction operation used to reduce computation while preserving feature information. It first appeared in YOLOv5 to replace the traditional computationally expensive downsampling method. Focus divides the input image into four sub-images in a 2x2 grid, and then concatenates these four sub-images along the channel dimension. This slice reconstruction of the input image reduces spatial resolution while increasing the number of channels, thereby reducing computation and preserving more information.

[0053] The neck network is designed to integrate multi-scale features generated by the backbone network.

[0054] The neck network uses a simple spatial pyramid fast pooling module to process the deepest feature maps from the backbone network. Furthermore, an attention layer is added after the simple spatial pyramid fast pooling module to further enhance feature extraction. The simple spatial pyramid fast pooling module includes an initial CBR module to pre-process the input feature map, reducing the number of channels to half of the original number. Its function is to perform convolution operations on the input feature map to extract features, and batch normalization accelerates the training process and improves model stability. The ReLU activation function introduces non-linearity, enhancing the model's expressive power. Multiple cascaded max pooling and concatenation operations: Through multiple cascaded max pooling operations and concatenation of the results of each max pooling operation, features at different scales are fused. Finally, a CBR module converts the fused feature map into a specified number of output channels. Through this design, the simple spatial pyramid fast pooling module can effectively extract multi-scale features and fuse these features to enhance the model's ability to recognize objects of different sizes. At the same time, the simplified design makes it more computationally efficient and suitable for computer vision tasks with high real-time requirements. Ablation experiments verified that the simple spatial pyramid fast pooling module balances speed and accuracy.

[0055] During feature fusion, the neck network employs a feature pyramid network structure, and the C2F module, which alternates with the CBS module, is optimized into a C2fA module to effectively combine features at different semantic levels. This design ultimately generates feature information encompassing multiple scales and semantic levels, which is then passed to three decoders.

[0056] The detection head aims to identify various object categories, including vehicles, traffic signs, and road markings. In this application, the detection head employs the anchor-based multi-scale detection head used in YOLOP and integrates a path aggregation network and a feature pyramid network. The feature pyramid network facilitates the top-down flow of semantic information, while the path aggregation network enables efficient bottom-up propagation of localization details; this synergistic combination enhances feature fusion capabilities. To improve the detection performance for small targets, a SimAttention mechanism is introduced before each detection head in the multi-scale detection head. Unlike existing channel or spatial attention modules, SimAttention primarily focuses on spatial dimension attention, assigning different weights to different spatial locations within the feature map, thereby improving the overall performance of the task. Notably, SimAttention also demonstrates advantages in computational cost. The detection head utilizes three anchor points with different aspect ratios to predict the positional offset and scale variation of the target in each grid cell of the multi-scale fused feature map. The process generates the probability, confidence score, and secondary label for each detection category. Specifically, it obtains information from three detection heads in the multi-scale detection head, with dimensions [20, 20, 10], [40, 40, 10], and [80, 80, 10]. The first two dimensions represent size information, and the last dimension represents the probability, confidence score, and secondary label for each detection category: the secondary label is a binary classification information for a stopped or moving vehicle. Based on the detection category of the secondary label, it determines whether to yield or follow the vehicle. Figure 6 As shown, the second-level tag is Figure 6 The D and L in the diagram represent information about stopped or live vehicles, respectively. The decision to follow or detour is based on this information.

[0057] The segmentation head and the deep information extraction head have the same structure, which includes the following modules in sequence: CBS module, upsampling using nearest neighbor interpolation, C2fA module, CBS module, upsampling, CBS module, C2fA module, upsampling, and CBS module.

[0058] The segmentation head uses the shallow feature map of the feature pyramid.

[0059] The depth information extraction head uses a simple spatial pyramid fast pooling module to extract depth information. The depth head defines D discrete depth planes parallel to the image plane along the optical axis. Each depth plane and the image feature plane together form the view frustum space. Assuming the lowest depth range is a meters, the highest depth range is b meters, and the interval is c meters, a total of D = (ba) / c + 1 discrete depth planes are generated.

[0060] Each pixel (h, w) corresponds to D possible 3D points within the view frustum, and the depth of each point is a discrete value d;

[0061] The input feature map is copied and expanded along the depth dimension into a 4-dimensional tensor as a depth feature, representing the feature response of each pixel in different depth planes; the depth feature is combined with depth encoding;

[0062] For each pixel (h, w) and depth plane d, calculate its 3D coordinates in the camera coordinate system;

[0063] The model predicts the probability distribution of pixels in the depth plane.

[0064] The final depth information includes: 3D coordinates, depth features combined with depth encoding, and probability distribution.

[0065] In the design of multi-task loss functions, traffic object detection loss Defined as a combination of the following components: penalized classification loss Confidence loss and bounding box regression loss , is represented as:

[0066] ;

[0067] in, They are in balance , and The balance factor, an example The values ​​are 0.5, 1.0, and 0.05. Among them, the penalized classification loss... and confidence loss A zoom loss is employed. Traditional focus loss uniformly reduces the weight of negative samples to balance the contributions of foreground and background categories, while zoom loss specifically reduces the weight of negative samples while maintaining emphasis on positive samples. Zoom loss is particularly effective at mitigating the loss contribution of well-classified examples, thus encouraging the network to focus more on difficult-to-classify negative samples. Bounding box regression loss. The Distance-IoU loss is adopted, which not only considers the overlap between the predicted box and the ground truth box, but also combines the distance, scale and aspect ratio similarity between the predicted box and the ground truth box.

[0068] The segmentation loss comprises the classification error between pixels in the network output and the target, the cross-entropy loss, and the IoU loss between the segmentation mask and the ground truth mask. Since the IoU loss is particularly effective for predicting sparse categories (such as lane lines), it is incorporated in addition to the cross-entropy loss with logits. middle.

[0069] To account for depth information loss, cross-entropy loss is used, and the cross-entropy loss is calculated between the depth distance predicted by the model and the ground truth.

[0070] The total loss function is as follows:

[0071] ;

[0072] in, These are all balance factors, one example. The values ​​are 0.3, 0.2, and 0.2 respectively.

[0073] use and Fine-tuning ensures that each component of the total loss function contributes harmoniously.

[0074] The obtained detection information, segmentation information, depth information, and state information are fused with the target detection information at time t-1 of the previous frame to obtain detection temporal information. Finally, the model integrates all the information to obtain the final result, which provides perception information and pre-decision information that are more conducive to subsequent tasks.

[0075] In this application, the performance of the multi-task detection model was evaluated on the BDD100K and SDExpressway datasets to demonstrate its effectiveness. To comprehensively evaluate the model's generalization ability and alleviate the relative scarcity of the SDExpressway dataset for multi-task highway detection, we collected and labeled an additional 2000 images to enrich the dataset, including nighttime and adverse weather conditions such as rain and fog. Extensive experiments on the challenging BDD100K dataset and the enhanced SDExpressway dataset demonstrate that the multi-task detection model in this application achieves state-of-the-art performance, significantly outperforming the baseline model.

[0076] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multi-task detection method for comprehensive driving perception, characterized in that, include: An end-to-end multi-task detection model is constructed to collaboratively process traffic object detection, segmentation, and deep generation tasks. The multi-task detection model includes a backbone network, a neck network, a detection head, a segmentation head, and a deep generation information extraction head. The backbone network and the neck network cooperate to form an encoder, wherein the backbone network is used to extract image features, and the neck network is used to fuse the features generated by the backbone network. The detection head identifies multiple targets, including vehicle and pedestrian detection bounding box information and confidence information, and provides secondary labels representing the status information of other vehicles. Based on the secondary labels, it determines whether to yield or follow the vehicle. The segmentation head segments multiple categories of targets, including drivable areas, vehicles, pedestrians, and background areas, and provides label information on whether the drivable area ahead is clean based on the classification information. The deep information extraction head generates depth information. It uses a simple spatial pyramid fast pooling module to extract depth information. D discrete depth planes parallel to the image plane are defined along the optical axis. Each depth plane, together with the image feature plane, constitutes the view frustum space. Assuming the lowest depth range is *a* meters, the highest depth range is *b* meters, and the interval is *c* meters, a total of D = (ba) / c + 1 discrete depth planes are generated. Each pixel (h, w) corresponds to D possible 3D points within the view frustum, and the depth of each point is a discrete value *d*. The input feature map is copied and expanded along the depth dimension into a 4D tensor as the depth feature, representing the feature response of each pixel in different depth planes. The depth features are combined with depth encoding. For each pixel (h, w) and depth plane *d*, its 3D coordinates in the camera coordinate system are calculated. The probability distribution of the pixel in the depth plane is predicted using a model. The final depth information includes: 3D coordinates, depth features combined with depth encoding, and probability distribution. The total loss function of the multi-task detection model is as follows: ； in, For detecting loss of traffic objects, To divide the loss, For depth information loss, Both are in equilibrium , and Balance factor; traffic object detection loss Defined as a combination of the following components: penalized classification loss Confidence loss and bounding box regression loss , represented as: ; in, They are in balance , and The balance factor, where the penalty classification loss is... and confidence loss Zoom loss and bounding box regression loss are used. Distance-IoU loss is used; It includes the classification error between pixels and targets in the network output, the cross-entropy loss, and the IoU loss between the segmentation mask and the ground truth mask in the network output; Cross-entropy loss is used, and the cross-entropy loss is calculated between the depth distance predicted by the model and the ground truth, thereby guiding the model's learning process; The obtained detection information, segmentation information, depth information, and state information are fused with the target detection information at time t-1 of the previous frame to obtain detection temporal information. Finally, the model integrates all the information to obtain the final result, which provides perception information and pre-decision information that are more conducive to subsequent tasks.

2. The multi-task detection method for comprehensive driving perception according to claim 1, characterized in that, The backbone network is based on CSPDarknet53 to achieve multi-scale feature extraction. In order to enhance the information extraction capability, the C2F module connected to the output stage in each stage of CSPDarknet53 is optimized into a C2fA module.

3. The multi-task detection method for comprehensive driving perception according to claim 1, characterized in that, The neck network uses a simple spatial pyramid fast pooling module to process the deepest feature maps from the backbone network. The simple spatial pyramid fast pooling module includes: an initial CBR module, which performs convolution operations on the input feature map to extract features and accelerates the training process and improves the stability of the model through batch normalization; the ReLU activation function introduces non-linearity to enhance the expressive power of the model and reduces the number of channels to half of the original number of channels; then, multiple cascaded max pooling operations are performed to concatenate the results of each max pooling operation; finally, a CBR module is used to convert the fused feature map into the specified number of output channels.

4. The multi-task detection method for comprehensive driving perception according to claim 3, characterized in that, During feature fusion, the neck network adopts a feature pyramid network structure, and the C2F module, which alternates with the CBS module, is optimized into a C2fA module to effectively combine features at different semantic levels.

5. The multi-task detection method for comprehensive driving perception according to claim 2 or 4, characterized in that, Any feature map input to the C2fA module is decomposed into two parts by the CBS module. The CBS module contains convolutional layers, batch normalization layers, and SiLU activation functions. One part is processed by a CBS module and then input into multiple convolutional layers with different kernel sizes. These multiple convolutional layers process the features in parallel. The results of the parallel processing of multiple convolutions are fused by the CBS module, processed by the CBR module, and then concatenated with the other part. The CBR module contains convolutional layers, batch normalization layers, and ReLU activation function layers.

6. The multi-task detection method for comprehensive driving perception according to claim 1, characterized in that, The detection head adopts the anchor-based multi-scale detection head used in YOLOP and integrates a path aggregation network and a feature pyramid network. A SimAttention mechanism is introduced before each detection head in the multi-scale detection head. The detection head uses three anchor points with different aspect ratios to predict the positional offset and scale change of the target in each grid cell on the multi-scale fused feature map. This process generates the probability, confidence score, and secondary label for each detection category. Specifically, the information from the three detection heads in the multi-scale detection head is obtained, with dimensions [20, 20, 10], [40, 40, 10], and [80, 80, 10]. The first two dimensions represent size information, and the last dimension represents the probability, confidence score, and secondary label for each detection category: the secondary label is a binary classification information for a stopped or moving vehicle. The state information of whether to yield or follow is determined based on the detection category of the secondary label.

7. The multi-task detection method for comprehensive driving perception according to claim 1, characterized in that, The segmentation head and the deep information extraction head have the same structure, which includes the following modules in sequence: CBS module, upsampling using nearest neighbor interpolation, C2fA module, CBS module, upsampling, CBS module, C2fA module, upsampling, and CBS module.

Images