Multi-task automatic driving perception method based on task prompt
Through the multi-task autonomous driving perception method with task prompts, the problems of computational burden and insufficient data sets of multiple perception modules are solved by using the segmentation-everything model and unified weak semantic correlation decoder, and the expansion of cross-domain perception tasks and performance maintenance are achieved.
Patent Information
- Application Number
- CN202410339741.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-25
- Publication Date
- 2025-09-26
AI Technical Summary
Existing autonomous driving perception technology is unable to handle the computational burden of multiple perception modules and the problem of insufficient data sets at the same time, resulting in the inability to expand perception tasks across domains and data sets.
A multi-task autonomous driving perception method based on task prompts is adopted. By splitting all model encoders and unifying weak semantic correlation decoders, combined with a task prompt generation module and an open vocabulary object detector, cross-domain and cross-dataset perception task expansion is achieved, avoiding an increase in the number of parameters.
It achieves the expansion of multi-task perception capabilities, avoids the increase of parameters, maintains the performance of a single task, can be trained on different source data sets, and improves the efficiency and accuracy of perception tasks.
Smart Images
Figure CN120708172A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in the field of image processing, specifically a multi-task autonomous driving perception method based on task prompts. Background Art
[0002] Autonomous driving perception technology relies on the superposition of multiple perception modules to achieve complete perception, but independent reasoning of multiple perception modules will increase the computational burden. At the same time, most existing datasets are task-specific. Although some comprehensive perception datasets attempt to cover a wider range of perception tasks, they are still not enough to meet diverse perception needs. Summary of the Invention
[0003] In response to the defects of existing technologies that cannot achieve simultaneous training of different source data sets and cannot process detection and segmentation tasks at the same time, the present invention proposes a multi-task autonomous driving perception method based on task prompts. The task prompts are used to achieve scalable perception of autonomous driving across domains and data sets, and the unified decoder within the task is used to reduce the increase in the number of additional parameters caused by multi-task parallelism.
[0004] The present invention is achieved through the following technical solutions:
[0005] The present invention relates to a multi-task autonomous driving perception method based on task prompts. In the offline phase, a multi-task autonomous driving perception network including a segmentation-everything model (SAM) encoder is constructed, and multi-source dataset samples including vehicle detection, traffic sign detection, traffic light detection, and lane segmentation are used for training. In the online phase, the trained network is used for real-time image detection.
[0006] The multi-task autonomous driving perception network includes: a segmentation everything model (SAM) encoder, a task prompt generation module, a segmentation everything model decoder and a unified weak semantic correlation decoder, wherein: the segmentation everything model encoder performs encoding processing based on the input image to obtain instance features; the task prompt generation module uses the superposition of N multi-head self-attention layers (MHSA) based on specific task knowledge to generate prompt information corresponding to the specific task based on the instance features; the segmentation everything model decoder performs decoding processing based on the segmentation prompt information to obtain lane line segmentation results; the unified weak semantic correlation decoder generates reliable weak semantic correlation decoding capabilities based on the prompt information and the pseudo-labels obtained by the open vocabulary target detector, and then performs decoding processing based on the detection prompt information to obtain vehicle detection, traffic sign detection, and traffic light detection results.
[0007] The task-specific knowledge refers to the unique implicit feature representations of lane lines, traffic elements, traffic signs, and traffic lights at the feature level (obtained based on learning and not explicitly given).
[0008] The prompt information refers to: the lane line position in the lane line segmentation task; the positioning and category information of each instance to be detected in the traffic element, traffic sign, and traffic light detection task. Technical Effects
[0009] The present invention adopts a split everything model (SAM) encoder to implicitly encode all potential instances in the feature space; adopts a task hint generation module to learn perception capabilities from a single annotated dataset and achieve task expansion; adopts an open vocabulary object detection method to provide a unified pseudo label to train a unified weak semantic correlation decoder, thereby avoiding an increase in the number of parameters; adopts task hints to achieve the expansion of perception tasks, and can learn complete perception capabilities from multiple single annotated datasets; adopts a unified task correlation decoder to avoid the problem of a significant increase in the number of parameters caused by stacking task heads; achieves the best performance in multi-task algorithms while avoiding a significant increase in the number of parameters; and achieves performance similar to that of a single model. Compared with the prior art, the present invention can handle implicit decoupled perception and classification tasks and achieve perception task expansion; compared with the existing multi-task framework, the task hint-based approach can be trained on different source datasets without affecting the performance of a single task; adopts an open vocabulary object detection-based approach to provide a unified pseudo label to train a unified weak semantic correlation decoder to achieve unified positioning of the detection box, avoiding a significant increase in the number of parameters due to task stacking. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 Schematic diagram of the multi-task autonomous driving perception network;
[0011] Figure 2 Schematic diagram of the system of the present invention;
[0012] Figure 3 Generate module diagram for task prompts;
[0013] Figure 4 Schematic diagram for unified weak semantic correlation decoder training;
[0014] Figure 5 is a flow chart of an embodiment;
[0015] Figure 6 Schematic diagram of the effects of different task prompt generation modules on the performance of four tasks;
[0016] Figure 7 A schematic diagram of the results of the perception task for existing open vocabulary object detection and segmentation work;
[0017] Figure 8 This is an example effect diagram. DETAILED DESCRIPTION
[0018] This embodiment relates to a multi-task autonomous driving perception method based on task prompts. In the offline stage, a multi-task autonomous driving perception network including a segmentation-everything model (SAM) encoder is constructed, and training is performed using samples from a multi-source dataset including vehicle detection, traffic sign detection, traffic light detection, and lane segmentation. In the online stage, the trained network is used for real-time image detection.
[0019] like Figure 1 As shown, the multi-task autonomous driving perception network includes: a segmentation model encoder, a task prompt generation module, a segmentation model decoder and its segmentation head, and a unified weak semantic correlation decoder and its detection head, wherein: the segmentation model encoder obtains instance features based on the implicit encoding processing of the input image in the feature space; the task prompt generation module uses the superposition of N multi-head self-attention layers (MHSA) based on the potential instance features to generate prompt information related to the downstream detection and segmentation tasks based on the instance features; the segmentation model decoder decodes the lane line segmentation task to be decoded based on the prompt information and the sparse prompts in the segmentation model; the unified weak semantic correlation decoder generates reliable weak semantic correlation decoding capabilities based on the pseudo labels obtained by the open vocabulary target detector according to the prompt information, and then performs decoding processing according to the detection prompt information to obtain the detection task to be decoded features; the segmentation head performs decoding processing based on the to-be-decoded feature information to obtain the lane line prediction result, and the detection head performs decoding processing based on the to-be-decoded feature information of the detection task to obtain the traffic element, traffic sign, and traffic light detection results.
[0020] The sparse hint is structurally implemented by but not limited to the technology described by A. Kirillov et al. in "Segment Anything" (2023 IEEE / CVF International Conference on Computer Vision (ICCV), pp. 3992–4003, 2023).
[0021] The segmentation anything model encoder and decoder are respectively implemented by but not limited to the technology described in "Segment Anything" by A. Kirillov et al. (2023 IEEE / CVF International Conference on Computer Vision (ICCV), pp. 3992–4003, 2023).
[0022] like Figure 3 As shown in FIG, the task prompt generation module consists of 4 layers of multi-head self-attention layers.
[0023] The superposition of the multi-head self-attention layer (MHSA) is to add position information to the image features through the sum operation to obtain After that, task-specific task hints are obtained through N multi-head self-attention layers (MHSA) Among them: After segmenting all model encoders, the input image features are given as F SAM , the position code is Pos, and the feature after integrating the position information is The features after fusion of position information The process of inputting N multi-head self-attention layers (MHSA) to calculate task-specific task prompts is denoted as TPGM t (), the task-specific task prompt is denoted as P t .
[0024] like Figure 4 As shown, the weak semantic correlation decoding refers to: constructing a text prompt set P text , after taking the perception target in the downstream task as its subset, the images in all subsequent subtasks are represented as Where: K is the number of tasks, and then all images and text prompts in subsequent subtasks are combined and passed to the open vocabulary object detector algorithm (OVOD) to obtain pseudo labels for all perceived objects. Finally, a unified category label instance is used instead of the pseudo label Used to train a unified weak semantic correlation decoder and obtain a unified decoder The combination of images and textual cues from all subsequent subtasks is passed to the open vocabulary object detector algorithm defined as OVOD(), and the resulting pseudo labels of all perceived objects are defined as The unified decoder is defined as Dec uni The process of training a unified decoder is defined as Update(). Then, the task prompts related to the detection task obtained by the task prompt generation module are input into the trained weak semantic correlation decoder, and the prompts related to the segmentation task are input into the segmentation model decoder to obtain the task-related features, that is, in: They are task prompts related to detection tasks, task prompts related to segmentation tasks, and sparse prompts in all segmentation models. They are weak semantic correlation decoder and segmentation model decoder, They are detection task related features and segmentation task related features respectively.
[0025] Preferably, the sparse hint is set as a learnable vector so that its weight is adaptively adjusted according to the statistical information of the dataset to avoid splitting the sparse hint P in all models. sparse It is inconsistent with the semantic information of downstream segmentation tasks.
[0026] The detection head Hbox and split head H map It is not frozen during the training process and uses task-specific information from the task hint generation module and the training dataset to enhance its prediction ability. Finally, the detection head and segmentation head output Among them: H box (),H map () represents inputting task-related features into the detection head and segmentation head and obtaining the task perception result O box ,O map , namely traffic elements, traffic signs, traffic lights, and lane line segmentation maps.
[0027] In practical experiments, we used the SAM-Tiny model with pre-trained parameters as the encoder for the segmentation model. During training, we set the number of task prompts for detection to 900 and for segmentation to 100. For optimization, we used the AdamW optimizer and trained on an Nvidia 3090 GPU with a batch size of 16. The mean average precision (mAP) was used as the evaluation metric, while the accuracy (Acc) was used for the segmentation task.
[0028] During the training process, multiple loss terms are used for step-by-step training, specifically: L = {L s ,s∈1,2,...,K}, where: K is the number of tasks, the detection task uses Hungarian matching loss, and the lane line segmentation task uses mean square error loss.
[0029] like Figure 5 As shown in Figure 1, after the image is input into the segmentation-everything model encoder for encoding, the detection task prompt and the segmentation task prompt are generated by two task prompt generation modules. Then, the unified weak semantic correlation decoder and the segmentation-everything model decoder are used to generate the detection result and the segmentation result respectively. Finally, the results are merged to obtain the perception content as shown in Table 1.
[0030] Table 1 Experimental comparison results
[0031] In the segmentation task, the present invention achieves high lane segmentation accuracy. Its reliable performance in detection-related tasks demonstrates that the features provided by the Segment Everything model encoder are indeed task-independent and can effectively capture the distinguishing characteristics between different instances at the feature level. Furthermore, the present invention significantly improves performance compared to the multi-task method DINO+SAM. This improvement can be attributed to the unified weak semantic correlation decoder, which learns more general representations. The decoder's decoding process is not limited to a specific category, but instead learns how to generate the final bounding box based on general image features.
[0032] like Figure 6 As shown in the figure, the impact of different task hint generation modules on the performance of four tasks is evaluated. The horizontal axis represents the number of task hint generation modules from 0 to 4, and the vertical axis represents the performance index. In tasks related to detection, it is observed that the model performance gradually improves with the increase in the number of task hint generation modules, which shows that the introduction of the task hint generation module has a positive impact on the integration of task-specific information. In contrast, in the segmentation task, the performance improvement brought by the addition of the task hint generation module is not as significant as that of the detection task. This phenomenon can be attributed to the fact that the segmentation model is specifically developed for the segmentation task, so in this specific task, the gap between the encoder and the downstream decoder is limited. Therefore, only a slight adjustment to the domain is sufficient to adapt to the lane segmentation task because it is closer to the distribution of the pre-training task.
[0033] Open vocabulary object detection methods aim to perceive instances in the natural world using multimodal inputs. As shown in Table 2, although open vocabulary object detection methods have the ability to perceive instances, their accuracy is not satisfactory and is far from practical application.
[0034] Table 2 Comparison results with open vocabulary object detection work
[0035] like Figure 7 The following figure shows the results of open-vocabulary object detection and segmentation, highlighting three major difficulties encountered in the three subtasks: For traffic elements, open-vocabulary object detection has a high miss detection rate (red is the prediction, blue is the ground truth), especially in complex and dense scenes. For traffic lights, the bounding boxes provided by open-vocabulary object detection do not fully align with the ground truth, resulting in a suboptimal score. For traffic signs, open-vocabulary object detection has difficulty distinguishing the three types, resulting in a large amount of overlap in the detection results. This suggests that open-vocabulary object detection lacks understanding of specific semantic details and only focuses on general categories such as "sign." Furthermore, open-vocabulary lane segmentation is used for lane segmentation. Despite this, open-vocabulary lane segmentation still differs fundamentally from the lane segmentation task used in autonomous driving in terms of semantic content, and therefore cannot be directly used.
[0036] Compared to existing technologies, the present invention implicitly decouples perception and classification tasks, enabling the expansion of perception tasks. Compared to existing multi-task frameworks, the task-cueing approach allows training on different source datasets without compromising the performance of individual tasks. By providing unified pseudo-labels based on open-vocabulary object detection, training a unified weak semantic correlation decoder achieves uniform detection box positioning, avoiding a significant increase in parameters due to task stacking. Task cues extract the required features from a single annotated dataset of different tasks, enabling multi-task parallel inference using a single model with different sources. In the facial domain, the present invention can be applied to the unification of multiple tasks, such as face recognition and facial attribute classification. Task cues can extract facial features for each task from existing datasets related to facial tasks. This enables the use of a single model to simultaneously process multiple facial tasks and conduct parallel inference, thereby improving efficiency and accuracy. In the autonomous driving domain, the present method and system can parallelize serial tasks. Task cues can capture temporal and spatial features required for downstream tasks from the scene. This allows for parallel inference of perception and decision-making tasks, resulting in faster and more accurate decisions. In addition to the aforementioned fields, the present system can also be applied to other multi-task scenarios. Through flexible task prompts and feature extraction, multiple tasks can be processed in parallel, improving the overall efficiency of the system.
[0037] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.
Claims
1. A multi-task autonomous driving perception method based on task prompts, characterized in that: By constructing a multi-task autonomous driving perception network that includes a segmentation model encoder in the offline stage, it is trained using multi-source dataset samples including vehicle detection, traffic sign detection, traffic light detection, and lane line segmentation; In the online stage, the trained network is used for real-time image detection; The multi-task autonomous driving perception network includes: a segmentation model encoder, a task prompt generation module, a segmentation model decoder, and a unified weak semantic correlation decoder, wherein: the segmentation model encoder performs encoding processing based on the input image to obtain instance features; the task prompt generation module uses the superposition of N multi-head self-attention layers based on specific task knowledge to generate prompt information corresponding to the specific task based on the instance features; the segmentation model decoder performs decoding processing based on the segmentation prompt information to obtain lane line segmentation results; the unified weak semantic correlation decoder generates reliable weak semantic correlation decoding capabilities based on the prompt information and pseudo labels obtained by the open vocabulary object detector, and then performs decoding processing based on the detection prompt information to obtain vehicle detection, traffic sign detection, and traffic light detection results; The task-specific knowledge mentioned above refers to the implicit feature representations of lane lines, traffic elements, traffic signs, and traffic lights at the feature level, which are obtained through learning. The prompt information refers to: the lane line position in the lane line segmentation task; the positioning and category information of each instance to be detected in the traffic element, traffic sign, and traffic light detection task.
2. The multi-task autonomous driving perception method based on task prompts according to claim 1 is characterized in that: The task prompt generation module consists of 4 layers of multi-head self-attention layers; The superposition of the multi-head self-attention layer means: adding position information to the image features by summing operations to obtain After that, the task-specific task hint is obtained through N multi-head self-attention layers. Among them: After segmenting all model encoders, the input image features are given as F SAM , the position code is Pos, and the feature after integrating the position information is The features after fusion of position information The process of inputting N multi-head self-attention layers to calculate task-specific task prompts is denoted as TPGM t (), the task-specific task prompt is denoted as P t .
3. The multi-task autonomous driving perception method based on task prompts according to claim 1 is characterized in that: The weak semantic correlation decoding is to construct a text prompt set P text , after taking the perception target in the downstream task as its subset, the images in all subsequent subtasks are represented as Where: K is the number of tasks, and then all subsequent subtasks of images and text prompts are combined and passed to the open vocabulary object detector algorithm to obtain all the perceived objects to generate pseudo labels. Finally, a unified category label instance is used instead of the pseudo label Used to train a unified weak semantic correlation decoder and obtain a unified decoder The combination of images and textual cues from all subsequent subtasks is passed to the open vocabulary object detector algorithm defined as OVOD(), and the resulting pseudo labels of all perceived objects are defined as The unified decoder is defined as Dec uni The process of training a unified decoder is defined as Update(). After that, the prompts related to the detection task in the task prompt generation module are input into the trained weak semantic correlation decoder, and the prompts related to the segmentation task are input into the segmentation model decoder to obtain the task-related features, that is, in: P sparse They are task prompts related to detection tasks, task prompts related to segmentation tasks, and sparse prompts in all segmentation models. They are weak semantic correlation decoder and segmentation model decoder, They are detection task related features and segmentation task related features respectively.
4. The multi-task autonomous driving perception method based on task prompts according to claim 3 is characterized in that: The sparse hints are set as learnable vectors so that they can adaptively adjust their weights according to the statistical information of the dataset to avoid splitting the sparse hints P in all models. sparse It is inconsistent with the semantic information of downstream segmentation tasks.
5. The multi-task autonomous driving perception method based on task prompts according to claim 1 is characterized in that: The detection head H box and split head H map After not being frozen during training and leveraging task-specific information from the task hint generation module and the training dataset to enhance their prediction capabilities, the detection and segmentation heads output Among them: H box (),H map () represents inputting task-related features into the detection head and segmentation head and obtaining the task perception result O box ,O map , namely traffic elements, traffic signs, traffic lights, and lane line segmentation maps.