A heterogeneous multi-task collaborative system for autonomous driving scenarios

Abstract

The present invention provides a heterogeneous multi-task collaborative system in an autonomous driving scenario, which learns prompts for specific tasks in an end-to-end manner, and then models adapter functions based on a cross-attention mechanism, thereby incorporating language information into visual features as prior knowledge; the system can significantly improve the performance of an existing pre-trained model with strong performance in single-task learning in multi-task learning, without the need to redesign the pre-training scheme or increase the overhead of the training model, thereby achieving better migration to specific downstream tasks during training.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and multi-task collaborative training, and more specifically, to a heterogeneous multi-task collaborative system in an autonomous driving scenario. Background Art

[0002] Heterogeneous multi-task learning is a long-standing research problem in computer vision and has become an emerging research direction for autonomous driving. Autonomous vehicles need to perform multiple perception tasks simultaneously, such as locating pedestrians and cars, determining whether a road is drivable, and detecting lane lines to perform the act of driving.

[0003] Joint learning of multiple perception tasks can not only reduce the training and inference time of autonomous driving models, but also learn generalized representations of sensor data. In addition to efficiency and simplicity, this integrated architecture for joint learning of multiple perception tasks has the potential to enhance the robustness of the driving system by implicitly learning the synergy between heterogeneous tasks.

[0004] However, multi-task joint learning often fails to achieve better results. The reason lies in the existence of negative transfer phenomenon: due to the conflict between different tasks, the performance of a single task may be degraded. The heterogeneity of different tasks further exacerbates this problem and brings many difficulties to training a unified model. First of all, the performance of multi-task learning depends on many factors, including the architecture of the model, the method of data augmentation, the hyperparameters of the model, and the convergence characteristics of the model. Specific techniques for specific types of network structures or tasks are usually not applicable to general architectures. Therefore, it is difficult for general feature representation methods to work well in specific downstream tasks during training.

[0005] Secondly, the difference in the objectives of the model during pre-training and fine-tuning can also lead to performance degradation in multi-task learning, as most supervised and self-supervised methods are tailored for specific types of objectives or tasks. In addition, since data annotation is a labor-intensive process, it is difficult to collect complete annotations of different granularities required for all tasks, which further complicates the situation.

[0006] In heterogeneous multi-task learning with only partial task annotations, the performance of different supervised and self-supervised pre-training methods has declined, and the performance of some methods has even dropped significantly, far behind single-task learning methods. Existing pre-training methods include supervised pre-training methods, classification-oriented pre-training methods such as SimCLR, unsupervised visual representation pre-training methods such as MoCo, detection-oriented pre-training methods such as DeTCo, and segmentation-oriented pre-training methods such as DenseCL. There are large differences in their performance in joint learning of three different fine-grained tasks (object detection, semantic segmentation, and drivable road segmentation) on the large autonomous driving dataset BDD100K. Some of these methods show very low performance on the semantic segmentation task.

[0007] In addition to supervised and self-supervised pre-training methods, CLIP is a cross-modal contrastive learning method that learns high-quality visual representations by performing contrastive learning using a large number of noisy image-text pairs. It uses ResNet and BERT as encoders for images and text. The CLIP model performs attention pooling and L2 normalization on the last output feature map of the fifth convolutional layer of ResNet to generate global image features for zero-shot reasoning. In order to extract text features for each class, the CLIP model constructs semantic category-specific prompt sentences through a prompt generation function and inputs them into the text encoder to obtain normalized output features for all categories. The contrastive language-image pre-training method (CLIP) has also achieved breakthrough performance in zero-shot image recognition, indicating that it has great potential in improving the versatility and robustness of multi-task learning. From the perspective of transfer learning, CLIP can be used as a complementary scheme to enhance the pre-training-fine-tuning method because it can understand concepts in natural language and the correspondence between visual and language features.

[0008] Although the global image features and category semantic features of the CLIP model can be well aligned during pre-training on image-text comparison, they do not preserve any spatial details and thus cannot be directly used for downstream fine-grained tasks.

[0009] In addition, if only the weights of the image encoder are reused and the text encoder is discarded, the knowledge in the CLIP model cannot be fully utilized. Therefore, we generate semantically stronger contextual features for downstream tasks by promoting the compatibility of semantic categories and image features of different tasks. In addition, the prompt generation function used in the CLIP model outputs hand-crafted prompts using predefined templates, which makes the model performance very sensitive to the form of the template and does not work well when transferred to heterogeneous downstream tasks.

[0010] The prior art discloses a patent for a crowd-sensing incentive mechanism under multi-task cooperative applications. This patent divides application scenarios into four categories according to different users and task types: complete information and incomplete information, homogeneous multi-tasks and heterogeneous multi-tasks. And corresponding multi-task cooperative incentive mechanisms based on the Steinberg game model are designed in different application scenarios. Secondly, according to different application requirements, two reward distribution functions are designed to solve the multi-task reward distribution problem, one is based on the number of users and the other is based on the task value. This patent can enable both users and system servers to obtain Nash equilibrium solutions in four scenarios, and has time validity, can be applied to real-time tasks, and ultimately enable the system server to obtain the maximum utility value; however, the patent does not propose a corresponding solution to the problem that general representation methods are difficult to migrate to specific downstream tasks during training. Summary of the invention

[0011] The present invention provides a heterogeneous multi-task collaborative system in an autonomous driving scenario, which can achieve better migration to specific downstream tasks during training.

[0012] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0013] A heterogeneous multi-task collaborative system for autonomous driving scenarios, including an image encoder, a text encoder, an adapter, an object detector, and a segmentation decoder;

[0014] The image encoder is used to encode the image in the collected vehicle driving road image;

[0015] The text encoder is used to encode the text in the collected vehicle driving road pictures;

[0016] The adapter is used to adapt the encoded images and text;

[0017] Object detectors are used to detect traffic objects;

[0018] The segmentation decoder processes the adapted text for semantic segmentation tasks, processes the adapted image for drivable road segmentation tasks, and processes the object detector data for traffic object detection tasks.

[0019] Furthermore, when processing semantic segmentation tasks and drivable road segmentation tasks, the head of the segmentation decoder includes a head based on MaskFormer; the MaskFormer is composed of a fully convolutional pixel decoder and a transformer decoder module, which regards the segmentation task as a mask classification task; the fully convolutional pixel decoder takes the output features of the second to fifth convolutional layers of the feature map pyramid network as input, and generates high-resolution pixel features after upsampling the features layer by layer; the transformer decoder module uses a fixed set of queries to focus on image features and generate mask features; the pixel features and the mask features are multiplied to generate a prediction mask.

[0020] Furthermore, when processing the traffic object detection task, the head of the segmentation decoder includes Sparse R-CNN, which inputs a fixed number of learnable region proposal boxes into a series of dynamic instance interaction head structures for prediction.

[0021] Furthermore, to process unlabeled data, the system uses a self-training method to train three single-task teacher models based on the available annotations for each task, and uses them to generate pseudo labels on the unlabeled data. The single-task teacher model uses the same network architecture and training method as Sparse R-CNN and MaskFormer trained on labeled data.

[0022] Furthermore, although the three tasks have inconsistent output forms, they all require determining the categories of boxes and pixels, and use prompts specific to the task category to complete the fine-tuning of model parameters:

[0023]

[0024] n i refers to the feature of the category name belonging to a specific task, v i is a learnable context parameter, Gen is the prompt generation function, TE is the text encoder, and L2_NORM is L2 regularization.

[0025] Furthermore, by learning an adapter function A to generate language-aware context for downstream tasks, the adapter uses the cross-attention mechanism in the transformer decoder to perform language-to-vision feature fusion:

[0026] A(T,z)=TransDecoder(q=z,k&v=T)

[0027] q, k, v represent query, key, and value, T is the regularized output feature of N classes, z represents the last layer feature of the fifth layer of FPN; the number of channels of T is adjusted through a fully connected layer, and the output of the above formula is used to replace the last layer feature of FPN, while keeping the head design of the specific task unchanged.

[0028] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0029] The present invention learns task-specific cues in an end-to-end manner, and then models the adapter function based on a cross-attention mechanism, thereby incorporating language information as prior knowledge into visual features. It can significantly improve the performance of existing pre-trained models with strong performance in single-task learning in multi-task learning, without the need to redesign the pre-training scheme or increase the overhead of the training model, thereby achieving better migration to specific downstream tasks during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the working process of this system;

[0031] Figure 2 Flow chart of the method in Example 3. DETAILED DESCRIPTION

[0032] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0033] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0034] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0035] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] Example 1

[0037] like Figure 1 As shown, a heterogeneous multi-task collaborative system in an autonomous driving scenario includes an image encoder, a text encoder, an adapter, an object detector, and a segmentation decoder;

[0038] The image encoder is used to encode the image in the collected vehicle driving road image;

[0039] The text encoder is used to encode the text in the collected vehicle driving road pictures;

[0040] The adapter is used to adapt the encoded images and text;

[0041] Object detectors are used to detect traffic objects;

[0042] The segmentation decoder processes the adapted text for semantic segmentation tasks, processes the adapted image for drivable road segmentation tasks, and processes the object detector data for traffic object detection tasks.

[0043] When processing semantic segmentation tasks and drivable road segmentation tasks, the head of the segmentation decoder includes a head based on MaskFormer; the MaskFormer consists of a fully convolutional pixel decoder and a transformer decoder module, which regards the segmentation task as a mask classification task; the fully convolutional pixel decoder takes the output features of the second to fifth convolutional layers of the feature map pyramid network as input, and generates high-resolution pixel features after upsampling the features layer by layer; the transformer decoder module uses a fixed set of queries to focus on image features and generate mask features; the pixel features and the mask features are multiplied to generate a prediction mask.

[0044] When processing traffic object detection tasks, the head of the segmentation decoder includes Sparse R-CNN, which inputs a fixed number of learnable region proposal boxes into a series of dynamic instance interaction head structures for prediction.

[0045] To process unlabeled data, the system uses a self-training method to train three single-task teacher models based on the available annotations for each task, and uses them to generate pseudo labels on unlabeled data. The single-task teacher model uses the same network architecture and training method as Sparse R-CNN and MaskFormer trained on labeled data.

[0046] Although the three tasks have inconsistent output forms, they all require determining the categories of boxes and pixels, and use prompts specific to the task category to complete fine-tuning of model parameters:

[0047]

[0048] n i refers to the feature of the category name belonging to a specific task, v i is a learnable context parameter, Gen is the prompt generation function, TE is the text encoder, and L2_NORM is L2 regularization.

[0049] By learning an adapter function A to generate language-aware context for downstream tasks, the adapter exploits the cross-attention mechanism in the transformer decoder for language-to-vision feature fusion:

[0050] A(T,z)=TransDecoder(q=z,k&v=T)

[0051] q, k, v represent query, key, and value, T is the regularized output feature of N classes, z represents the last layer feature of the fifth layer of FPN; the number of channels of T is adjusted through a fully connected layer, and the output of the above formula is used to replace the last layer feature of FPN, while keeping the head design of the specific task unchanged.

[0052] Example 2

[0053] like Figure 1 As shown in the figure, a heterogeneous multi-task collaborative system in an autonomous driving scenario includes an image encoder, a text encoder, an adapter, an object detector, and a segmentation decoder. This model simultaneously solves three challenging heterogeneous tasks in autonomous driving, namely, traffic object detection, semantic segmentation, and drivable road segmentation. Following the hard parameter sharing scheme, each task has its own specific decoder head and shares the same network backbone. The network backbone uses the joint structure of the classic ResNet (residual neural network) and FPN (feature map pyramid network) to generate multi-scale features. The decoder head is also different for different tasks. The decoder head for semantic segmentation and drivable road segmentation tasks is based on the head of MaskFormer. MaskFormer consists of a fully convolutional pixel decoder and a transformer decoder module, which treats the segmentation task as a mask classification task. The pixel decoder takes the output features of the second to fifth convolutional layers of the feature map pyramid network as input, and upsamples the features layer by layer to produce high-resolution pixel features. The transformer decoder module uses a fixed set of queries to focus on image features and generate mask features. The pixel features and mask features are multiplied to generate a predicted mask. The decoder head for the detection task consists of SparseR-CNN, which feeds a fixed number of learnable region proposals into a series of dynamic instance interaction head structures for prediction. The dynamic instance interaction head structure dynamically generates the weights of the convolutional filters and predicts the location and category of the object in a cascaded manner.

[0054] In order to process unlabeled data, the present invention uses a self-training method to train three single-task teacher models based on the available annotations of each task, and uses them to generate pseudo labels on unlabeled data, where the single-task teacher model uses the same network architecture and training method as Sparse R-CNN and MaskFormer trained on labeled data. For detection tasks, the model uses a score threshold of 0.5 to select pseudo-box labels. For semantic segmentation and drivable road segmentation tasks, the model uses a score threshold of 0.3 to select segmentation masks, and pixels with lower scores will have their labels ignored when calculating the loss function. After this operation, each image has annotations for all tasks.

[0055] Prompt fine-tuning emerges as a new alternative to fine-tuning in prompt-based learning. Its approach is to adjust only a small number of learnable parameters while freezing the large-scale language model. Based on this, we propose a simple and effective pre-train-adapt-fine-tuning paradigm for effectively adjusting the off-the-shelf pre-trained model without redesigning the resource-intensive pre-training stage. In the adaptation stage, we can get the pre-trained model weights inherited from the pre-training stage. In order to transform the model with a small number of learnable parameters so that the pre-trained weight knowledge can be adapted to the multi-task scenario, the model freezes the randomly initialized task-specific head and the shared backbone network, and adjusts the parameters of the FPN (Feature Pyramid Network) supervised by the multi-task loss function. In the multi-task model, the loss function is the weighted sum of all objective losses for each task.

[0056] The adaptation stage bridges the gap between pre-training and fine-tuning by including both pre-trained weights and multi-task objectives. At the same time, the frozen backbone prevents the pre-trained weights from being destroyed before the fine-tuning stage. FPN (Feature Pyramid Network), as a multi-scale adapter, has a very large capacity and can provide more semantically robust features for fine-grained downstream tasks. Combining FPN with CLIP, while using the CLIP model to enhance the pre-training-adaptation mode, the multi-level features output by FPN contain rich semantic information in the spatial dimension, which can better support the compatibility between the semantic concepts of each task and the image features, so as to generate more semantically robust contextual features for downstream tasks. The present invention proposes a language-vision adapter to enhance the correspondence between pixels and semantic categories, and its structure is shown in the accompanying figure. Although the three tasks have inconsistent output forms, they all need to determine the categories of boxes and pixels. We use task category-specific prompts as follows:

[0057]

[0058] ni refers to the feature of the category name belonging to a specific task, vi is a learnable context parameter, Gen is the prompt generation function, TE is the text encoder, and L2_NORM is L2 regularization.

[0059] In order to learn an adapter from language to vision, it is necessary to align text features and dense FPN features and fuse language priors into visual features. This paper learns an adapter function A to generate language-aware context for downstream tasks. The adapter uses the cross-attention mechanism in the transformer decoder to perform language-to-vision feature fusion:

[0060] A(T,z)=TransDecoder(q=z,k&v=T)

[0061] q, k, v represent query, key, and value, T is the normalized output feature of N classes, and z represents the last layer feature of FPN layer 5. The number of channels of T is adjusted through a fully connected layer, and the output of the above formula is used to replace the last layer feature of FPN, while keeping the head design of the specific task unchanged.

[0062] Example 3

[0063] like Figure 2 As shown, a heterogeneous multi-task collaborative training method in an autonomous driving scenario includes the following steps:

[0064] S1: an image encoder is used to encode an image in the collected vehicle driving road image;

[0065] S2: The text encoder encodes the text in the collected vehicle driving road picture;

[0066] S3: The adapter is used to adapt the encoded image and text;

[0067] S4: object detector performs traffic object detection;

[0068] S5: The segmentation decoder processes the adapted text for a semantic segmentation task, processes the adapted image for a drivable road segmentation task, and processes the object detector data for a traffic object detection task.

[0069] When processing semantic segmentation tasks and drivable road segmentation tasks, the head of the segmentation decoder includes a head based on MaskFormer; the MaskFormer consists of a fully convolutional pixel decoder and a transformer decoder module, which regards the segmentation task as a mask classification task; the fully convolutional pixel decoder takes the output features of the second to fifth convolutional layers of the feature map pyramid network as input, and generates high-resolution pixel features after upsampling the features layer by layer; the transformer decoder module uses a fixed set of queries to focus on image features and generate mask features; the pixel features and the mask features are multiplied to generate a prediction mask.

[0070] When processing traffic object detection tasks, the head of the segmentation decoder includes Sparse R-CNN, which inputs a fixed number of learnable region proposal boxes into a series of dynamic instance interaction head structures for prediction.

[0071] To process unlabeled data, the system uses a self-training method to train three single-task teacher models based on the available annotations for each task, and uses them to generate pseudo labels on unlabeled data. The single-task teacher model uses the same network architecture and training method as Sparse R-CNN and MaskFormer trained on labeled data.

[0072] Although the three tasks have inconsistent output forms, they all require determining the categories of boxes and pixels, and use prompts specific to the task category to complete fine-tuning of model parameters:

[0073]

[0074] n i refers to the feature of the category name belonging to a specific task, v i is a learnable context parameter, Gen is the prompt generation function, TE is the text encoder, and L2_NORM is L2 regularization.

[0075] By learning an adapter function A to generate language-aware context for downstream tasks, the adapter exploits the cross-attention mechanism in the transformer decoder for language-to-vision feature fusion:

[0076] A(T,z)=TransDecoder(q=z,k&v=T)

[0077] q, k, v represent query, key, and value, T is the regularized output feature of N classes, z represents the last layer feature of the fifth layer of FPN; the number of channels of T is adjusted through a fully connected layer, and the output of the above formula is used to replace the last layer feature of FPN, while keeping the head design of the specific task unchanged.

[0078] The same or similar reference numerals correspond to the same or similar components;

[0079] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0080] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A heterogeneous multi-task collaborative system in an autonomous driving scenario, characterized in that: Includes image encoder, text encoder, adapter, object detector, and segmentation decoder; The image encoder is used to encode the image in the collected vehicle driving road image; The text encoder is used to encode the text in the collected vehicle driving road pictures; The adapter is used to adapt the encoded images and text; Object detectors are used to perform traffic object detection; The segmentation decoder processes the adapted text for semantic segmentation, the adapted image for drivable road segmentation, and the object detector data for traffic object detection. When processing semantic segmentation tasks and drivable road segmentation tasks, the segmentation decoder head includes a MaskFormer-based head; The MaskFormer consists of a fully convolutional pixel decoder and a transformer decoder module, which treats the segmentation task as a mask classification task; When processing traffic object detection tasks, the segmentation decoder head includes Sparse R-CNN, which feeds a fixed number of learnable region proposals into a series of dynamic instance interaction head structures for prediction; To handle unlabeled data, the system uses a self-training approach to train three single-task teacher models based on the available annotations for each task, and uses them to generate pseudo labels on unlabeled data. The single-task teacher model uses the same network architecture and training method as Sparse R-CNN and MaskFormer, which have been trained on labeled data. Although the three tasks have inconsistent output forms, they all require determining the categories of boxes and pixels, and use prompts specific to the task category to complete fine-tuning of model parameters: n i refers to the feature of the category name belonging to a specific task, v i is a learnable context parameter, Gen is the prompt generation function, TE is the text encoder, and L2_NORM is L2 regularization.

2. The heterogeneous multi-task collaborative system in the autonomous driving scenario according to claim 1, characterized in that: The fully convolutional pixel decoder takes the output features of the second to fifth convolutional layers of the feature map pyramid network as input, and generates high-resolution pixel features after upsampling the features layer by layer.

3. The heterogeneous multi-task collaborative system in the autonomous driving scenario according to claim 2, characterized in that: The transformer decoder module utilizes a fixed set of queries to focus on image features and produces mask features.

4. The heterogeneous multi-task collaborative system in the autonomous driving scenario according to claim 1, characterized in that: Multiply the pixel features and mask features to generate a predicted mask.

5. The heterogeneous multi-task collaborative system in the autonomous driving scenario according to claim 1, characterized in that: By learning an adapter function A to generate language-aware context for downstream tasks, the adapter exploits the cross-attention mechanism in the transformer decoder for language-to-vision feature fusion: A(T,z)=TransDecoder(q=z,k&v=T) q, k, v represent query, key, and value, T is the regularized output feature of N classes, and z represents the last layer feature of the fifth layer of FPN; The number of channels of T is adjusted through a fully connected layer, and the output of the above formula is used to replace the last layer of FPN features, while keeping the head design of the specific task unchanged.

Images