An assisted driving AI middle platform system based on confidence learning and multi-task learning

Abstract

This invention discloses an AI platform system for assisted driving based on confidence learning and multi-task learning. It includes a data processing module for receiving raw data, using confidence learning for data annotation and filtering, and constructing a dataset; a model building module for constructing encoders and decoders for multi-task models, and providing context information transmission paths and fusion pooling enhancement modules for models with strong task relevance, enabling context interaction in the decoder; a model training module for defining and training parameters, converting and outputting the model after training to a specified number of iterations or accuracy; and a model deployment and monitoring module for deploying the best model on the device and simultaneously monitoring it in real time in the cloud; and saving and transmitting scenario data with poor model performance to form a new dataset. This invention effectively utilizes simple tasks to assist complex tasks and compensates for some of the losses in simple tasks, ultimately improving the recognition accuracy of assisted driving perception.

Description

Technical Field

[0001] This invention relates to the field of AI algorithm platform system technology, and in particular to an AI platform system for assisted driving based on confidence learning and multi-task learning. Background Technology

[0002] With the development of deep learning, assisted driving technology is gradually being applied through algorithms. How to process massive amounts of data and annotations, and build suitable models, has become a key issue. The AI ​​platform integrates these issues end-to-end to build large-scale intelligent services. It is a complete artificial intelligence model lifecycle management platform and service system that provides support for data processing, model building, model training, model deployment, and monitoring.

[0003] Currently, most AI middleware systems are still immature, with problems such as rudimentary functions or incomplete services. For example, Lenovo's LeapAI platform lacks model building for specific scenarios and does not pay enough attention to visual tasks; Alibaba Cloud's AI platform provides model building in the field of vision, but it only focuses on image recognition applications and does not have detection functions for visual perception tasks in assisted driving.

[0004] In terms of model building, assisted driving involves multiple visual perception requirements, generally including challenging tasks such as lane detection, drivable area segmentation, and object detection. The mainstream approach is to build an end-to-end deep learning multi-task network to address these challenges. Existing multi-task assisted driving visual perception systems are typically improved during the training phase of the deep learning model, mainly through three approaches: structure, optimization methods, and task relationships.

[0005] Multi-task networks generally adopt an encoder-decoder structure. The encoder consists of a robust feature extraction network as its backbone, often with an additional neck network for feature fusion. The decoder is designed for different tasks. For example, the YOLOP network proposed in the article "YOLOP: You Only Look Once for Panoptic Driving Perception" published in the journal Machine Intelligence Research in 2022 (pages 550-562) contains three task heads for object detection, lane line and drivable area segmentation, etc.

[0006] Optimization methods primarily involve optimizing the loss function and gradient. The former involves weighted summations for different tasks, and current methods utilize task uncertainty, learning rate, model performance, and geometric mean. The latter focuses on gradient tuning, mainly adjusting the gradient to balance the training rates across tasks. For example, the GradNorm method proposed in the 2018 International Conference on Machine Learning paper "GradNorm: GradentNormalization for Adaptive Loss Balancing in Deep Multitask Networks" aims to balance the gradient and optimize the Grad Loss, thereby dynamically adjusting the weights of each task.

[0007] Tasks with weak correlation in terms of task relationships may lead to negative transfer effects, thus requiring models to learn specific task representations or relationships between tasks. For example, the paper "DeMT: Deformable MixerTransformer for Multi-Task Learning of Dense" presented at the 2023 Association for the Advancement of Artificial Intelligence international conference combines CNN and Transformer to achieve multi-task learning. The task-aware decoder captures task interaction features and uses Transformer to generate corresponding task-specific features for prediction.

[0008] The above methods have also achieved certain results when applied to the field of driver assistance. However, current end-to-end driver assistance multi-task networks rarely consider the interaction between tasks. Even tasks with strong correlation in visual perception are decoded independently. Summary of the Invention

[0009] This invention provides an AI platform system for assisted driving based on confidence learning and multi-task learning. It designs feature transfer paths between tasks according to the difficulty of different tasks, effectively uses simple tasks to assist complex tasks, and adds an enhanced pooling fusion module to compensate for some of the losses of simple tasks, which can ultimately improve the recognition accuracy of assisted driving perception.

[0010] An AI platform system for assisted driving based on confidence learning and multi-task learning is characterized by including a data processing module, a model building module, a model training module, and a model deployment monitoring module.

[0011] The data processing module is used to receive raw data, perform data annotation and filtering using confidence learning, and construct a dataset.

[0012] The model building module is used to build encoders and decoders for multi-task models, and to provide context information transmission paths and fusion pooling enhancement modules for models with strong task relevance, enabling the decoder's context to interact.

[0013] The model training module is used to define and train parameters, and after training to a specified number of times or accuracy, convert and output the model.

[0014] The model deployment monitoring module is used to deploy the best model on the device and monitor it in real time in the cloud; and to save and send back scene data with poor model performance to form a new dataset.

[0015] This invention addresses the feature loss problem caused by the independent nature of multi-task network decoders by constructing a multi-task model that integrates a joint contextual information transmission path and a fusion pooling enhancement module. It effectively utilizes the semantic information of the context to provide more comprehensive features for complex tasks.

[0016] In the data processing module, the process of data annotation and filtering using confidence learning is as follows:

[0017] (1) Estimate the joint distribution of noise labels and real labels;

[0018] (2) Filter the samples according to the PBC rules and delete erroneous samples;

[0019] (3) After deleting erroneous samples, the sample class weights are readjusted and the training is performed.

[0020] The specific process of step (1) is as follows:

[0021] (1-1) The probability P that the i-th sample belongs to the j-th class is obtained through cross-validation. ij Calculate the average probability t under manually labeled category j. j As a confidence threshold;

[0022] (1-2) Calculate the counting matrix P is obtained in all classification cases ij >t j The number of samples;

[0023] (1-3) According to The calibration is performed so that the total count is the same as the total number of manually labeled samples, as shown in the following formula, where... For manual labeling Total number of samples

[0024]

[0025] (1-4) Estimating noise labels and real label y * joint distribution Then normalize it, as shown in the following formula:

[0026]

[0027] In step (2), according to the PBC rule, for each manually labeled category i∈1,2,…m, select… A sample, according to probability Filter by sorting in ascending order.

[0028] In step (3), according to the joint distribution Adjust the loss weights for each category i as follows: in, Then, retrain using the Co-teaching framework.

[0029] Step (1) yields the joint distribution of true and noisy labels, not just the distribution of noise characteristics, thus providing more effective information. This distribution approximates the distribution of data in the real scene as the dataset grows, which serves as a standard, justifying the use of PBC rules to filter out mislabeled data in step (2). Step (3) corrects the sample weights based on this joint distribution, resulting in greater accuracy when retraining using Co-teaching.

[0030] In the model building module, when constructing the encoder of the multi-task model, the feature extraction model is selected as the backbone network, denoted as E. b Based on real-time requirements, the neck network is selected for feature fusion as part of the encoder, denoted as E. n ;

[0031] Simultaneously, a corresponding decoder is constructed based on the nature of the task. Let the total number of tasks be t, and the header of each task be denoted as D. i Then the overall decoder is D = {D1, D2, ..., D} t}

[0032] For models with strong task relevance, the following specific features are provided: Contextual information passing pathways and fusion pooling enhancement modules are implemented:

[0033] The tasks are arranged in ascending order of difficulty, and a context information transmission path is constructed. The features of simple tasks are passed up and fused into complex tasks to enhance the semantics of the surrounding environment of complex tasks. At the same time, a fusion pooling enhancement module is added to compensate for the loss caused by the context information transmission path to tasks with lower difficulty.

[0034] The fusion pooling enhancement module includes two parallel branches. The main branch consists of consecutive convolutional layers, using 1×1 convolutional kernels to adjust the number of feature channels. The secondary branch adds two 3×3 depthwise separable convolutions and a GAP layer, which are then adjusted in size and added to the features of the main branch.

[0035] For multi-task models, the overall loss function L all The calculation process is as follows:

[0036] The input X is encoded to obtain a feature map M. The inputs of different decoders are features M1, M2, ..., M. t The output Y is shown in the following formula:

[0037]

[0038] After obtaining the output Y of each task decoder, compare it with the true value Y. * The comparisons are then performed, and the overall loss is calculated using the following function. The overall loss function is composed of the loss functions of each subtask, constructed using a weighted approach with a weighting coefficient of α. i

[0039] L all =∑ i＝1…t α i L i (Y,Y * ).

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention can effectively detect mislabeled samples and improve data utilization. A confidence-based learning method is used to filter out mislabeled samples before model training, avoiding noise samples from contaminating model performance. This method, as part of data preprocessing, is decoupled from the model and training, allowing the use of any model and improving experimental efficiency.

[0042] 2. This invention addresses the real-time requirements of assisted driving by providing a deep model based on multi-task learning. It integrates contextual information transmission pathways and fusion pooling enhancement modules, significantly improving network prediction capabilities. This reduces the number of network parameters and computational load, while also decreasing online inference time, making it highly beneficial for practical applications.

[0043] 3. Experiments have shown that the AI ​​platform system proposed in this invention can improve the accuracy of multi-task models and performs excellently in visual perception tasks. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of an AI platform system for assisted driving based on confidence learning and multi-task learning according to the present invention.

[0045] Figure 2 This is a framework diagram of the confidence learning algorithm used in this invention;

[0046] Figure 3 This is a structural diagram of the multi-task model constructed in an embodiment of the present invention;

[0047] Figure 4 This is a diagram illustrating the context information transmission path structure in an embodiment of the present invention.

[0048] Figure 5 This is a structural diagram of the fusion pooling enhancement module in an embodiment of the present invention;

[0049] Figure 6 This is a diagram illustrating the reasoning effect of the present invention in a real-world scenario. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0051] like Figure 1 As shown, an AI platform system for assisted driving based on confidence learning and multi-task learning includes a data processing module, a model building module, a model training module, and a model deployment and monitoring module. The following embodiment describes the system's structure, using a port as a practical application scenario for autonomous driving. The dataset consists of raw RGB images acquired at 10 frames per second by onboard cameras in unmanned trucks. The assisted driving tasks to be addressed include object detection, lane line segmentation, and drivable area segmentation, with a total of t=3 tasks. Data is collected for each of the three tasks under different scenarios. This embodiment uses lane lines as an example to detail the confidence learning framework. The lane line dataset has seven categories: left second lane line, left first lane line, right first lane line, right second lane line, coastline, left curb, and right curb, with approximately 70,000 samples.

[0052] First, the raw data is manually labeled, and then the confidence learning framework proposed in this invention is used to filter out mislabeled samples, forming a high-confidence dataset as input for the next step of the AI ​​platform system. The confidence learning principle utilizes the joint distribution of noise and true labels to filter out mislabeled samples; this joint distribution reflects the distribution of noise and true labels in the real scene. As the data scale increases, this estimation method becomes closer to the true distribution. For ease of illustration, a small sample dataset is constructed from the original lane line data, with m=2 categories and n=10 samples after manual labeling.

[0053] In step 1, the joint distribution is calculated. First, the probability P that the i-th sample belongs to the j-th class is obtained through cross-validation. ij Calculate the average probability t under manually labeled category j. jAs a confidence threshold, as shown in Table 1.

[0054] Table 1 Cross-validation prediction probability P ij

[0055]

[0056]

[0057] Next, calculate the counting matrix. The algorithm steps are as follows:

[0058]

[0059] Table 2 shows the values ​​of the counting matrix.

[0060] Table 2 Counting Matrix

[0061]

[0062]

[0063] Then, the counting matrix is ​​calibrated to ensure that the total count equals the total number of manually labeled samples. This can be achieved by weighting the original manually labeled sample counts using the following formula:

[0064]

[0065] The results are shown in Table 3.

[0066] Table 3. Counting matrix after calibration

[0067]

[0068] Finally, normalizing the probabilities in the table above yields the joint distribution required in step 1. The results are shown in Table 4.

[0069] Table 4 Joint Distribution

[0070]

[0071] In step 2, the samples are screened according to the joint distribution and PBC rule. For each manually labeled category i∈1,2,…m, samples are selected. A sample, according to probability Filter by sorting in ascending order. For category 0, select approximately 2 samples (10 × (0.12 + 0.06 + 0.06)) for filtering, according to P... ijSort the samples in ascending order and remove samples 1 and 2. Table 1 shows that the true and noise labels for samples 1 and 2 are inconsistent, indicating a reasonable removal result. Following this, for category 1, remove sample 4; for category 2, remove samples 7 and 8; and for category 3, remove sample 9. These six abnormal samples were removed correctly, successfully cleaning the data by removing incorrectly labeled samples.

[0072] After data processing, the model building stage begins. This embodiment constructs an end-to-end model that simultaneously solves target detection, lane segmentation, and driving region segmentation, consisting of an encoder and a decoder, as follows: Figure 3 The diagram shown illustrates the overall model structure. Considering the significant size variation of targets in a port scenario—from large targets like unmanned trucks occupying large areas of the field of view to smaller targets such as traffic signs and pedestrians located further out—a high-resolution network, HRNet, was chosen as the backbone network. b Simultaneously using BiPFN as the neck network E n This is used to fully fuse features at different scales. Then, corresponding decoders D are designed for the three tasks.

[0073] This embodiment uses HRNet-W32 as the backbone network. b First, two 3×3 convolutional layers are used to downsample the original image by a factor of 4. Then, the number of channels is adjusted by repeatedly stacking the Bottle Neck in ResNet. Next, a series of Transition and Stage structures are passed through, with an additional scale branch added at each Transition structure. In the Stage stage, each scale branch is fused, ultimately resulting in four scale outputs, which are then fed into the neck network E. n To merge. n Using BiFPN, a fusion path is combined in two directions, namely top-down and bottom-up, to convey high-level semantic information and low-level semantic features respectively.

[0074] Regarding the relationship between decoders, this invention additionally proposes a context information transmission path and a fusion pooling enhancement module to interact with the decoder's context. In this embodiment, both drivable region segmentation and lane line segmentation can be viewed as semantic segmentation tasks based on pixel classification, with two categories. Compared to the relatively large drivable region, lane lines are long and thin with a very small number of pixels, making them difficult to identify, while object detection requires simultaneous object classification and localization. Therefore, the three tasks are ordered by difficulty as follows: drivable region segmentation, lane line segmentation, and object detection, with corresponding decoders D1, D2, and D3. Based on this order, the deep features of each decoder are connected through a context information transmission path, enhancing the expressive power of the task heads.

[0075] like Figure 4The image shows the size of the input features for different decoders in this embodiment, as well as the sampling or convolution operations used for context information transmission. The original image size is 720×1280, scaled to 360×640 according to the aspect ratio, and then processed by the backbone network E. b The output then shows features at four scales: P3, P4, P5, and P6. Since the drivable area task is relatively simple, it does not require a feature fusion network E. n Therefore, the smallest feature P6, with a size of 24×40, is sent to the drivable region decoder D1. From the features at the four scales, the smallest scale P6 is downsampled to obtain P7, and the features at all five scales are then sent to E. n The data is fused together for lane segmentation and object detection.

[0076] To compensate for some of the losses caused by missions in drivable areas, this invention adds a fusion pooling enhancement module, such as... Figure 5 As shown, a global average pooling branch is added during the upsampling process to restore the original image. This branch utilizes global region features to enhance pixel prediction in feasible regions, resulting in more accurate edge segmentation. The fusion pooling enhancement module consists of two branches. The main branch performs multiple upsampling operations using 1×1 convolutions and bilinear interpolation. The secondary branch consists of two 3×3 depthwise separable convolutions and a global average pooling layer, providing global edge information. This information can be passed up the context information propagation path to features related to the surrounding environment, enhancing the ability of these two complex tasks to recognize occlusion.

[0077] After the model is built, the training phase of the AI ​​platform system can begin. In this embodiment, the model is trained for 160 rounds, using the SGD optimizer and cosine annealing algorithm to dynamically adjust the learning rate. The initial learning rate is set to 0.001. A mini-batch method is used to traverse the dataset, with a batch size of 24.

[0078] After the model has converged during training, it is deployed and tested on the autonomous vehicle equipment through the deployment module of the AI ​​platform system, such as... Figure 6 The diagram shows the inference performance of this embodiment in a port scenario. The light gray area represents feasible road areas, the yellow and pink areas indicate lane lines, and the red rectangles mark obstacles. It can be seen that the lane line detection and feasible area detection are effective, and obstacles can also be identified. This demonstrates the effectiveness of the context-interaction-based assisted driving visual perception multi-task model proposed in this invention.

[0079] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A driver assistance AI platform system based on confidence learning and multi-task learning, characterized in that, It includes a data processing module, a model building module, a model training module, and a model deployment monitoring module; The data processing module is used to receive raw data, perform data annotation and filtering using confidence learning, and construct a dataset. The model building module is used to build the encoder and decoder of a multi-task model, and provides a context information transmission path and a fusion pooling enhancement module for models with strong task relevance, enabling the decoder's context to interact; specifically: The tasks are arranged in ascending order of difficulty, and a contextual information transmission path is constructed. Features from simple tasks are passed up and fused into complex tasks, increasing the semantic meaning of the surrounding environment of complex tasks. At the same time, a fusion pooling enhancement module is added to compensate for the loss caused by the contextual information transmission path to tasks with lower difficulty. The fusion pooling enhancement module contains two parallel branches. The main branch consists of consecutive convolutional layers, using 1×1 convolutional kernels to adjust the number of feature channels. The secondary branch adds two 3×3 depthwise separable convolutions and a GAP layer, which are then adjusted in size and added to the features of the main branch. The model training module is used to define and train parameters, and after training to a specified number of times or accuracy, convert and output the model. The model deployment monitoring module is used to deploy the best model on the device and monitor it in real time in the cloud; and to save and send back scene data with poor model performance to form a new dataset.

2. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 1, characterized in that, In the data processing module, the process of data annotation and filtering using confidence learning is as follows: (1) Estimate the joint distribution of noise labels and real labels; (2) Filter the samples according to the PBC rules and delete erroneous samples; (3) After deleting erroneous samples, the sample class weights are readjusted and the training is performed.

3. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 2, characterized in that, The specific process of step (1) is as follows: (1-1) The probability P that the i-th sample belongs to the j-th class is obtained through cross-validation. ij Calculate the average probability t under manually labeled category j. j As a confidence threshold; (1-2) Calculate the counting matrix P is obtained in all classification cases ij >t j The number of samples; (1-3) According to The calibration is performed so that the total count is the same as the total number of manually labeled samples, as shown in the following formula, where... For manual labeling Total number of samples (1-4) Estimating noise labels and real label y * joint distribution Then normalize it, as shown in the following formula:

4. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 3, characterized in that, In step (2), according to the PBC rule, for each manually labeled category i∈1,2,…m, select… A sample, according to probability Filter by sorting in ascending order.

5. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 4, characterized in that, In step (3), according to the joint distribution Adjust the loss weights for each category i as follows: in, Then, retrain using the Co-teaching framework.

6. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 1, characterized in that, In the model building module, when constructing the encoder of the multi-task model, the feature extraction model is selected as the backbone network, denoted as E. b Based on real-time requirements, the neck network is selected for feature fusion as part of the encoder, denoted as E. n ; Simultaneously, a corresponding decoder is constructed based on the nature of the task. Let the total number of tasks be t, and the header of each task be denoted as D. i Then the overall decoder is D = {D1, D2, ..., D} t } 7. The assisted driving AI platform system based on confidence learning and multi-task learning according to claim 1, characterized in that, In the model building module, for multi-task models, the overall loss function L all The calculation process is as follows: The input X is encoded to obtain a feature map M. The inputs of different decoders are features M1, M2, ..., M. t The output Y is shown in the following formula: After obtaining the output Y of each task decoder, compare it with the true value Y. * The comparisons are then performed, and the overall loss is calculated using the following function. The overall loss function is composed of the loss functions of each subtask, constructed using a weighted approach with a weighting coefficient of α. i L all =∑ i＝1…t α i L i (Y,Y * )。

Images