Tail category target data enhancement method of multi-task network in automatic driving scene
By building and integrating tail-class target databases in an autonomous driving multi-task network, the problem of insufficient number of tail-class tags is solved, the target detection effect is improved, and the quality of tail-class target data is improved without affecting semantic segmentation. .
Patent Information
- Application Number
- CN202411013589.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
AI Technical Summary
In the autonomous driving multi-task network, the number of tail-class tag instances is small, resulting in poor vehicle detection results. Common data enhancement methods will destroy the semantic structure of the image and affect the effect of semantic segmentation tasks.
By defining the multi-task learning network structure, building the head and tail target databases, fusing the tail data into the batch task training data, and recalculating the truth value labels, generating the enhanced tail target data to ensure that the semantic segmentation task does not Affected.
It effectively improves the model inference effect of target detection tail samples, increases the proportion of tail tag data, and improves the generation of binary graph truth values of semantic segmentation without destroying the semantic segmentation label structure.
Smart Images

Figure CN119091244A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of autonomous driving, and specifically relates to a method for enhancing tail category target data of a multi-task network in an autonomous driving scenario. Background Art
[0002] Extensive research on autonomous driving has revealed the importance of panoramic driving perception systems. When understanding scenes through visual information, panoramic driving perception systems often complete vehicle planning and control through the following information, namely, detection of multiple categories of obstacles in front, identification of drivable areas, and detection of lane lines. The panoramic driving perception multi-task network model based on image processing can well implement these three functions. It aims to share knowledge by training multiple related tasks at the same time, thereby improving the generalization ability and performance of the model. Compared with the inference results of the network model that uses three different functions one by one, the multi-task network model has obvious advantages in inference speed and low computing power.
[0003] However, in multi-task learning, the number of instances in the training dataset under different labels varies greatly. The head class label has sufficient training instances, but the tail class label has only a few training instances, which accounts for a small number of the total number of labels, resulting in poor performance of the model in the tail class of vehicle detection. (Note: In the target detection label category distribution of the dataset, the category label containing a large number of samples is called the head class label, while some category labels with only a very small number of samples are called the tail class label; and the critical number of these category labels will be specifically defined according to the actual application.)
[0004] In the training of conventional network models, data augmentation is a commonly used dataset expansion technique. It uses methods such as geometric transformation, color transformation, and noise addition to generate new training samples by performing various transformations and processing on existing data, thereby increasing the diversity and quantity of datasets. However, in multi-task networks, methods such as Cutout, Mixup, and Cutmix destroy the semantic structure of image data, resulting in poor results in semantic segmentation tasks such as lane lines and drivable areas. Therefore, how to explore data augmentation methods suitable for multi-task networks, especially more effective use of tail class labels to improve the overall performance of multi-task models, has strong practical significance. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a method for enhancing the tail category target data of a multi-task network in an autonomous driving scenario. The method combines the characteristics of multi-task learning and data enhancement technology to effectively transform and expand the original training data, thereby improving the performance of the multi-task learning network in the target detection task without affecting the semantic segmentation task.
[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions.
[0007] The tail category target data enhancement method of the multi-task network in the autonomous driving scenario mainly includes the following steps:
[0008] First, a multi-task learning network structure in an autonomous driving scenario is defined. The multi-task learning network structure includes more than one task branch network, each task branch network corresponds to a specific task, and each task branch network achieves the purpose of information sharing and task collaborative learning by sharing some network layers (such as convolutional layers) or parameters;
[0009] Secondly, construct the head class target and tail class target, and label the tail class target database;
[0010] Finally, during the model training process of the multi-task learning network structure, the tail category data is extracted from the tail category target database and integrated into the current batch task training data (i.e., "batch data", that is, the current data trained in batch order in data training), and the true value label of the current task target data is recalculated. The true value label data is then input into the model of the multi-task learning network structure to obtain the enhanced tail category target data.
[0011] As a further improvement of the above scheme, the task branch network of the multi-task learning network structure includes a target detection branch, a lane line detection branch and a drivable area detection branch.
[0012] As a further improvement of the above solution, the second step of constructing a tail target and labeling a tail target database mainly includes the following steps:
[0013] The first step is to define the tail target and traverse the target detection training data set. According to the target category of the annotated true value, the tail target data annotation box (such as ice cream cones, warning columns, etc.) of the current cycle data is filtered;
[0014] The second step is to convert the annotation coordinate array into the form of "xywh" (i.e. the x-coordinate and y-coordinate of the center of the annotation area, as well as the width w and height h of the annotation area). Use the OpenCV library to extract the annotation area from the image according to the "xywh" coordinates, and then save the extracted annotation area in jpg format.
[0015] In the third step, repeat the first and second steps above until the entire target detection training data set is traversed.
[0016] As a further improvement of the above scheme, the third step mainly includes the following steps:
[0017] The first step is to traverse the task data of the head class target in the training process, and randomly extract at least one labeled tail class data from the tail class target database corresponding to the head class target according to each training task data of the head class target;
[0018] The second step is to select the region of the fused label tail data and calculate the coordinates of the region according to the specific task of the task branch network, and then use the improved Poisson fusion to fuse the label tail data into the selected region of the current batch task training data;
[0019] The third step is to replace the training images of the current batch task training data with the images after fusion of the tail class data, and update the fused label truth value (including the true value label after fusion of the tail class target label, and the recalculation of the drivable area and the labeled binary image of the lane line after being occluded by the fusion label);
[0020] Finally, the operations from the first step to the third step are repeated until the acquisition of the specific task training data is completed, thereby obtaining the enhanced tail category target data.
[0021] Compared with the prior art, the advantages of the present invention are:
[0022] 1. Quickly and reasonably increase the proportion of target detection tail class label data in the data set, and effectively improve the model reasoning effect of target detection tail class samples.
[0023] 2. Regenerate the binary image truth of semantic segmentation without destroying the contextual information of the semantic segmentation label structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is a schematic diagram of a multi-task learning network structure in an autonomous driving scenario according to an embodiment of the present invention;
[0025] Figure 2 This is a flow chart of a label tail target database according to an embodiment of the present invention;
[0026] Figure 3 Schematic diagram of the process of fusing the tail class label database into task-specific training data in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Reference Figure 1 , Figure 2and Figure 3 The method for enhancing tail category target data of a multi-task network in an autonomous driving scenario according to an embodiment of the present invention mainly includes the following steps:
[0029] First, a multi-task learning network structure in an autonomous driving scenario is defined. The multi-task learning network structure includes more than one task branch network, each task branch network corresponds to a specific task, and each task branch network achieves the purpose of information sharing and task collaborative learning by sharing some network layers (such as convolutional layers) or parameters;
[0030] Secondly, construct the head class target and tail class target, and label the tail class target database;
[0031] Finally, during the model training process of the multi-task learning network structure, the tail category data is extracted from the tail category target database and integrated into the current batch task training data, and the true value label of the current task target data is recalculated. Then, the true value label data is input into the model of the multi-task learning network structure to obtain the enhanced tail category target data.
[0032] The task branch network of the multi-task learning network structure includes a target detection branch, a lane line detection branch and a drivable area detection branch.
[0033] The second step constructs a tail target and labels the tail target database, which mainly includes the following steps:
[0034] The first step is to define the tail target and traverse the target detection training data set. According to the target category of the annotated true value, the tail target data annotation box (such as ice cream cones, warning columns, etc.) of the current cycle data is filtered;
[0035] The second step is to convert the annotation coordinate array into "xywh" format, use the OpenCV library to extract the annotation area from the image according to the "xywh" coordinates, and then save the extracted annotation area in jpg format;
[0036] In the third step, repeat the first and second steps above until the entire target detection training data set is traversed.
[0037] The third step mainly includes the following steps:
[0038] The first step is to traverse the task data of the head class target in the training process, and randomly extract at least one labeled tail class data from the tail class target database corresponding to the head class target according to each training task data of the head class target;
[0039] The second step is to select the region of the fused label tail data and calculate the coordinates of the region according to the specific task of the task branch network, and then use the improved Poisson fusion to fuse the label tail data into the selected region of the current batch task training data;
[0040] The third step is to replace the training images of the current batch task training data with the images after fusion of the tail class data, and update the fused label true value;
[0041] Finally, the operations from the first step to the third step are repeated until the acquisition of the specific task training data is completed, thereby obtaining the enhanced tail category target data.
[0042] The following uses a self-built dataset for quantitative analysis and two comparative experiments using 10,000 training data. Conventional data augmentation technology and tail label data augmentation methods are used for indicator comparison. Recall and mAP50 are used for target detection, Accuracy and IOU are used for lane segmentation, and mIOU is used for driving area segmentation.
[0043] Table 1:
[0044]
[0045] In Table 1 above, the use of tail class label data enhancement method improves recall by 3.1% and mAP50 by 1.2% compared to conventional data enhancement methods, while the indicator fluctuation for lane line and drivable area tasks of semantic segmentation is only about 1%. Therefore, the use of tail class label data enhancement method can improve the reasoning effect of target detection without affecting other tasks.
[0046] Compared with the prior art, the advantages of the present invention are:
[0047] 1. Quickly and reasonably increase the proportion of target detection tail class label data in the data set, and effectively improve the model reasoning effect of target detection tail class samples.
[0048] 2. Regenerate the binary image truth of semantic segmentation without destroying the contextual information of the semantic segmentation label structure.
[0049] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.
Claims
1. A method for enhancing tail category target data of a multi-task network in an autonomous driving scenario, characterized in that: The main steps include: First, a multi-task learning network structure in an autonomous driving scenario is defined. The multi-task learning network structure contains more than one task branch network, each task branch network corresponds to a specific task, and each task branch network achieves the purpose of information sharing and task collaborative learning by sharing network layers or parameters; Secondly, construct the head and tail class targets, and label the tail class target database; Finally, during the model training process of the multi-task learning network structure, the tail category data is extracted from the tail category target database and integrated into the current batch task training data, and the true value label of the current task target data is recalculated. Then, the true value label data is input into the model of the multi-task learning network structure to obtain the enhanced tail category target data.
2. The method for enhancing tail category target data of a multi-task network in an autonomous driving scenario according to claim 1, characterized in that: The task branch network of the multi-task learning network structure includes a target detection branch, a lane line detection branch and a drivable area detection branch.
3. The method for enhancing tail category target data of a multi-task network in an autonomous driving scenario according to claim 1, characterized in that: The second step constructs a tail target and labels the tail target database, which mainly includes the following steps: The first step is to define the tail class target and traverse the target detection training data set, and filter the tail class target data annotation box of the current cycle data according to the target category of the annotated true value; The second step is to convert the annotation coordinate array into "xywh" format, use the OpenCV library to extract the annotation area from the image according to the "xywh" coordinates, and then save the extracted annotation area in jpg format; In the third step, repeat the first and second steps above until the entire target detection training data set is traversed.
4. The method for enhancing tail category target data of a multi-task network in an autonomous driving scenario according to claim 1, characterized in that: The third step mainly includes the following steps: The first step is to traverse the task data of the head class target in the training process, and randomly extract at least one labeled tail class data from the tail class target database corresponding to the head class target according to each training task data of the head class target; The second step is to select the region of the fused label tail data and calculate the coordinates of the region according to the specific task of the task branch network, and then use the improved Poisson fusion to fuse the label tail data into the selected region of the current batch task training data; The third step is to replace the training images of the current batch task training data with the images after fusion of the tail class data, and update the fused label true value; Finally, the operations from the first step to the third step are repeated until the acquisition of the specific task training data is completed, thereby obtaining the enhanced tail category target data.
