Methods and systems for data augmentation of rare classes in vulnerable road users

Abstract

The present application relates to the field of deep learning. The present application provides a method for data augmentation of rare classes in vulnerable road users, comprising the following steps: obtaining original data samples of vulnerable road users; preprocessing the original data samples to extract rare classes; generating additional data samples based on a generative model; and mixing the generated additional data samples with the original data samples to form balanced data samples. The present application also provides a system for data augmentation of rare classes in vulnerable road users and a machine-readable storage medium. The present application aims to solve the problem of uneven distribution of training data sets of road participants with different social attributes in the machine learning-based perception system training process of autonomous vehicles, in order to ultimately improve the protection of vulnerable road users.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for data augmentation of rare classes in vulnerable road users (VRU), a system for data augmentation of rare classes in vulnerable road users (VRU) and a machine-readable storage medium. BACKGROUND

[0002] In intelligent driving systems, the perception system plays an important role because it can feedback the current environment of the car and provide information for the downstream system. The accuracy and richness of its information are closely related to the safety and reliability of the car.

[0003] Existing intelligent driving perception systems mostly focus on the identification and positioning of targets such as vehicles, pedestrians, traffic signs, lane lines, passable areas, obstacles and scenes. In real road traffic environment, people are the most important participants in road traffic. In addition to their natural attributes such as their own categories, the social identity attributes of each individual determine their general behavior patterns in road traffic activities, and often contain some behavior meanings unique to their professional identity. For example, a waving hand gesture may be a taxi hailing for an ordinary pedestrian, a traffic police may be directing traffic, and a road construction worker may be directing construction, etc. For intelligent driving systems, identifying the professional information or social attributes of pedestrians not only helps better understand their intentions, but also helps make more accurate and reasonable predictions of the next behavior and movement of each individual pedestrian, thereby greatly improving their environmental perception and decision-making capabilities.

[0004] Using a convolutional neural network for professional classification is a viable option. A classifier based on a convolutional neural network needs a large amount of actually collected data to train the network so that it can learn to distinguish the categories of pedestrians. However, the number of categories such as traffic police, police, soldiers, construction workers, etc. contained in the data collected in general traffic roads is far less than that of ordinary pedestrians. Therefore, the naturally collected data set has a significant problem of uneven data distribution. If a classifier that can distinguish VRU categories is trained on this data set, the classifier will perform well on the majority of categories, but perform poorly on the minority categories. The main reason is that on the one hand, a small number of pictures is not enough for the neural network to learn enough matching relationships, and on the other hand, the evaluation criteria for the model in the training process are not suitable for the extremely uneven distribution of data, which makes the majority of categories play a decisive role in the evaluation results.

[0005] Traditional methods for addressing uneven data distribution mainly fall into two categories. One approach addresses the data itself by performing data mining to identify problems and fill in the gaps in the data by allocating fewer categories. This method requires significant resources and becomes difficult to implement effectively if the data gap reaches a certain scale. The other approach improves the training process by applying different sampling weights to different categories or modifying the loss function to amplify the influence of smaller categories. This method, on the one hand, requires extensive data mining and continuous adjustments during neural network training; on the other hand, its improvement potential is limited, reaching a certain level before significant further improvement becomes difficult. Summary of the Invention

[0006] The present invention aims to provide a method for data augmentation of rare categories among vulnerable traffic participants, a system for data augmentation of rare categories among vulnerable traffic participants, and a machine-readable storage medium.

[0007] According to a first aspect of the present invention, a method for data augmentation of rare categories among vulnerable traffic participants is provided, the method comprising the steps of:

[0008] S1: Obtain raw data samples of vulnerable traffic participants;

[0009] S2: Preprocess the original data sample to extract rare categories;

[0010] S3: Based on a generative model, targeted generation of data for the rare categories is performed to form additional data samples; and

[0011] S4: Mix the generated additional data sample with the original data sample to form a balanced data sample.

[0012] The present invention particularly includes the following technical concept: on the one hand, by generating rare categories among vulnerable traffic participants in a targeted manner, the significant gap between the number of rare categories and rich categories in image classification can be reduced; on the other hand, by remixing the newly generated additional data samples into the original data, the share of rare categories can be improved while maintaining the original training data scale, thus avoiding information loss and sample loss.

[0013] Optionally, the rare category is a category divided according to social attributes, wherein the number of data samples in the category is no more than 15%, preferably 13%, and more preferably 10% of the original number of data samples.

[0014] Here, the screening of the rare categories can be performed in particular in accordance with a predefined share threshold (for example 15% or 13% or 10%). It is also conceivable, however, that the share threshold can be dynamically adjusted in accordance with factors such as the country, region, culture, customs, weather conditions, topographical conditions, etc. of the sample collection.

[0015] Optionally, in step S1, images of the vulnerable road users are collected at a determined time interval.

[0016] Thereby, by specifying a strict time interval for the data collection process, it can be advantageously ensured that the detected data scenarios and the richness of the person categories are ensured and at the same time the number of duplicate samples is reduced.

[0017] Optionally, in step S2, the step of preprocessing comprises:

[0018] In the acquired raw data samples, a bounding box is drawn for the vulnerable road user in the image data and the image data is segmented by means of the bounding box; and

[0019] The segmented image data is labeled, wherein the labeling information at least includes the category of the vulnerable road user

[0020] Thereby, by segmenting and labeling the region of interest in the image, the number distribution of the cumulative data samples of the respective categories can be derived, so that it can be determined on the basis thereof which data category has a small number of samples and is therefore defined as a rare category.

[0021] Optionally, the labeling comprises labeling the posture information of the vulnerable road user by means of manual labeling and / or by means of automatic labeling by means of a posture detection network or a person detection model

[0022] Here, since the social attributes of the vulnerable road user have a significant influence on the meaning of the action posture, the particularity of the social attributes or the professional category of the vulnerable road user can also be reflected in the action posture of the person. By labeling the posture information instead of directly distinguishing the categories, the labeling difficulty can be reduced and the labeling efficiency can be improved, whether for manual labeling or machine labeling, so that time overheads can be saved.

[0023] Optionally, the step S3 comprises inputting the data samples of the rare category into a generative model for training to form additional data samples comprising only the rare category.

[0024] Thereby, the number of data samples to be generated can be more specifically controlled, the data enhancement efficiency can be improved, and the samples can be quickly balanced. Here, the model can be optimized, for example, by continuously adjusting and updating the parameters of the generative model, so that the proportion of the rare category in the output result reaches the desired proportion.

[0025] Optionally, the step S3 comprises: extracting the pose information and the appearance information in the data sample image of the rare category, and inputting the extracted pose information and appearance information into the generative model respectively to generate the mixed restoration image.

[0026] Thus, the pose features and appearance features of the person are decoupled in an advantageous manner, the controllability of each parameter in the generation process is realized, the generated image quality is higher, and the fusion of the appearance and pose of the person is more natural. In addition, the pose features and appearance features from different persons can be recombined with higher freedom to rapidly enrich the sample types.

[0027] Optionally, the generative model is a generative adversarial network or a variational autoencoder and a derivative model thereof.

[0028] Optionally, the method further comprises: training the VRU classification model using the balanced data samples formed by mixing; and

[0029] The training result in the case of using the balanced data samples is evaluated by means of a model evaluation method, in particular a confusion matrix.

[0030] Thus, the following technical advantages are achieved, in particular: by judging the performance of the classification network by means of a suitable model evaluation method, the dynamic changes of the data enhancement result and the classification result can be truly fed back, and based on this, specific parameters (such as convolution kernels, weight vectors) in the VRU classification model can be updated and optimized more targetedly.

[0031] According to a second aspect of the present application, a system for data augmentation of a rare category in vulnerable road users is proposed, comprising:

[0032] A data acquisition module configured to be capable of acquiring original data samples of vulnerable road users;

[0033] A preprocessing module configured to be capable of preprocessing the original data samples to extract the rare category;

[0034] A data generation module configured to be capable of generating additional data samples of the rare category based on a generative model to form balanced data samples; and

[0035] A mixing module configured to be capable of mixing the generated additional data samples with the original data samples to form balanced data samples.

[0036] According to a third aspect of the present application, a machine-readable storage medium is proposed, on which a computer program is stored, the computer program being capable of implementing the method according to the first aspect of the present application when it is run on a computer. Attached Figure Description

[0037] The invention will now be described in more detail with reference to the accompanying drawings, which will provide a better understanding of its principles, features, and advantages. The drawings include:

[0038] Figure 1 A method for data augmentation of rare categories among vulnerable traffic participants is illustrated according to an exemplary embodiment of the present invention.

[0039] Figure 2 A step of a method for data augmentation of rare categories among vulnerable traffic participants according to an exemplary embodiment of the present invention is shown;

[0040] Figure 3 A system for data augmentation of rare categories among vulnerable traffic participants, according to an exemplary embodiment of the present invention, is shown; and

[0041] Figure 4 An example of targeted generation of sample data for rare categories using generative adversarial networks is shown. Detailed Implementation

[0042] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0043] In the context of this specification, vulnerable road users specifically refer to road users who are not protected by a shell (e.g., vehicle body) and are therefore at higher risk of injury or death because they are not surrounded by a protective shell.

[0044] Typically, vulnerable road users (VRUs) are categorized by vehicle type, such as pedestrians, cyclists, scooter riders, and electric vehicle riders. However, this invention does not simply use a single VRU tag or differentiate them based on whether they have a vehicle or the type of vehicle they are riding in; rather, it categorizes VRUs according to their social attributes. Therefore, considering the above factors, the VRU categories according to this invention can specifically include: ordinary pedestrians, police officers, construction workers, military personnel, elderly or disabled people in wheelchairs, infants in vehicles, and children.

[0045] Figure 1 A method for data augmentation of rare categories among vulnerable traffic participants is shown according to an exemplary embodiment of the present invention.

[0046] In step S1, original data samples of vulnerable road users in an autonomous driving scenario are acquired. Here, the scenario in the actual road can be continuously acquired by a data acquisition device (e.g., a camera) and stored in a storage medium.

[0047] Here, the acquired data should have a certain number, especially different categories, to ensure the diversity of picture poses, appearances, and backgrounds, so that the training result of the classification model has good generalization ability.

[0048] In addition, the acquisition process or the storage process can optionally have a fixed time interval specification, and the number of acquisition frames of the same scene should not be greatly different from that of other scenes. Optionally, in the acquisition process, the images can be distinguished by a timestamp representing a unique code to prevent similar individuals from appearing multiple times or repeatedly.

[0049] In step S2, the original data samples are preprocessed to extract rare categories.

[0050] In the sense of the present specification, the rare category is particularly understood as a VRU category whose number is significantly lower than that of ordinary pedestrians in the case of category division according to a certain division standard (e.g., social attributes or occupation types), which can be, for example: criminal police, urban management, traffic police, military personnel, construction workers, disabled persons, etc.

[0051] Here, the preprocessing of the original data samples includes, for example: circumscribing the region of interest (here, the region where the vulnerable participants are located) in the image by a bounding box; and labeling the segmented image data, wherein the labeling information at least includes the category of the image.

[0052] Here, the term "labeling" particularly means assigning a unique code to the object segmented by the bounding box in order to achieve differentiation between different categories. Such differentiation requires category division of VRUs according to their social attributes, rather than using a single VRU label or simply distinguishing by whether there is a vehicle. Here, in particular, persons with special identities in terms of social attributes are labeled.

[0053] The labeling process is generally completed manually, and in some cases, a person detection algorithm (such as OpenPose, Pifpaf, HRNet, etc.) can also be used for extraction. It is worth noting that in order to ensure data diversity, the acquired target person image should contain as many body poses as possible.

[0054] Optionally, the labeling includes labeling the pose information of the VRU.

[0055] Here, since the social attribute of the VRU has a significant impact on the meaning of its action posture, and the accuracy of labeling the character posture information has obvious advantages compared to directly labeling the social attribute category, the accuracy of the training result of the generative model in step S3 can be improved. Therefore, it is conceivable to convert the labeling of the social attribute (category) into a posture estimation problem, for example, the human skeleton point position can be located first, then the shape of the human body is drawn by connecting the key points in the image, and then the posture recognition is performed.

[0056] Exemplarily, the posture information can be expressed by the position of the human key points. For example, common human posture labeling points can include: eyes, nose, mouth, ear, shoulder, elbow, hip, knee, ankle.

[0057] Optionally, a deep convolutional neural network, especially YOLO, can be used as a posture detection network or a character detection model. The input of which is image data, and the output is the predicted character bounding box position and size parameters (x, y, w, h). After the training process, the network parameters are updated and iterated, and the performance of the character posture detection will be optimal.

[0058] In step S3, the data of the rare category is generated directionally based on the generative model to form additional data samples.

[0059] Specifically, the data samples of the rare category can be input into the generative model for training to form additional data samples including only the rare category.

[0060] Exemplarily, for example, the appearance information and action posture information of the characters in each sample data extracted can be combined into a posture database, and then the posture generative adversarial network is trained using the posture database. The posture generative adversarial network is composed of three parts, namely: posture detection network P, generative network G and discriminant network D.

[0061] The generative model is a general term for a way of generating data by a deep learning model, in which the posture generative adversarial network, autoencoder and other ways are widely used in the field of data generation.

[0062] In step S4, the generated additional data samples are mixed with the original data samples to form balanced data samples.

[0063] Optionally, the generated additional data samples can be shuffled before mixing, or random sampling can be performed during the training of the classification model, so as to prevent the decline of the model generalization ability caused by the continuous use of data samples from similar VRU categories during training.

[0064] Optionally, steps S5 and S6 (shown in dashed line) can also be performed next. In step S5, the VRU classification model can be trained with the mixed balanced data samples. Exemplarily, the VRU classification model can be a convolutional neural network.

[0065] Then in step S6, the classification network performance can be evaluated by means of a proper model evaluation method.

[0066] Figure 2 An exemplary embodiment of step S6 of the above method according to the present application is shown. In the present application, the essential structure of the classification network does not change, only the training data used is different, so the performance improvement brought by the additional image data generated by the generative model to the classification network can be evaluated by means of the following model evaluation method, for example.

[0067] In step S61, the data for evaluation (e.g. validation set or test set) is input into the VRU classification model to obtain the category judgment of the classification model;

[0068] In step S62, a confusion matrix is obtained according to the category labels of the pictures. As shown in the following exemplary, the confusion matrix can be a two-dimensional matrix, for example, each row of which corresponds to the label information in the data samples, especially the data labeled as a certain category, and each column of which corresponds to the data judged by the classification model to belong to a certain category. The confusion matrix shows the correct and false judgments of all data categories, as well as the number of different wrong judgment categories.

[0069]

[0070] Table 1

[0071] For example, the number of samples for testing or validation is 100, and the number of samples of the first and second categories is 50 respectively. As shown in Table 1, the current result of the classifier shows that 47 samples belonging to the first category are correctly predicted as the first category, and 3 samples belonging to the first category are incorrectly classified as the second category. Similarly, 48 samples are correctly classified into the second category, and 2 samples are incorrectly classified into the first category.

[0072] In order to obtain better model evaluation effect, three secondary indicators are further extended on the basis of the confusion matrix, so that the number results in the confusion matrix can be converted into a ratio between 0 and 1, which is convenient for standardized measurement. Therefore, in step S63, the accuracy and recall rate of each category are calculated according to the categories of the VRU, so as to calculate the F1 indicator of each category, wherein the precision, recall rate and F1 indicator (also known as F1-Score) are defined as follows:

[0073] Precision:

[0074] Recall:

[0075] F1 Score:

[0076] where TP (True Positive) represents the number of correctly predicted positive samples;

[0077] FP (False Positive) represents the number of incorrectly predicted positive samples;

[0078] TN (True Negative) represents the number of correctly predicted negative samples;

[0079] FN (False Negative) represents the number of incorrectly predicted negative samples;

[0080] Precision represents the accuracy, i.e. the ratio of the number of correctly predicted positive samples to the actual number of positive samples;

[0081] Recall represents the recall, i.e. the ratio of the number of correctly predicted positive samples to the total number of predicted positive samples;

[0082] F1 represents the F1 score, i.e. the harmonic mean of the accuracy and the recall.

[0083] It should be noted that in the above exemplary embodiments, the positive samples refer to the first class and the negative samples refer to the second class.

[0084] In step S64, the F1 scores of the respective classes are averaged, i.e. a parameter for evaluating the classification model is derived.

[0085] In step S65, the parameter derived in step S64 can be compared to a pre-set threshold in order to determine whether the model meets the expected standards.

[0086] Figure 3 A system for data augmentation of rare classes in vulnerable road users is shown, according to one exemplary embodiment of the present application.

[0087] The system 30 comprises a data acquisition module 31, a pre-processing module 32, a data generation module 33 and a mixing module 34.

[0088] The acquisition module 31 can for example be configured as an image sensor or a video sensor configured to be able to take pictures of a road in order to acquire raw data samples of vulnerable road users.

[0089] After the raw data samples have been collected, they are transferred to a pre-processing module 32, in order to pre-process the raw data samples there, in order to extract rare classes. Here, the pre-processing can comprise, inter alia, segmenting the person regions by means of bounding boxes and annotating the person pose information and / or the person classes.

[0090] The training set of raw data samples and / or the extracted rare data samples are then provided, directly or indirectly, e.g. via a mixing module, to a data generation module 33. In the data generation module 33, data of the rare classes are generated directionally based on a generative model, thereby forming additional data samples.

[0091] The generated additional data samples are mixed with the pre-processed raw data samples in a mixing module 34, thereby forming data samples with a balanced data distribution.

[0092] Optionally, these balanced data samples can be further provided to a VRU classifier 35 for training.

[0093] Figure 4 An example of a directed generation of sample data of rare classes using a generative adversarial network is shown.

[0094] First, a raw image containing a person (vulnerable road user) is input into a feature extraction network (not shown) for feature extraction, thereby obtaining decoupled pose features 41 and appearance features 42.

[0095] The extracted pose features 41 and appearance features 42 are then input into generative models 43, 44 for generation, thereby obtaining several pose images and appearance images, respectively. Next, the pose images and appearance images are synthesized in a synthesis network 45 to obtain a restored image 46, which can have, inter alia, mixed pose features and appearance features. This means that the restored image can have, for example, the pose features of a first image and the appearance features of a second image.

[0096] Finally, the generated restored image 46 can be input together with real person images 47 into a discriminator network 48 to discriminate between true and false. Depending on the discrimination result, the generative models can be fed back accordingly, in order to optimize the generation result. In this form of adversarial generation, the quality of the generated images can be greatly improved, laying the foundation for the subsequent optimization of the training result of the VRU classifier.

[0097] Although specific embodiments of the present application have been described in detail hereinabove, they are offered by way of illustration and not by way of limitation. Various substitutions, alterations and modifications can be conceived by those skilled in the art without departing from the spirit and scope of the present application.

Claims

1. A method for data augmentation of rare classes in vulnerable road users, the method comprising the steps of: S1: obtaining original data samples of vulnerable road users; S2: pre-processing the original data samples to extract rare classes, the rare classes are divided according to social attributes, and the rare classes include police, traffic police, construction workers, old people or disabled people in wheelchairs, and babies in cars, the pre-processing includes labeling vulnerable road users, and the labeling information includes the classes of vulnerable road users, and the labeling includes labeling the posture information of vulnerable road users through manual labeling and / or automatic labeling by means of a posture detection network or a person detection model; S3: generating data of the rare classes based on a generative model to form additional data samples, wherein the data samples of the rare classes are input into the generative model for training to form additional data samples including only the rare classes; S4: mixing the generated additional data samples with the original data samples to form balanced data samples so that the proportion of the rare classes reaches an expected proportion; and training a vulnerable road user classification model using the balanced data samples formed by mixing.

2. The method of claim 1, wherein, The rare classes are classes divided according to social attributes, and the number of data samples under the classes is not greater than 15% or 13% or 10% of the number of original data samples.

3. The method of claim 1 or 2, wherein, The step S1 includes: collecting images of vulnerable road users at a determined time interval.

4. The method of claim 1 or 2, wherein, The step S2 includes: In the obtained original data samples, a bounding box is drawn for the vulnerable road users in the image data, and the image data is segmented by the bounding box; and the segmented image data is labeled, wherein the labeling information at least includes the classes of the vulnerable road users.

5. The method of claim 3, wherein, The step S2 includes: In the obtained original data samples, a bounding box is drawn for the vulnerable road users in the image data, and the image data is segmented by the bounding box; and the segmented image data is labeled, wherein the labeling information at least includes the classes of the vulnerable road users.

6. The method of claim 1 or 2 or 5, wherein, The step S3 includes: extracting posture information and appearance information in the data sample images of the rare classes, and inputting the extracted posture information and appearance information into the generative model respectively to generate mixed restoration images.

7. The method of claim 3, wherein, The step S3 includes: extracting posture information and appearance information in the data sample images of the rare classes, and inputting the extracted posture information and appearance information into the generative model respectively to generate mixed restoration images.

8. The method of claim 4, wherein, The step S3 includes: extracting posture information and appearance information in the data sample images of the rare classes, and inputting the extracted posture information and appearance information into the generative model respectively to generate mixed restoration images.

9. The method of any one of claims 1, 2, 5, 7, 8, wherein, The generative model is a generative adversarial neural network or a variational autoencoder and its derivative model.

10. The method of claim 3, wherein, The generative model is a generative adversarial neural network or a variational autoencoder and its derivative model.

11. The method of claim 4, wherein, The generative model is a generative adversarial neural network or a variational autoencoder and its derivative model.

12. The method of claim 6, wherein, The generative model is a generative adversarial neural network or a variational autoencoder and its derivative models.

13. The method of any one of claims 1, 2, 5, 7, 8, 10, 11, 12, wherein, The method further comprises: The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

14. The method of claim 3, wherein, The method further comprises: The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

15. The method of claim 4, wherein, The method further comprises: The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

16. The method of claim 6, wherein, The method further comprises: The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

17. The method of claim 9, wherein, The method further comprises: The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

18. The method of claim 13, wherein, The training result under the use of the balanced data sample is evaluated by means of a model evaluation method.

19. A system (30) for data augmentation of rare classes in vulnerable road users, the system being configured to perform the method according to any one of claims 1-18, the system comprising: a data acquisition module (31) configured to acquire raw data samples of vulnerable road users; a preprocessing module (32) configured to preprocess the raw data samples to extract rare classes; a data generation module (33) configured to directionally generate data of rare classes based on a generative model to form additional data samples; and a mixing module (34) configured to mix the generated additional data samples with the raw data samples to form a balanced data sample.

20. A machine-readable storage medium having stored thereon a computer program for implementing the method according to any one of claims 1-18 when the computer program is run on a computer.

Images