A method and device for abnormal driving data annotation

Abstract

This specification provides a method and device for annotating abnormal driving data. The method includes: dividing the historical driving data of an intelligent vehicle into abnormal driving data and normal driving data based on an abnormal driving data division rule; for the normal driving data, randomly sampling the randomly selected normal driving data and sending it to an inspection expert; receiving a new abnormal driving data division rule extracted by the inspection expert based on the normal driving data, and re-dividing the normal driving data based on the new abnormal driving data division rule; annotating the divided abnormal driving data based on a preset abnormal type; sending the unannotatable abnormal driving data to a annotation expert for the annotation expert to manually annotate the unannotatable abnormal driving data. By applying the technical solution provided in this application, more abnormal driving data for training the perception system can be extracted.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and particularly to a method and device for annotating abnormal driving data. Background Art

[0002] In an intelligent driving system, the perception system takes the data of sensors such as lidar, cameras, millimeter-wave radars, and IMUs, and also takes high-precision map data when necessary as the input of the perception system. Then, through a series of calculations and processes, it provides rich and accurate information about the environment around the intelligent driving vehicle for downstream systems such as planning and control.

[0003] The perception system can implement functions such as object detection and classification, multi-object tracking, and scene understanding. Among them, object detection and classification are used to segment the lidar and camera data into individual obstacles and identify the categories of the obstacles, such as vehicles, pedestrians, bicycles, etc.; multi-object tracking continuously tracks the segmented obstacles and estimates their speeds, etc.; scene understanding is used to extract and identify traffic-related information from the lidar and camera data, such as signal lights, construction areas, special vehicles and crowds, etc. The working process of the perception system is usually as follows: perform object detection, semantic analysis, and scene understanding on single-frame point cloud data, image data, etc., obtain basic obstacle, traffic information, scene semantic information, etc. from a single frame, and then perform object tracking and richer scene understanding based on multi-frame results (continuous multiple single-frame results).

[0004] In an intelligent driving system, the perception system completes the above working process through rules or machine learning; rules are judgment methods set manually and summarized based on existing data. For example, an obstacle traveling along the lane direction and with a speed greater than 20 m / s is identified as a vehicle; machine learning is to train and learn through pre-labeled existing data, learn the features corresponding to the labels, and use these features for result judgment during subsequent prediction. That is to say, both rules and machine learning are based on the learning of existing data. If an unlearned situation occurs in actual applications, the perception system will not be able to work effectively, enter an unstable state, and the judgment of the vehicle's surrounding environment will be abnormal, thus affecting the user's driving experience, driving safety, etc. Summary of the Invention

[0005] In view of this, this application provides a method and device for annotating abnormal driving data to obtain more abnormal driving data for training the intelligent vehicle perception system, thereby improving the perception system.

[0006] According to the first aspect of this application, a method for annotating abnormal driving data is provided. The method includes:

[0007] Divide the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on the abnormal driving data division rule;

[0008] For the normal driving data, randomly sample the randomly selected normal driving data and send it to the inspection expert;

[0009] Receive the new abnormal driving data division rule extracted by the inspection expert based on the normal driving data, and re-divide the normal driving data based on the new abnormal driving data division rule;

[0010] Label the abnormal driving data obtained by division based on the preset abnormal types;

[0011] Send the unlabelable abnormal driving data to the labeling expert for the labeling expert to manually label the unlabelable abnormal driving data; the labeled abnormal driving data is used for the training of the intelligent vehicle perception system.

[0012] Optionally, the step of randomly sampling the randomly selected normal driving data and sending it to the inspection expert includes:

[0013] Randomly select a first preset number of normal driving data from the normal driving data, and send the selected normal driving data to the inspection expert for the inspection expert to make an abnormal judgment to identify the abnormal driving data not identified by the abnormal driving data division rule from the normal driving data;

[0014] Judge whether the number of the normal driving data not randomly selected is not less than a first threshold; the first threshold is not less than the first preset number;

[0015] If the number of the normal driving data not randomly selected is not less than the first threshold, continue to randomly select a first preset number of normal driving data from the normal driving data not randomly selected and send it to the inspection expert.

[0016] Optionally, the method further includes:

[0017] If the number of the normal driving data not randomly selected is less than the first threshold, judge whether the number of the abnormal driving data not identified by the abnormal driving data division rule identified from the normal driving data is not less than a second threshold;

[0018] When the number of the identified abnormal driving data is not less than the second threshold, return a prompt for extracting a new abnormal driving data division rule to the inspection expert to prompt the inspection expert to extract a new abnormal driving data division rule.

[0019] Optionally, sending the abnormally driving data that cannot be labeled to a labeling expert for the labeling expert to manually label the abnormally driving data that cannot be labeled includes:

[0020] Randomly extracting a second preset number of abnormally driving data that cannot be labeled from the abnormally driving data that cannot be labeled, and sending the extracted abnormally driving data that cannot be labeled to the labeling expert for the labeling expert to manually label the abnormally driving data that cannot be labeled;

[0021] Determining whether the number of abnormally driving data that cannot be labeled and have not been randomly extracted is not less than a third threshold; the third threshold is not less than the second preset number;

[0022] If the number of abnormally driving data that cannot be labeled and have not been randomly extracted is not less than the third threshold, continue to randomly extract a second preset number of abnormally driving data from the abnormally driving data that cannot be labeled and have not been randomly extracted and send them to the labeling expert.

[0023] Optionally, the method further includes:

[0024] If the number of abnormally driving data that cannot be labeled and have not been randomly extracted is less than the third threshold, return a new abnormal type entry prompt to the labeling expert to prompt the labeling expert to determine and enter a new abnormal type based on the label;

[0025] After adding a new abnormal type, label the abnormally driving data obtained by division based on the abnormal type; if there is still abnormally driving data that cannot be labeled, send the still unlabeled abnormally driving data to the labeling expert for the labeling expert to manually label the abnormally driving data that cannot be labeled.

[0026] Optionally, the method further includes:

[0027] Adding metadata to the abnormally driving data, where the metadata is characterized by a number of information factors and is used to describe the scenario information and vehicle information of the abnormally driving data;

[0028] Determining an information factor combination, and constructing simulated abnormally driving data based on the information factor combination for training an intelligent vehicle perception system.

[0029] Optionally, the method further includes:

[0030] For each information factor, summarize the abnormally driving data with that information factor;

[0031] Save the summarized abnormally driving data to a database corresponding to the information factor.

[0032] According to a second aspect of the present application, there is provided an apparatus for abnormal driving data annotation, the apparatus comprising:

[0033] A data division module, configured to divide the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on an abnormal driving data division rule;

[0034] A data extraction module, configured to, for the normal driving data, send the randomly extracted normal driving data to an inspection expert by means of random sampling;

[0035] A rule receiving module, configured to receive a new abnormal driving data division rule extracted by the inspection expert based on the normal driving data, and re-divide the normal driving data based on the new abnormal driving data division rule;

[0036] A data annotation module, configured to annotate the divided abnormal driving data based on a preset abnormal type;

[0037] A data sending module, configured to send the abnormal driving data that cannot be annotated to an annotation expert for the annotation expert to perform manual annotation on the abnormal driving data that cannot be annotated; the annotated abnormal driving data is used for the training of the intelligent vehicle perception system.

[0038] According to a third aspect of the present application, there is provided an electronic device, the electronic device comprising a communication interface, a processor, a memory, and a bus, and the communication interface, the processor, and the memory are interconnected with each other through the bus; machine-readable instructions are stored in the memory, and the processor executes the steps in the foregoing method for abnormal driving data annotation by calling the machine-readable instructions.

[0039] According to a fourth aspect of the present application, there is provided a machine-readable storage medium, the machine-readable storage medium storing machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the steps in the foregoing method for abnormal driving data annotation are implemented.

[0040] For the technical solution provided by this application regarding the annotation of abnormal driving data, on the one hand, historical driving data is divided into abnormal driving data and normal driving data through the abnormal driving data division rule; for the normal driving data, the extracted normal driving data is further sent to inspection experts by means of random sampling to extract new abnormal driving data division rules from the normal driving data, and then the normal driving data is re-divided based on the updated abnormal driving data division rule. By continuously iterating new abnormal driving data division rules in combination with manual inspection, more abnormal driving data can be extracted from historical driving data, including more abnormal situations, such as long tails; on the other hand, the abnormal types of abnormal driving data are annotated; for the abnormal driving data that cannot be annotated, it is sent to annotation experts to complete manual annotation, and the annotation of all abnormal driving data is completed in combination with manual annotation. The annotated abnormal driving data is used for the training of the intelligent vehicle perception system, greatly enriching the abnormal driving data samples for perception system training, and thus improving the prediction accuracy and stability of the perception system.

[0041] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this specification, and are used together with the specification to explain the principles of this specification.

[0043] Figure 1 It is a schematic flowchart of a method for annotating abnormal driving data shown in this application.

[0044] Figure 2 It is a schematic flowchart of a method for inspecting normal driving data shown in this application.

[0045] Figure 3 It is a schematic flowchart of a method for annotating abnormal driving data shown in this application.

[0046] Figure 4 It is a schematic flowchart of a method for constructing simulated abnormal driving data shown in this application.

[0047] Figure 5 It is a schematic flowchart of a method for managing abnormal driving data shown in this application.

[0048] Figure 6 It is a hardware structure diagram of an electronic device shown in this application.

[0049] Figure 7 It is a schematic block diagram of a device for annotating abnormal driving data shown in this application.

[0050] Figure 8 It is a schematic block diagram of a data extraction module shown in the present application.

[0051] Figure 9 It is a schematic block diagram of a data annotation module shown in the present application.

[0052] Figure 10 It is a schematic block diagram of a simulation abnormal driving data construction module shown in the present application.

[0053] Figure 11 It is a schematic block diagram of an abnormal driving data management sub-module shown in the present application. Detailed Description of the Invention

[0054] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0055] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the", and "said" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0056] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0057] To avoid an unstable state in the intelligent vehicle perception system, the training of the intelligent vehicle perception system can be completed through abnormal driving data, thereby improving the intelligent vehicle perception system. In the existing technical solutions, abnormal driving data is mainly obtained through the following two methods:

[0058] One is through manual reporting. However, there is a high possibility of oversight in manual reporting, and some situations may not be noticed, resulting in the omission of some abnormal driving data that is not reported.

[0059] The other is to obtain abnormal driving data through virtual scenarios, data synthesis, etc. The specific implementation process is as follows: In the simulation system, input real data or scenario description data, such as pedestrians, vehicles, overlap, and generate simulation data based on the input data; input the simulation data into the perception system to obtain the perception system output data carrying the determination result of the perception system; compare whether the perception system output data is consistent with the input data. If not, it is abnormal driving data.

[0060] Among them, the real data can be driving data that has been determined to be abnormal driving data; the generation of simulation data based on the input data includes: if the input data is driving data, add scenario constraints on the basis of the driving data, such as limiting it to rainy days, to generate simulation data; if the input data is scenario description data, virtual physics engines, lidar simulation software, or lidar simulation algorithms can be used to generate simulation data.

[0061] However, the input data is known and conventional in advance, and there is no way to cover other situations such as long tails.

[0062] That is to say, existing methods for obtaining abnormal driving data all have the problem of omission of abnormal driving data.

[0063] To solve the above defects, this application proposes a method for annotating abnormal driving data, which can extract more abnormal driving data from existing historical driving data of intelligent vehicles for improving the perception system.

[0064] Among them, the historical driving data may include: the original data collected by various sensors of the intelligent vehicle, and the determination result after the intelligent vehicle perception system determines the original data.

[0065] For example, the original data in a certain historical driving data is the point cloud data of the lidar, and the determination result is that the vehicle driving speed is 15 m / s.

[0066] The abnormal driving data generally refers to the driving data for which the perception system gives abnormal results. For example, there are unknown objects in the images taken by the intelligent vehicle during driving. Among them, for some abnormal driving data, the determination result can reflect the abnormality. For some abnormal driving data, the determination result cannot intuitively reflect the abnormality and requires manual intervention to identify through abnormal judgment.

[0067] To implement the technical solution of this application, before annotating abnormal driving data, data preparation needs to be carried out first; the data preparation is to extract the historical driving data of the intelligent vehicle from the corresponding storage medium to provide a data basis for subsequent annotation of abnormal driving data; the extracted historical driving data includes various sensor data carrying the determination results of the abnormal perception system, for example, the point cloud data of the lidar, the image data of the camera, the pose and positioning information of the IMU (Inertial Measurement Unit), and the high-precision map data corresponding to the position where the intelligent vehicle is located at that time, etc.

[0068] In this application, the data preparation can be to read data from the in-vehicle storage medium, or to read data from the classification database, or to read data from the offline data warehouse. This specification does not specifically limit the way of data preparation.

[0069] It should be noted that for the abnormalities that have been reported manually, according to the manually reported abnormal information, such as the vehicle and time, they can be filtered out from the historical driving data prepared in the data preparation stage first, and the subsequent operations will not be performed.

[0070] Furthermore, since the historical driving data prepared in the data preparation stage may come from different intelligent vehicles, and the data formats of the different intelligent vehicles may be different, in order to enable the historical driving data to use the same abnormal driving data annotation process, it is necessary to perform data preprocessing on the historical driving data to normalize various different formats of data into the same format.

[0071] In this application, during the process of data normalization, the historical driving data can also be filtered to eliminate the historical driving data that does not meet the requirements, for example, the historical driving data with incomplete data storage and the historical driving data that does not include high-precision map data.

[0072] After the data preparation and data preprocessing are completed, the corresponding abnormal driving data is discovered from the historical driving data through the technical solution of this application.

[0073] Next, the embodiments of this specification will be described in detail.

[0074] Figure 1 It is a schematic flowchart of a method for annotating abnormal driving data shown in this application, as Figure 1 shown. The method includes the following steps:

[0075] Step S101, divide the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on the abnormal driving data division rule.

[0076] Among them, the abnormal driving data division rule can be used to determine abnormal driving data. The abnormal driving data division rule may include: the abnormal driving data can characterize that the state of the intelligent vehicle changes drastically during actual driving, such as the speed change is not less than the speed change threshold, the acceleration change is not less than the speed change threshold, braking, sudden stop, etc. In these cases, it can be determined that the historical driving data conforms to the abnormal driving data division rule, and the corresponding historical driving data is abnormal driving data; both the speed change threshold and the acceleration change threshold can vary flexibly according to the actual situation;

[0077] The abnormal driving data can characterize that the intelligent vehicle stops for too long during actual driving, such as the stop time exceeds the preset time threshold. In this case, it can be determined that the historical driving data conforms to the abnormal driving data division rule, and the corresponding historical driving data is abnormal driving data; the preset time threshold can vary flexibly according to the actual situation;

[0078] The abnormal driving data can characterize that there are unknown objects in the images captured by the camera during the actual driving of the intelligent vehicle. In this case, it can be determined that the historical driving data conforms to the abnormal driving data division rule, and the corresponding historical driving data is abnormal driving data;

[0079] The abnormal driving data can characterize that when the intelligent vehicle combines the high-precision map during actual driving, the scene and objects that do not match are abnormal driving data. For example, a pedestrian appears in the middle of an intersection. In this case, it can be determined that the historical driving data conforms to the abnormal driving data division rule, and the corresponding historical driving data is abnormal driving data;

[0080] Here is only an exemplary description of the pre-established rules, without specific limitations.

[0081] If the historical driving data does not conform to the abnormal data driving division rule, the corresponding historical driving data is normal driving data.

[0082] Step S103, for the normal driving data, randomly sample the randomly selected normal driving data and send it to the inspection expert.

[0083] Due to the limitations of the existing abnormal driving data division rules, not all abnormal driving data in the historical driving data can be extracted. Among the historical driving data classified as normal driving data, there may still be missing abnormal driving data.

[0084] To solve the above situation, a random sampling method is adopted to randomly extract normal driving data from the normal driving data and send it to the inspection expert for the inspection expert to detect the abnormal driving data therein, and then a new abnormal driving data division rule is extracted based on the abnormal driving data, so that more abnormal driving data can be divided based on more abnormal driving data division rules in the future.

[0085] Step S105: Receive the new abnormal driving data division rule extracted by the inspection expert based on the normal driving data, and re-divide the normal driving data based on the new abnormal driving data division rule.

[0086] Receive the new abnormal driving data division rule extracted in step S103. The normal driving data can be re-divided using the new abnormal driving data division rule, or the normal driving data can be re-divided using the expanded abnormal driving data division rule, or the expanded abnormal driving data division rule can be used to re-divide all historical driving data to obtain more abnormal driving data.

[0087] Step S107: Label the divided abnormal driving data based on the preset abnormal types.

[0088] Since the amount of abnormal driving data obtained by division is very large and its specific abnormalities are various, in order to facilitate the targeted management of the abnormal driving data, the abnormal driving data is labeled according to the preset abnormal types.

[0089] Among them, the preset abnormal types include: the object type in the image based on the camera is unknown, the target detection segmentation error based on the lidar, the speed tracking error based on the fusion sensor, etc.; for example, if there is an object for which the perception system does not give a category in a certain data, then the data is labeled as the object type in the image based on the camera is unknown; here is only an exemplary description of the preset abnormal types, without specific limitations.

[0090] For each preset abnormal type, use the preset abnormal type to match the abnormal driving data, and add the label of the preset abnormal type to the successfully matched abnormal driving data.

[0091] Step S109: Send the abnormal driving data that cannot be labeled to the labeling expert for the labeling expert to manually label the abnormal driving data that cannot be labeled; the labeled abnormal driving data is used for the training of the intelligent vehicle perception system.

[0092] Consistent with the existing abnormal driving data division rules, the existing abnormal types also have limitations. Among the abnormal driving data mentioned above, there may be abnormal driving data that cannot be labeled by the existing abnormal types.

[0093] To complete the labeling of the abnormal driving data that cannot be labeled by the existing abnormal types, the unlabeled abnormal driving data is sent to the labeling expert for manual labeling, thereby completing the labeling of all abnormal driving data for use in the training of the intelligent vehicle perception system.

[0094] In the above technical solution, on the one hand, the historical driving data is divided into abnormal driving data and normal driving data through the abnormal driving data division rules; for the normal driving data, the extracted normal driving data is further sent to the inspection expert by means of random sampling to extract new abnormal driving data division rules from the normal driving data, and then the historical driving data is re-divided based on the updated abnormal driving data division rules. By continuously iterating the new abnormal driving data division rules, more abnormal driving data, including more abnormal situations such as long tails, can be extracted from the historical driving data. On the other hand, the abnormal types of the abnormal driving data are labeled; for the abnormal driving data that cannot be labeled, it is sent to the labeling expert to complete the manual labeling, and the labeling of all abnormal driving data is completed in combination with the manual labeling. The labeled abnormal driving data is used for the training of the intelligent vehicle perception system, greatly enriching the abnormal driving data samples for the training of the perception system, and thus improving the prediction accuracy and stability of the perception system.

[0095] Figure 2 It is a schematic flowchart of a method for inspecting normal driving data shown in this application. As Figure 2 shown, on the basis of the embodiment shown in Figure 1 the method for inspecting normal driving data includes the following steps:

[0096] Step S202: Randomly extract a first preset number of normal driving data from the normal driving data and send the extracted normal driving data to the inspection expert for abnormal judgment to identify the abnormal driving data that cannot be identified by the abnormal driving data division rules from the normal driving data.

[0097] Among them, the purpose of the random extraction is to quickly extract diverse normal driving data. Since random extraction gives each normal driving data an equal chance of being extracted, it has the greatest possibility of showing certain characteristics of all normal driving data in the extracted normal driving data.

[0098] The first preset quantity can be determined by the inspection expert based on the quantity of the divided normal driving data. For different quantities of normal driving data, the corresponding first preset quantity can be different or the same. This application does not make specific limitations on this.

[0099] In this embodiment, assume that the quantity of the normal driving data divided in step S101 is 1063. Then the first preset quantity can be 100, that is, 100 pieces of normal driving data are randomly selected from 1063 pieces of normal driving data and sent to the inspection expert for the inspection expert to make an abnormality judgment.

[0100] For example, assume that the 100 pieces of normal driving data extracted include:

[0101] Driving data A with the determination result of the perception system being rainy day, at an intersection, the distance between two vehicles being 5.2 m, the driving speed of the vehicle itself (the forward direction of the vehicle is the positive direction) being 8 m / s, and the driving speed of the rear vehicle being 15 m / s. When the inspection expert makes an abnormality judgment on the driving data A, it is found that there are no traffic accidents such as collisions in the real scene corresponding to the driving data A. However, based on the determination result of the perception system it carries, the speed difference between the rear vehicle and the vehicle itself is greater than the distance between the two vehicles, and there may be traffic accidents such as collisions. Therefore, the inspection expert determines that the driving data A is abnormal driving data;

[0102] Driving data B with the determination result of the perception system being foggy, there being road signs, a car 1 with a vehicle length of 6 m, a vehicle height of 4 m, and a vehicle width of 1 m, and a car 2 with a vehicle length of 6 m, a vehicle height of 4 m, and a vehicle width of 1 m. When the inspection expert makes an abnormality judgment on the driving data B, it is found that in the determination result of the perception system carried by the driving data B, the vehicle widths of both vehicles are less than the vehicle width of a general car, and the sum of the vehicle widths of the two vehicles is still less than the vehicle width of a general car, and the object size does not match the object category. And it is found that in the real scene corresponding to the driving data B, there is only one car blocked by the road sign. The inspection expert can determine that the driving data B is abnormal driving data.

[0103] Step S204, determine whether the quantity of the normal driving data not randomly selected is not less than the first threshold; the first threshold is not less than the first preset quantity.

[0104] To identify as many abnormal driving data not identified by the abnormal driving data division rule as possible from the normal driving data, it is necessary to randomly select normal driving data from the normal driving data multiple times for abnormality judgment.

[0105] Among them, the first threshold can be flexibly set by the inspection expert based on the actual situation, and its function is to determine whether it is necessary to continue randomly extracting normal driving data for the inspection expert to make abnormal judgments; the first threshold can be equal to the first preset quantity or greater than the first preset quantity, and the present application does not make specific limitations thereto.

[0106] If the quantity of the normal driving data that has not been randomly extracted is not less than the first threshold, then step S206 is executed; if the quantity of the normal driving data that has not been randomly extracted is less than the first threshold, then step S208 is executed.

[0107] In this embodiment, it is assumed that the first threshold is equal to the first preset quantity and is equal to 100.

[0108] Step S206, if the quantity of the normal driving data that has not been randomly extracted is not less than the first threshold, then continue to randomly extract the first preset quantity of normal driving data from the normal driving data that has not been randomly extracted and send it to the inspection expert.

[0109] Continuing with the above example, the quantity of the normal driving data is 1063, the first preset quantity and the first threshold are both 100. After randomly extracting 100 pieces of normal driving data for abnormal judgment for the first time, the quantity of the remaining normal driving data that has not been randomly extracted is 963, and its quantity is greater than the first threshold (963>100), then continue to randomly extract 100 pieces of normal driving data for abnormal judgment. And so on, it is necessary to randomly extract 10 times without replacement; for each time 100 pieces of normal driving data are extracted, the inspection expert makes abnormal judgments one by one.

[0110] Step S208, if the quantity of the normal driving data that has not been randomly extracted is less than the first threshold, then judge whether the quantity of the abnormal driving data that has not been identified by the abnormal driving data division rule identified from the normal driving data is not less than the second threshold.

[0111] Among them, the second threshold is also flexibly set by the inspection expert based on the actual situation, and its function is to judge whether a new abnormal driving data division rule can be extracted from the abnormal driving data identified through abnormal judgment; the second threshold can be equal to the first threshold or not equal to the first threshold, and the present application does not make specific limitations thereto.

[0112] After determining that it is not necessary to continue randomly extracting normal driving data for abnormal judgment, count the quantity of the abnormal driving data identified during the abnormal judgment process, compare the quantity with the second threshold. If the quantity is not less than the second threshold, then step S210 is executed.

[0113] In this embodiment, it is assumed that the second threshold is equal to 500.

[0114] Continuing with the above example, after randomly extracting 100 pieces of normal driving data from the normal driving data that has never been randomly selected for the 10th time for abnormal judgment, the number of normal driving data that has still not been randomly selected is 63, and since this number is less than the first threshold (63 < 100), the random extraction is stopped. The number of abnormal driving data identified during the previous 10 abnormal judgment processes is counted as 780, and since this number is greater than the second threshold (780 > 500), step S210 is executed.

[0115] Step S210, when the number of the identified abnormal driving data is not less than the second threshold, return a prompt for extracting a new abnormal driving data division rule to the inspection expert, so as to prompt the inspection expert to extract a new abnormal driving data division rule.

[0116] Continuing with the above example, the number of abnormal driving data identified through the abnormal judgment of the inspection expert is greater than the second threshold, reminding the inspection expert to extract a new abnormal driving data division rule from the abnormal driving data; the inspection expert can, based on the identified abnormal driving data A and other driving data similar to its abnormality, such as the abnormality that the distance between two vehicles is 4.6m, but the speed of the vehicle in front is 10m / s and the speed of the following vehicle is 15m / s, extract a new abnormal division rule: historical driving data where the speed difference between the latter object and the former object of two objects within 10m is not less than 2m / s is abnormal driving data; based on the identified abnormal driving data B and other driving data similar to its abnormality, such as the abnormality that a large truck is divided into two small cars due to road sign occlusion, extract a new abnormal division rule: historical driving data where the object size does not match the object category is abnormal driving data.

[0117] It should be noted that if there is a threshold in the newly extracted abnormal division rule, the threshold can be flexibly set by the inspection expert based on the actual situation.

[0118] Figure 3 is a flowchart showing a method for annotating abnormal driving data according to the present application. As Figure 3 shown, on the basis of the embodiment shown in Figure 1 the method for annotating abnormal driving data includes the following steps:

[0119] Step S301, randomly extract a second preset number of unannotatable abnormal driving data from the unannotatable abnormal driving data, and send the extracted unannotatable abnormal driving data to the annotation expert for the annotation expert to perform manual annotation on the unannotatable abnormal driving data.

[0120] Among them, the purpose of the random extraction is to quickly extract diverse unlabelable abnormal driving data. Since the random extraction gives each unlabelable abnormal driving data an equal chance of being extracted, it is most likely that certain characteristics of all unlabelable abnormal driving data are manifested in the extracted unlabelable abnormal driving data.

[0121] The second preset quantity is formulated by the inspection expert based on the quantity of all unlabelable abnormal driving data. For different quantities of unlabelable abnormal driving data, the corresponding second preset quantity can be different or the same, and the present application does not make specific limitations thereto.

[0122] In this embodiment, assume that the quantity of unlabelable abnormal driving data that cannot be labeled in step S109 is 208. Then the second preset quantity can be 50, that is, 50 abnormal driving data are randomly extracted from 208 unlabelable abnormal driving data and sent to the labeling expert for manual labeling by the labeling expert.

[0123] Continuing with the above example, the abnormality of driving data A cannot be successfully labeled by the preset abnormal type. Assume that the extracted 50 abnormal driving data include the driving data A. When the labeling expert labels the 50 abnormal driving data, the labeling expert can add a label to the driving data A: speed perception error based on lidar.

[0124] Step S303, determine whether the quantity of unlabelable abnormal driving data that has not been randomly extracted is not less than a third threshold; the third threshold is not less than the second preset quantity.

[0125] To reduce the time consumed by manual labeling, it is necessary to set a third threshold for determining whether manual labeling is still required; the third threshold is flexibly set by the labeling expert based on the actual situation, and can be equal to the second preset quantity or greater than the second preset quantity, and the present application does not make specific limitations thereto.

[0126] If the quantity of unlabelable abnormal driving data that has not been randomly extracted is not less than the third threshold, then execute step S305; if the quantity of unlabelable abnormal driving data that has not been randomly extracted is less than the third threshold, then execute step S307.

[0127] In this embodiment, assume that the third threshold is equal to 110.

[0128] Step S305, if the quantity of unlabelable abnormal driving data that has not been randomly extracted is not less than the third threshold, then continue to randomly extract the second preset quantity of abnormal driving data from the unlabelable abnormal driving data that has not been randomly extracted and send them to the labeling expert.

[0129] Continuing with the above example, the number of abnormally driving data that cannot be labeled is 208, the second preset number is 50, and the third threshold is 110. After randomly extracting 50 unlabeled abnormally driving data for manual labeling for the first time, the number of unlabeled abnormally driving data that have not been randomly extracted is 158, and its number is greater than the third threshold (158 > 110), so continue to randomly extract 50 unlabeled abnormally driving data for manual labeling. And so on, it is necessary to randomly extract 2 times without replacement; for each time 50 unlabeled abnormally driving data are extracted, the labeling expert manually labels them one by one.

[0130] Step S307, if the number of unlabeled abnormally driving data that have not been randomly extracted is less than the third threshold, return a new abnormal type entry prompt to the labeling expert to prompt the labeling expert to determine and enter a new abnormal type based on the labeling.

[0131] After determining that there is no need to continue randomly extracting normal driving data for abnormal judgment, prompt the labeling expert to enter a new abnormal type for subsequent labeling of abnormally driving data.

[0132] Continuing with the above example, the labeling expert can enter a new abnormal type: speed perception error based on lidar.

[0133] Step S309, after adding a new abnormal type, label the abnormally driving data obtained by division based on the abnormal type; if there are still abnormally driving data that cannot be labeled, send the still unlabeled abnormally driving data to the labeling expert for the labeling expert to manually label the unlabeled abnormally driving data.

[0134] After adding a new abnormal type, based on the new abnormal type and the original abnormal types, re-label all the abnormally driving data to determine whether there are abnormally driving data that meet the new abnormal type among the originally successfully labeled abnormally driving data. If there are still abnormally driving data that cannot be labeled, to complete the labeling of all the abnormally driving data, send the still unlabeled abnormally driving data to the labeling expert for manual labeling

[0135] In this application, to obtain more abnormally driving data, a method for constructing simulated abnormally driving data is provided. Figure 4 It is a schematic flowchart of a method for constructing simulated abnormally driving data shown in this application. As Figure 4 shown, on the basis of the embodiment shown in Figure 1 the method further includes the following steps:

[0136] Step S402: Add metadata to the abnormal driving data. The metadata is characterized by several information factors and is used to describe the scenario information and vehicle information of the abnormal driving data.

[0137] Among them, the scenario information includes: weather, such as sunny, rainy, foggy; obstacles, such as vehicles, pedestrians; road signs; occlusion; intersections; time. Here is only an exemplary description of the scenario information, without specific limitations.

[0138] The vehicle information includes: vehicle model; configurations of various sensors. Here is only an exemplary description of the vehicle information, without specific limitations.

[0139] Continuing with the above examples, metadata including rainy weather, vehicle as the obstacle, and intersection can be added to driving data A; metadata including fog, vehicle as the obstacle, road signs, and occlusion can be added to driving data B.

[0140] Step S404: Determine the combination of information factors and construct simulated abnormal driving data based on the combination of information factors for training the intelligent vehicle perception system.

[0141] The combination of information factors can be selected by the construction expert or randomly selected by the device. This application does not have specific limitations on this.

[0142] Continuing with the above examples, fog, vehicle as the obstacle, road signs, and occlusion can be selected to construct more simulated abnormal driving data; rainy weather, intersection, and vehicle as the obstacle can also be selected to construct more simulated abnormal driving data. The simulated abnormal driving data is constructed by the simulation system according to the determined combination of information factors.

[0143] Adding metadata including several information factors to each abnormal driving data enables the construction of more abnormal driving data based on the information factors.

[0144] In this application, to more conveniently manage the abnormal driving data, a method for managing abnormal driving data is provided. Figure 5 It is a schematic flowchart of a method for managing abnormal driving data shown in this application. As Figure 5 shown, based on the embodiment shown in Figure 4 , the method further includes the following steps:

[0145] Step S501: For each information factor, summarize the abnormal driving data with this information factor.

[0146] After adding metadata to all abnormal driving data, count the information factors included in each metadata, and establish a corresponding database for each information factor; then, for each information factor, summarize the abnormal driving data with that information factor.

[0147] Continuing with the above example, the metadata includes information factors such as rainy day, fog, obstacle being a vehicle, crossroads, road sign, and occlusion. For these 6 information factors, 6 corresponding databases are established respectively.

[0148] For the information factor of rainy day, driving data A has this information factor; for the information factor of fog, driving data B has this information factor; for the information factor of obstacle being a vehicle, both driving data A and driving data B have this information factor; for the information factor of crossroads, driving data A has this information factor; for the information factor of road sign, driving data B has this information factor; for the information factor of occlusion, driving data B has this information factor.

[0149] Step S503, save the summarized abnormal driving data to the database corresponding to the information factor.

[0150] In the metadata of each abnormal driving data, there may be several information factors. Save each abnormal driving data to the database corresponding to the information factors it includes, so that it can be more convenient to extract the corresponding abnormal driving data based on the information factors for the training and testing of the perception system later.

[0151] Continuing with the above example, driving data A is stored in the database corresponding to rainy day; driving data A and driving data B are stored in the database corresponding to obstacle being a vehicle; driving data A is corresponding in the database corresponding to crossroads; driving data B is stored in the database corresponding to road sign; driving data B is stored in the database corresponding to occlusion.

[0152] Establish a database for each information factor, and save the abnormal driving data with that information factor to its database, which is convenient for the management of abnormal driving data and the extraction of the required abnormal driving data based on the information factor, making the training of the intelligent vehicle perception system more convenient.

[0153] Figure 6 It is the hardware structure diagram of an electronic device shown in this application. As Figure 6As shown, the electronic device includes: a processor 610, a network interface 620, a machine-readable storage medium 630, a non-volatile memory 640, and a bus 650; wherein, the processor 610, the network interface 620, and the machine-readable storage medium 630 communicate with each other through the bus 650. The processor 610 can execute the method for annotating abnormal driving data described above by reading and executing the machine-executable instructions corresponding to the method logic for annotating abnormal driving data in the machine-readable storage medium 630.

[0154] The machine-readable storage medium 630 mentioned in this application can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0155] Corresponding to the embodiments of the foregoing method, this specification also provides embodiments of a device.

[0156] Figure 7 is a schematic block diagram of a device for annotating abnormal driving data shown in this application. As Figure 7 shown, the device includes:

[0157] A data partitioning module 710, configured to partition the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on an abnormal driving data partitioning rule;

[0158] A data extraction module 720, configured to, for the normal driving data, randomly sample the randomly extracted normal driving data and send it to an inspection expert;

[0159] A rule receiving module 730, configured to receive a new abnormal driving data partitioning rule extracted by the inspection expert based on the normal driving data, and re-partition the normal driving data based on the new abnormal driving data partitioning rule;

[0160] A data annotation module 740, configured to annotate the partitioned abnormal driving data based on a preset abnormal type;

[0161] A data sending module 750, configured to send the unannotatable abnormal driving data to an annotation expert for the annotation expert to perform manual annotation on the unannotatable abnormal driving data; the annotated abnormal driving data is used for the training of the intelligent vehicle perception system.

[0162] In this embodiment, to obtain more abnormal driving data, the device further includes:

[0163] A simulated abnormal driving data construction module 760 ( Figure 7 not shown in the figure) for constructing simulated abnormal driving data.

[0164] Figure 8 is a schematic block diagram of a data extraction module shown in this application. As Figure 8 shown, on the basis of Figure 7 , the data extraction module 720 further includes:

[0165] A normal driving data extraction sub-module 721 for randomly extracting a first preset number of normal driving data from the normal driving data and sending the extracted normal driving data to an inspection expert for the inspection expert to perform abnormal judgment to identify abnormal driving data not identified by the abnormal driving data division rule from the normal driving data.

[0166] A normal driving data quantity judgment sub-module 722 for judging whether the quantity of the normal driving data not randomly extracted is not less than a first threshold; the first threshold is not less than the first preset number.

[0167] A normal driving data continuous extraction sub-module 723 for, if the quantity of the normal driving data not randomly extracted is not less than the first threshold, continuously randomly extracting a first preset number of normal driving data from the normal driving data not randomly extracted and sending them to the inspection expert.

[0168] An abnormal driving data quantity judgment sub-module 724 for, if the quantity of the normal driving data not randomly extracted is less than the first threshold, judging whether the quantity of the abnormal driving data not identified by the abnormal driving data division rule identified from the normal driving data is not less than a second threshold.

[0169] A prompt inspection expert sub-module 725 for, when the quantity of the identified abnormal driving data is not less than the second threshold, returning a prompt for extracting a new abnormal driving data division rule to the inspection expert to prompt the inspection expert to extract a new abnormal driving data division rule.

[0170] It should be noted that in an implementable embodiment, the normal driving data extraction sub-module 721 and the normal driving data continuous extraction sub-module 723 may be the same module for randomly extracting a first preset number of normal driving data from the normal driving data.

[0171] Figure 9 is a schematic block diagram of a data annotation module shown in this application. As Figure 9 shown, inFigure 7 Based on the above embodiments, the data annotation module 740 further includes:

[0172] A data extraction sub-module 741, configured to randomly extract a second preset number of unannotatable abnormal driving data from the unannotatable abnormal driving data, and send the extracted unannotatable abnormal driving data to an annotation expert for the annotation expert to perform manual annotation on the unannotatable abnormal driving data.

[0173] A quantity judgment sub-module 742, configured to judge whether the quantity of the unannotatable abnormal driving data that has not been randomly extracted is not less than a third threshold; the third threshold is not less than the second preset quantity.

[0174] A data continuous extraction sub-module 743, configured to, if the quantity of the unannotatable abnormal driving data that has not been randomly extracted is not less than the third threshold, continue to randomly extract a second preset number of abnormal driving data from the unannotatable abnormal driving data that has not been randomly extracted and send it to the annotation expert.

[0175] A prompt annotation expert sub-module 744, configured to, if the quantity of the unannotatable abnormal driving data that has not been randomly extracted is less than the third threshold, return a new abnormal type entry prompt to the annotation expert to prompt the annotation expert to determine and enter a new abnormal type based on the annotation.

[0176] A re-annotation sub-module 745, configured to, after adding a new abnormal type, perform annotation on the divided abnormal driving data based on the abnormal type; if there is still unannotatable abnormal driving data, send the still unannotatable abnormal driving data to the annotation expert for the annotation expert to perform manual annotation on the unannotatable abnormal driving data.

[0177] It should be noted that in an implementable embodiment, the data extraction sub-module 741 and the data continuous extraction sub-module 743 may be the same module, configured to randomly extract a second preset number of unannotatable abnormal driving data from the unannotatable abnormal driving data.

[0178] Figure 10 is a schematic block diagram of a simulation abnormal driving data construction module shown in the present application. As Figure 10 shown, based on the above embodiments, the simulation abnormal driving data construction module 760 further includes: Figure 7 shown, based on the above embodiments, the simulation abnormal driving data construction module 760 further includes:

[0179] A metadata addition module sub 761, configured to add metadata to the abnormal driving data, where the metadata is characterized by a plurality of information factors and is used to describe the scenario information and vehicle information of the abnormal driving data.

[0180] The simulation data construction sub-module 762 is used to determine an information factor combination and construct simulated abnormal driving data based on the information factor combination for training an intelligent vehicle perception system.

[0181] Figure 11 It is a schematic block diagram of an abnormal driving data management sub-module shown in the present application. As Figure 11 shown, on the basis of the embodiment shown in Figure 10 , the simulated abnormal driving data construction module 760 further includes an abnormal driving data management sub-module 763 ( Figure 10 not shown in the figure):

[0182] An abnormal driving data summarization unit 7631 is used to summarize abnormal driving data having each information factor for each information factor.

[0183] An abnormal driving data storage unit 7632 is used to store the summarized abnormal driving data into a database corresponding to the information factor.

[0184] For the implementation processes of the functions and roles of each module in the above device, specifically refer to the implementation processes of the corresponding steps in the above method, which will not be elaborated here.

[0185] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0186] The specific embodiments of this specification are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0187] Other embodiments of the present specification will be readily contemplated by those skilled in the art after considering the specification and practicing the invention herein. This specification is intended to cover any variations, uses, or adaptations of the specification, which follow the general principles of the specification and include known common knowledge or conventional technical means in the technical field not claimed in this specification. The specification and examples are only illustrative, and the true scope and spirit of this specification are pointed out by the following claims.

[0188] It should be understood that this specification is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is only limited by the appended claims.

[0189] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. A method for abnormal driving data annotation, characterized in that, The method includes: Dividing the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on the abnormal driving data division rule; Randomly extracting a first preset number of normal driving data from the normal driving data and sending the extracted normal driving data to the inspection expert, where the first preset number is determined by the inspection expert based on the quantity of the divided normal driving data; Receiving a new abnormal driving data division rule and re-dividing the normal driving data based on the new abnormal driving data division rule; the new abnormal driving data division rule is extracted from the abnormal driving data detected by the inspection expert from the extracted normal driving data; Labeling the divided abnormal driving data based on a preset abnormal type; Sending the unlabelable abnormal driving data to the labeling expert for the labeling expert to perform manual labeling on the unlabelable abnormal driving data; the labeled abnormal driving data is used for the training of the intelligent vehicle perception system.

2. The method according to claim 1, wherein The step of randomly extracting a first preset number of normal driving data from the normal driving data and sending the extracted normal driving data to the inspection expert includes: Randomly extracting a first preset number of normal driving data from the normal driving data and sending the extracted normal driving data to the inspection expert for the inspection expert to perform abnormal judgment to identify the abnormal driving data not identified by the abnormal driving data division rule from the normal driving data; Judging whether the quantity of the normal driving data not randomly extracted is not less than a first threshold; the first threshold is not less than the first preset number; If the quantity of the normal driving data not randomly extracted is not less than the first threshold, continue to randomly extract a first preset number of normal driving data from the normal driving data not randomly extracted and send it to the inspection expert.

3. The method according to claim 2, wherein The method further includes: If the quantity of the normal driving data not randomly extracted is less than the first threshold, judging whether the quantity of the abnormal driving data not identified by the abnormal driving data division rule identified from the normal driving data is not less than a second threshold; When the quantity of the identified abnormal driving data is not less than the second threshold, returning a prompt for extracting a new abnormal driving data division rule to the inspection expert to prompt the inspection expert to extract a new abnormal driving data division rule.

4. The method according to claim 1, characterized in that The step of sending the unlabelable abnormal driving data to the labeling expert for the labeling expert to perform manual labeling on the unlabelable abnormal driving data includes: Randomly extracting a second preset number of unlabelable abnormal driving data from the unlabelable abnormal driving data and sending the extracted unlabelable abnormal driving data to the labeling expert for the labeling expert to perform manual labeling on the unlabelable abnormal driving data; Judging whether the quantity of the unlabelable abnormal driving data not randomly extracted is not less than a third threshold; the third threshold is not less than the second preset number; If the number of unlabeled abnormal driving data that have not been randomly selected is not less than the third threshold, then continue to randomly select a second preset number of abnormal driving data from the unlabeled abnormal driving data that have not been randomly selected and send them to the annotation expert.

5. The method according to claim 4, wherein The method further includes: If the number of unlabeled abnormal driving data that have not been randomly selected is less than the third threshold, then return a new abnormal type entry prompt to the annotation expert to prompt the annotation expert to determine and enter a new abnormal type based on the manual annotation of the unlabeled abnormal driving data; After adding a new abnormal type, annotate the divided abnormal driving data based on the abnormal type; if there is still unlabeled abnormal driving data, send the still unlabeled abnormal driving data to the annotation expert for the annotation expert to manually annotate the still unlabeled abnormal driving data.

6. The method according to claim 1, wherein The method further includes: Add metadata to the abnormal driving data, where the metadata is characterized by a number of information factors and is used to describe the scenario information and vehicle information of the abnormal driving data; Determine an information factor combination, and construct simulated abnormal driving data based on the information factor combination for training the intelligent vehicle perception system.

7. The method according to claim 6, wherein The method further includes: For each information factor, summarize the abnormal driving data having the information factor; Save the summarized abnormal driving data to a database corresponding to the information factor.

8. An apparatus for abnormal driving data annotation, characterized in that The device includes: A data division module, configured to divide the historical driving data of the intelligent vehicle into abnormal driving data and normal driving data based on an abnormal driving data division rule; A data extraction module, configured to randomly extract a first preset number of normal driving data from the normal driving data and send the extracted normal driving data to the inspection expert, where the first preset number is formulated by the inspection expert based on the number of the divided normal driving data; A rule receiving module, configured to receive a new abnormal driving data division rule and re-divide the normal driving data based on the new abnormal driving data division rule; the new abnormal driving data division rule is extracted from the abnormal driving data detected by the inspection expert from the extracted normal driving data; A data annotation module, configured to annotate the divided abnormal driving data based on a preset abnormal type; A data sending module, configured to send the unlabeled abnormal driving data to the annotation expert for the annotation expert to manually annotate the unlabeled abnormal driving data; the annotated abnormal driving data is used for training the intelligent vehicle perception system.

9. An electronic device, characterized in that, It includes a communication interface, a processor, a memory, and a bus, and the communication interface, the processor, and the memory are interconnected through the bus; Machine-readable instructions are stored in the memory, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images