Data labeling method, device, equipment and storage medium

Abstract

Disclosed are a data labeling method, device, equipment and storage medium. The method includes: determining a first time range in which a preset scene appears in the automatic driving data of a target vehicle; determining a first target timestamp of a target event in the preset scene based on driving information of the target vehicle within the first time range; obtaining target data corresponding to the first target timestamp from scene data corresponding to the preset scene, and determining first labeling information based on the target data; and labeling the scene data using the first labeling information.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data labeling method, device, equipment and storage medium. Background Art

[0002] Autonomous driving usually requires the use of a large amount of labeled data for simulation testing or field testing, such as scene labeled data, event labeled data, etc., to ensure the safety of autonomous driving. At present, driving data is usually labeled manually, which is not only inefficient, but also has a coarse granularity for labeling scenes and event classifications, which is not conducive to the precise driving control of autonomous driving vehicles based on scene classification. How to automatically and effectively classify and label autonomous driving data is a difficult problem that needs to be solved in the development of autonomous driving technology. Summary of the invention

[0003] The present disclosure proposes a data annotation solution.

[0004] According to one aspect of the present disclosure, a data labeling method is provided, the method comprising: determining a first time range in which a preset scene appears in the autonomous driving data of a target vehicle; determining a first target timestamp of a target event in the preset scene based on driving information of the target vehicle within the first time range; obtaining target data corresponding to the first target timestamp from scene data corresponding to the preset scene, and determining first labeling information based on the target data; and labeling the scene data using the first labeling information.

[0005] In combination with any embodiment provided in the present disclosure, the scene data includes first data for the target vehicle and second data for at least one target object within a set range from the target vehicle.

[0006] In combination with any embodiment provided in the present disclosure, the target data corresponding to the first target timestamp is obtained from the scene data corresponding to the preset scene, and the first annotation information is determined based on the target data, including: obtaining the first data and the second data corresponding to the first target timestamp in the scene data; determining the first driving status information of the target vehicle based on the first data, and determining the second driving status information of the target object based on the second data; and using the first driving status information and / or the second driving status information as the first annotation information.

[0007] In combination with any embodiment provided by the present disclosure, the target data corresponding to the first target timestamp is obtained from the scene data corresponding to the preset scene, and the first annotation information is determined based on the target data, including: determining the probability of collision between the target vehicle and each target object within a first set time range after the first target timestamp based on the first driving state information of the target vehicle and the second driving state information of each target object; determining the target object whose collision probability is greater than a set probability threshold as the first object; determining a first risk coefficient of the first object based on the probability of collision between the target vehicle and the first object, the first risk coefficient indicating the degree of danger of the first object to the target vehicle; determining a second risk coefficient of the target vehicle based on the probability of collision between the target vehicle and each first object and the number of the first objects, the second risk coefficient indicating the degree of danger of the target vehicle itself; using the first risk coefficient and / or the second risk coefficient as the first annotation information.

[0008] In combination with any embodiment provided in the present disclosure, the method also includes: obtaining standard data corresponding to the preset scene; obtaining second annotation information based on a comparison result of the scene data and the standard data, wherein the second annotation information indicates an abnormality type in the preset scene, and the abnormality type includes at least one of positioning abnormality, predicting abnormality, and controlling abnormality; and annotating the scene data using the second annotation information.

[0009] In combination with any implementation manner provided in the present disclosure, the method further includes: obtaining a plurality of tags set for the category according to the category of the preset scene; obtaining at least one target tag from the plurality of tags to label the scene data.

[0010] In combination with any embodiment provided in the present disclosure, the method further includes: in response to the occurrence of the preset scene, obtaining third annotation information according to the category of the preset scene and the real scene corresponding to the preset scene, wherein the third annotation information indicates the reason for the occurrence of the preset scene and / or the target object related to the preset scene; and annotating the scene data according to the third annotation information.

[0011] In combination with any embodiment provided by the present disclosure, the autonomous driving data includes multiple data frames; determining a first time range in which a preset scene appears in the autonomous driving data of a target vehicle, comprising: obtaining a first data sequence for the target vehicle from the multiple data frames contained in the autonomous driving data; for each target object, obtaining a second data sequence of the target object from the multiple data frames contained in the autonomous driving data; determining a target data frame that meets the preset scene standard based on the first data sequence and the second data sequence of each target object; marking the target data frame with a label corresponding to the preset scene; determining the first time range in which the preset scene appears in the autonomous driving data of the target vehicle, comprising: determining the first time range in which the preset scene appears in the autonomous driving data of the target vehicle based on the data frame marked with a label corresponding to the preset scene.

[0012] According to one aspect of the present disclosure, a data labeling device is provided, comprising: a first determination unit, configured to determine a first time range in which a preset scene appears in the automatic driving data of a target vehicle; a second determination unit, configured to determine a first target timestamp of a target event occurring in the preset scene based on driving information of the target vehicle within the first time range; a third determination unit, configured to obtain target data corresponding to the first target timestamp from scene data corresponding to the preset scene, and to determine first labeling information based on the target data; and a labeling unit, configured to label the scene data using the first labeling information.

[0013] According to one aspect of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the data labeling method described in any embodiment of the present disclosure when executing the computer instructions.

[0014] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data labeling method described in any embodiment of the present disclosure is implemented.

[0015] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the program is executed by a processor, the data labeling method described in any embodiment of the present disclosure is implemented.

[0016] The data labeling method, apparatus, device and storage medium of one or more embodiments of the present disclosure determine a first time range in which a preset scene appears in the autonomous driving data of a target vehicle, and determine a first target timestamp of a target event in the preset scene based on the driving information of the target vehicle within the first time range. Then, target data corresponding to the first target timestamp is obtained from the scene data corresponding to the preset scene, and first labeling information is determined based on the target data. The scene data is labeled using the first labeling information, thereby automatically labeling the scene data in the autonomous driving data in a fine-grained manner. On the one hand, it can realize operations such as automatic classification, retrieval, and association testing of the scene data, and on the other hand, it is beneficial for the autonomous driving vehicle to make precise driving control according to the scene classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A flowchart of an autonomous driving data labeling method provided by at least one embodiment of the present disclosure;

[0019] Figure 2 A flow chart of a method for acquiring first annotation data from the scene data proposed by at least one embodiment of the present disclosure;

[0020] Figure 3 A flowchart of another method for obtaining first annotation data from the scene data proposed by at least one embodiment of the present disclosure;

[0021] Figure 4 A flow chart of an offline evaluation method proposed for at least one embodiment of the present disclosure;

[0022] Figure 5 A flow chart of a potential danger scenario mining method proposed by at least one embodiment of the present disclosure;

[0023] Figure 6 A flow chart of a method for determining a dangerous scenario proposed in at least one embodiment of the present disclosure;

[0024] Figure 7 A schematic diagram showing a data annotation device proposed by at least one embodiment of the present disclosure;

[0025] Figure 8A schematic diagram of the structure of an electronic device proposed by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0027] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0029] The data labeling method provided by the embodiment of the present disclosure can be executed by a terminal device, a server or other types of electronic devices, wherein the terminal device can be a vehicle-mounted device installed in a vehicle; the server can be a local server or a cloud server.

[0030] Figure 1 is a flow chart of a data labeling method proposed by at least one embodiment of the present disclosure. Figure 1 As shown, the method includes steps 101-104.

[0031] In step 101, a first time range in which a preset scenario appears in the autonomous driving data of a target vehicle is determined.

[0032] In the disclosed embodiment, the target vehicle is a vehicle equipped with an automatic driving system to control the vehicle to drive automatically without human drivers. The automatic driving data of the target vehicle is data generated by the automatic driving system during actual automatic driving or automatic driving testing.

[0033] For the autonomous driving data, the autonomous driving data can be divided into scene data corresponding to at least one preset scene according to the standard of the preset scene set in advance. For example, for the scene of following a vehicle at an intersection, the corresponding scene standard is that at the intersection, there is at least one vehicle in front of the target vehicle, the vehicle is in the same lane as the target vehicle and both vehicles are not stationary; for the scene of free lane change, the corresponding scene standard is that the target vehicle goes straight, there is at least one vehicle in front of the target vehicle, the vehicle is stationary or has a lower speed than the target vehicle, and the lane ID of the target vehicle changes. Among them, the lane ID comes from the annotated high-precision map. In the high-precision map, in addition to the lane, the intersection, traffic lights, etc. have unique IDs. According to the vehicle status data of the target vehicle and the environmental perception data of the surrounding objects of the target vehicle, the target data that meets the preset scene standard can be determined from the autonomous driving data.

[0034] Since each data frame in the autonomous driving data is time-stamped, the first time range in which each preset scene occurs can be determined.

[0035] In step 102, a first target timestamp of a target event occurring in the preset scenario is determined according to the driving information of the target vehicle within the first time range.

[0036] In one example, a driving record within the first time range can be obtained, and the state information of the target vehicle can be obtained based on the automatic driving control information of the target vehicle in the driving record, that is, the state of the vehicle can be obtained, so that the first target timestamp of the target event occurring in the preset scene can be determined. The first target timestamp can be a timestamp corresponding to a data frame, or can include timestamps corresponding to multiple time frames.

[0037] For example, in the autonomous driving data, in the "takeover" scenario, there may be many data frames marked with the "takeover" tag. According to the vehicle state determined by rosbag, the precise time point of the "takeover event" can be known, and the timestamp of the data frame corresponding to the precise time point is the first target timestamp. In the embodiment of the present disclosure, the precise time point can usually reach the millisecond level.

[0038] In step 103, target data corresponding to the first target timestamp is obtained from the scene data corresponding to the preset scene, and first annotation information is determined according to the target data.

[0039] In the disclosed embodiment, the scene data corresponding to the preset scene is data in the autonomous driving data marked with a label corresponding to the preset scene. The scene data can be subjected to simulation testing or field testing. When running to the scene data marked with the preset scene, it is processed according to the process corresponding to the preset scene, that is, the control signal of the target vehicle is output according to the preset scene, and the control signal includes one or more of the following control signals: throttle, brake, gear, steering wheel angle, etc.

[0040] For example, if the scenario indicated by the preset scenario standard is "following a vehicle at an intersection", the target data that meets the preset scenario standard is labeled "following a vehicle at an intersection". In simulation and testing, when the data labeled "following a vehicle at an intersection" is encountered, the target vehicle is controlled to maintain a low speed and go straight to avoid collision with the vehicle in front at the intersection.

[0041] According to the target data corresponding to the first target timestamp in the scene data, information about the target vehicle and target objects around the target vehicle at the precise time when the target event occurs can be obtained. According to the type of data to be annotated, the first annotation information, such as the driving state information, risk factor, etc. of the target vehicle and / or target object, can be obtained from the scene data corresponding to the first target timestamp.

[0042] In step 104, the scene data is labeled using the first labeling information.

[0043] According to the precise time point when the target event (abnormality) occurs in the preset scene, further information about the target vehicle and / or the target object is obtained as the first annotation information, and the scene data is annotated, so that the classification of the scene data can be more refined and clear.

[0044] In an embodiment of the present disclosure, a first time range in which a preset scene appears in the autonomous driving data of a target vehicle is determined, and a first target timestamp of a target event occurring in the preset scene is determined based on the driving information of the target vehicle within the first time range. Then, based on the first target timestamp, target data corresponding to the first target timestamp is obtained from scene data corresponding to the preset scene, and first annotation information is determined based on the target data. The scene data is annotated using the first annotation information, thereby automatically annotating the scene data in the autonomous driving data in a fine-grained manner. On the one hand, it can realize operations such as automatic classification, retrieval, and association testing of scene data, and on the other hand, it is beneficial for the autonomous driving vehicle to make precise driving control according to scene classification in simulation and testing.

[0045] In some implementations, scene data can be divided according to the object from which the data originates, including first data of the target vehicle itself (the ego vehicle) and second data of at least one target object (surrounding object) within a set range of the target vehicle, wherein the surrounding objects include other vehicles, lane lines, objects in other traffic environments, and the like.

[0046] Among them, the first data of the target vehicle itself includes the vehicle status data of the target vehicle, such as the vehicle status data such as vehicle position, speed, driving direction, etc.; the second data of the target object is obtained from the environmental perception data detected by the sensor arranged in the vehicle, including one or more of the distance data, speed data, and status data of the target object, wherein the sensor is such as a camera, a lidar, a millimeter wave radar, etc.

[0047] Assuming that within the set range of the target vehicle, for example, 100 meters, there are a first vehicle, a second vehicle, and a traffic light, the second data of the first vehicle, the second vehicle, and the traffic light are respectively obtained from the scene data. The second data of the first vehicle and the second vehicle respectively include the distance from the target vehicle, the vehicle speed, etc., and the second data of the traffic light includes the distance from the target vehicle and the state of the light.

[0048] Figure 2 A flow chart of a method for acquiring first annotation data from the scene data proposed in at least one embodiment of the present disclosure.

[0049] In step 201, the first data and the second data corresponding to the first target timestamp in the scene data are obtained, that is, the data of the target vehicle and the target object at the precise time point when the abnormality occurs are obtained.

[0050] In step 202, the first driving state information of the target vehicle is determined according to the first data, and the second driving state information of the target object is determined according to the second data. The first driving state information of the target vehicle is, for example, at least one of the position, speed, and direction of the target vehicle, and the second driving state information of the target object within a set range from the target vehicle, i.e., the surrounding object, is determined, for example, at least one of the position, speed, and relative distance of the target object from the target vehicle.

[0051] In step 203, the first driving state information and / or the second driving state information is used as first annotation information.

[0052] In one example, for scene data marked with emergency braking, the speed of the vehicle at the precise time when the braking occurs and the distance to the nearest vehicle to the vehicle may be used as first marking information to further mark the scene data.

[0053] Figure 3 A flowchart of another method for obtaining first annotation data from the scene data proposed for at least one embodiment of the present disclosure.

[0054] In step 301, based on the first driving state information of the target vehicle and the second driving state information of each target object, the probability of the target vehicle colliding with each target object within a first set time range after the first target timestamp is determined.

[0055] In this step, the probability of collision between the vehicle and the surrounding objects can be determined by calculating the overlap between the vehicle and the objects on the same side within a first set time range after the first timestamp, for example, within 3 seconds. The overlap can be determined based on the position, speed, and direction of the vehicle, and the position and speed of the surrounding objects.

[0056] In step 302, a target object whose collision probability is greater than a set probability threshold is determined as a first object. The first object is a dangerous obstacle, that is, surrounding objects whose collision probability is higher than a certain standard are classified as dangerous obstacles.

[0057] In step 303, a first risk coefficient of the first object is determined according to the probability of the target vehicle colliding with the first object, where the first risk coefficient indicates the degree of danger of the first object to the target vehicle.

[0058] The higher the probability of the target vehicle colliding with the first object, the higher the probability that the first object will cause danger to the target vehicle, that is, the higher the danger level of the first vehicle.

[0059] In the case where the first object includes multiple target objects, the multiple target objects may be ranked according to the probability of collision between each target object and the target vehicle. The preset scene may be marked with the highest ranking one or more target objects.

[0060] In step 304, a second risk factor of the target vehicle is determined according to the probability of the target vehicle colliding with each first object and the number of the first objects, where the second risk factor indicates the risk level of the target vehicle itself.

[0061] For the target vehicle, the higher the probability that the surrounding vehicles will overlap with itself in the future, and the greater the number of vehicles that may overlap, the higher the possibility that the vehicle is in danger.

[0062] In step 305, the first risk factor and / or the second risk factor is used as the first labeling information.

[0063] In one example, for scene data marked with emergency braking, the scene data may be marked with the risk factor of the own vehicle and the highest risk factor of other vehicles at the precise time point when the braking occurs.

[0064] In the disclosed embodiment, by further annotating the scene data according to the second risk factor of the target vehicle and / or the first risk factor of the target object, it is easier to detect difficult scene data with reference value in the huge amount of data of autonomous driving, which can be better used for simulation and testing of automatic vehicle driving.

[0065] In some implementations, the autonomous driving data may be annotated in an offline state. For example, the data generated by the target vehicle during the actual autonomous driving process or the autonomous driving test process may be uploaded to the server in real time during the driving process, or may be first stored in the vehicle-mounted device and then uploaded to the server after the driving is completed. The autonomous driving data uploaded to the server may be annotated in an offline state.

[0066] In some implementations, autonomous driving data may be acquired in real time during an actual autonomous driving process or an autonomous driving test, and the autonomous driving data may be labeled online.

[0067] The disclosed embodiment also proposes a method for performing offline evaluation on autonomous driving data to further add evaluation labels to the scene data of a preset scene. Figure 4 The flowchart of the offline evaluation method is shown as follows: Figure 4 As shown, the method includes steps 401 to 403.

[0068] In step 401, standard data corresponding to the preset scenario is obtained. The standard data is, for example, simulation test data of the target vehicle. For example, for a takeover scenario, standard data of the takeover scenario in the simulation test data is obtained.

[0069] In step 402, second annotation information is obtained based on the comparison result between the scene data and the standard data, and the second annotation information indicates anomalies in the scene data, and the anomalies in the scene data include at least one of positioning anomalies, prediction anomalies, and control anomalies.

[0070] In step 403, the scene data is labeled using the obtained second labeling information.

[0071] By comparing the scene data with the standard data, if the scene data generated during driving deviates from the standard data and the deviation exceeds a set degree, it can be determined that there is an abnormality in the scene data. According to the type of data with the abnormality, second annotation information indicating the abnormality can be obtained. For example, if there is an abnormality in the positioning data, "positioning abnormality" can be used as the second annotation information to annotate the scene data.

[0072] In the disclosed embodiment, by adding evaluation labels to the scene data, the autonomous driving system is helped to correct erroneous operations that occur during the autonomous driving process.

[0073] The disclosed embodiment also proposes a method for further adding customized annotation labels or custom labels to scene data. Specifically, according to the category of the preset scene, multiple labels set for the category can be obtained; at least one target label is obtained from the multiple labels to label the scene data. Among them, the category of the preset scene can be determined according to various factors involved in autonomous driving, for example, it can include at least one of the following categories: weather, suspended matter, lighting, vehicle attributes, other vehicle attributes, traffic conditions, etc. For each category, multiple labels can be preset. For example, sunny, cloudy, rainy and other categories can be set for the weather. By adding more fine-grained annotation information to the scene data classification, it is beneficial for the autonomous driving vehicle to make precise driving control according to the scene classification.

[0074] The disclosed embodiment also proposes a method for online annotation. In response to the occurrence of a preset scene, third annotation information can be obtained according to the category of the preset scene and the real scene corresponding to the preset scene, and the third annotation information indicates the reason for the occurrence of the preset scene and / or the target object related to the preset scene; the scene data is further annotated according to the third annotation information.

[0075] For example, the third annotation information can be set to two categories, one category indicates the cause of the scene, including missed detection, false pick-up, jitter, trajectory, traffic light recognition error, scene change, etc.; the other category indicates the target object related to the preset scene, including pedestrians, two-wheeled vehicles, large vehicles, small vehicles, obstacles, out of perception range, etc. During the testing of the autonomous driving vehicle, when the preset scene appears, such as takeover, the third annotation information can be generated according to the actual scene of the takeover, including the cause of the takeover (for example, the driver takes over due to missed detection) and the target object (pedestrian) that caused the takeover, so as to annotate the scene data of the takeover.

[0076] In the embodiment of the present disclosure, by obtaining third annotation information based on the category of the preset scene and the real scene corresponding to the preset scene, the autonomous driving data corresponding to the preset scene is labeled, which can supplement the true value information that the autonomous driving system cannot obtain, and is conducive to improving the safety of autonomous driving.

[0077] The disclosed embodiment also proposes a method for mining data that meets a preset scenario, including: obtaining a first data sequence for the target vehicle from multiple data frames included in the autonomous driving data; for each target object, obtaining a second data sequence for the target object from the multiple data frames included in the autonomous driving data; determining a target data frame that meets the preset scenario standard based on the first data sequence and the second data sequence of each target object; labeling the target data frame with a label corresponding to the preset scenario; and determining a first time range in which the preset scenario appears in the autonomous driving data of the target vehicle based on the data frame labeled with the label corresponding to the preset scenario.

[0078] Each data frame in the autonomous driving data may include the vehicle state data of the target vehicle and the environmental perception data detected by the sensor set in the target vehicle. According to the multiple data frames included in the autonomous driving data, a first data sequence for the target vehicle can be obtained, that is, the first data sequence of the target vehicle can be obtained according to the vehicle state data of the target vehicle contained in each data frame; for each target object, a second data sequence of the target object can be obtained from the multiple data frames included in the autonomous driving data, that is, the second data sequence of the target object can be obtained from the environmental perception data contained in each data frame. Next, according to the first data sequence and the second data sequence of each target object, a target data frame that meets the preset scene standard is determined. That is, for each frame of the scene, the sequence of the self-vehicle and the surrounding objects is maintained. For each type of scene to be mined, the frame is judged whether it meets the standard according to the standard corresponding to the scene, and the data frame that meets the standard is determined as the target data frame. Finally, the target data frame is annotated according to the label corresponding to the preset scene, that is, the first time range in which the preset scene appears in the autonomous driving data of the target vehicle can be determined by the timestamp of the target data frame.

[0079] By establishing data sequences for the target vehicle and each object around the target vehicle respectively, the relative relationship between the target vehicle and the surrounding objects and the changing state of the relative relationship can be displayed, so that the target data frame that meets the preset scene standards can be accurately judged, thereby improving the efficiency and accuracy of the autonomous driving data standards.

[0080] In the disclosed embodiment, the data frames in the autonomous driving data have timestamps, so each frame in the first data sequence and the second data sequence obtained based on the multiple data frames contained in the autonomous driving data also has timestamps. The first data sequence and the second data sequence are aligned according to the timestamps, so as to more clearly show the relative relationship between the target vehicle and the surrounding objects and the changing state of the relative relationship. Among them, the relative relationship includes the relative position, distance, etc. of the target vehicle and the surrounding objects. In some implementations, target data that meets the standards of the preset scene can be determined based on the first data sequence and the second data sequence carrying the same timestamp.

[0081] Figure 5 A flow chart of a potential danger scene mining method proposed in at least one embodiment of the present disclosure is shown, wherein the method is used to analyze existing autonomous driving data and mark the data corresponding to the potential danger scenes contained therein. In the method, the multiple data frames contained in the autonomous driving data are divided into multiple periods, and each period contains multiple data frames. The number of periods contained in the autonomous driving data and the number of data frames contained in each period are related to the total number of data frames contained in the autonomous driving data. Figure 5 As shown, the method includes steps 501 to 503.

[0082] In step 501, the probability of the target vehicle colliding with each target object is determined based on the first data sequence in at least one cycle and the second data sequence of each target object.

[0083] In this step, the probability of collision between the vehicle and the surrounding objects is determined by calculating the overlap between the vehicle and the surrounding objects within a certain period, such as within each period or within every two periods. The overlap can be determined based on the position, speed, and direction of the vehicle, and the position and speed of the surrounding objects.

[0084] In step 502, a target object whose collision probability is greater than a first set probability threshold is determined as a first object. The first object is a dangerous obstacle, that is, surrounding objects whose collision probability is higher than a certain standard are classified as dangerous obstacles.

[0085] In step 503, in response to the number of the first objects being greater than a first set number within the at least one period, the data frames within the at least one period are determined to be target data frames.

[0086] In step 504, the target data frame is labeled with a label corresponding to the potential danger scene. That is, the N data frames in the at least one cycle are labeled with the label corresponding to the potential danger scene.

[0087] If the number of dangerous obstacles is greater than the first set number, it indicates that the scene may be a potential dangerous scene. In this case, if normal driving is still maintained, the possibility of a collision is high, so when a potential dangerous scene is detected, the process corresponding to the dangerous scene can be called for processing, that is, the control signals of the throttle, brake, gear, steering wheel angle, etc. of the target vehicle in the dangerous scene are output.

[0088] In the disclosed embodiment, by mining the data of potential dangerous scenes in the driving data, potential dangerous scenes can be discovered in a timely manner during the autonomous driving process, and response plans for the dangerous scenes can be taken in a timely manner to ensure the safety of the vehicle's autonomous driving.

[0089] Figure 6 A flow chart of a method for determining a dangerous scene proposed by at least one embodiment of the present disclosure is shown, wherein the method is used to further analyze the mined potential dangerous scene to determine that the potential dangerous scene is indeed a dangerous scene. Figure 6 As shown, the method includes steps 601-605.

[0090] In step 601, first driving state information of the target vehicle and second driving state information of each target object are determined based on a data frame marked with a label corresponding to a potential danger scene.

[0091] For data frames marked as potentially dangerous scenes, the first driving state information of the target vehicle, such as at least one of the position, speed, direction, etc. of the target vehicle, and the second driving state information of the target object within a set range from the target vehicle, that is, the surrounding objects, such as at least one of the position, speed, etc. of the target object, can be determined from the vehicle state information and environmental perception information contained in the data frames.

[0092] In one example, the position, speed, direction, etc. of the target vehicle in the ENU coordinate system (Local Cartesian Coordinats coordinate system, station center coordinate system or station coordinate system) can be obtained based on the data frames in the at least one period marked with tags corresponding to the potential danger scene. Since the ENU coordinate system is used to understand the motion law of other objects centered on the observer, the relative relationship between the target vehicle and the target object can be determined by obtaining the speed and position of the target object in the coordinate system of the target vehicle.

[0093] In step 602, for each frame in the data frame marked with a label corresponding to a potential dangerous scene, the size of the target vehicle and each target object in the frame is expanded at a set ratio.

[0094] For example, the shapes of the target vehicle and each target object can be simplified to a rectangle, and the rectangular area corresponding to the target vehicle and the rectangular area corresponding to each target object can be expanded by a set ratio, for example, by 1.2 times. Those skilled in the art should understand that the set ratio can be set according to actual conditions, and the present disclosure does not limit the specific value of the set ratio.

[0095] In step 603, the overlap degree between the expanded target vehicle and each expanded target object is determined according to the first driving state information of the target vehicle and the second driving state information of each target object.

[0096] For example, based on the position information of the target vehicle and the position information of any one of the target objects, the relative position relationship between the two can be determined, so that the overlap degree between the expanded target vehicle and the expanded target object can be determined.

[0097] In step 604, the target object whose overlap degree is greater than a second set threshold is determined to be the second object.

[0098] In response to the fact that the overlap between the expanded target vehicle and any one of the target objects in one of the data frames marked with a label corresponding to the potential dangerous scene is greater than the second set threshold, the target object is determined as a second object. The second object is a dangerous obstacle, that is, surrounding objects with a probability of collision higher than a certain standard are listed as dangerous obstacles.

[0099] The second object determined in this step may be one target object, or may include two or more target objects. In the case where the second object includes two or more target objects, subsequent judgments are performed on the target objects included in the second object respectively.

[0100] In one example, for the determined second object, the second data frame sequence of the second object can be maintained. Furthermore, a maximum number of frames can be set for the maintained second data sequence of the second object. When the number of frames of the sequence reaches the maximum number of frames, each time a new frame is added, the earliest frame is deleted to keep the number of frames of the second data sequence of the second object not exceeding the maximum number of frames.

[0101] In step 605, in response to the continuous sequence formed by the continuously collected data frames in the data frames in which the overlap degree between the second object and the target vehicle is greater than the second set threshold value meets the set conditions, the label of the data frame containing the continuous sequence and marked with a label corresponding to the potential dangerous scene is modified to a label corresponding to the dangerous scene.

[0102] In one example, the set conditions satisfied by the continuous sequence include: the duration of the continuous sequence is greater than the first time threshold; and / or the target count value is greater than a first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than the second time threshold for a number of times greater than a second set number, and the initial value of the target count value is 0, wherein the first time threshold is greater than the second time threshold. That is, when the duration of a dangerous obstacle is long, or the duration of a dangerous obstacle is long for too many times, the potential dangerous scene is determined as a dangerous scene.

[0103] For example, when the duration of a continuous sequence is greater than the first time threshold (for example, 1s), the label corresponding to the potential dangerous scene is modified to the label corresponding to the dangerous scene; for another example, if the duration of a continuous sequence is greater than the second time threshold (for example, 0.2s) and the number of times is greater than a second set number (for example, 2 times), the target count value is increased by 1, and when the target count value is greater than the first set number (for example, 2 times), it is determined to be a dangerous scene.

[0104] In the disclosed embodiments, by analyzing potential dangerous scenarios and further determining them as dangerous scenarios, dangerous scenarios can be detected more accurately during the autonomous driving process, and response plans for dangerous scenarios can be taken in a timely manner to ensure the safety of the vehicle's autonomous driving.

[0105] In some implementations, continuously collected data frames marked with labels corresponding to dangerous scenes constitute a dangerous scene. The dangerous scenes in the autonomous driving data can be sorted according to the timestamps carried by the data frames in each dangerous scene. In response to the time interval between two adjacent dangerous scenes of the same type being less than a third time threshold, the labels of the data frames constituting the two adjacent dangerous scenes are modified to labels corresponding to the same dangerous scene, that is, the autonomous driving data corresponding to the adjacent dangerous scenes are merged. Among them, the time interval between adjacent dangerous scenes can be determined based on the time interval between the timestamp of the last frame of the previous dangerous scene and the timestamp of the first frame of the next dangerous scene, or it can be determined in other ways. Through the above method, dangerous scenes can be processed continuously and stably.

[0106] Figure 7 A schematic diagram of a data annotation device proposed in at least one embodiment of the present disclosure is shown. Figure 7 As shown, the device includes: a first determination unit 701, used to determine a first time range in which a preset scene appears in the automatic driving data of the target vehicle; a second determination unit 702, used to determine a first target timestamp of a target event in the preset scene based on the driving information of the target vehicle within the first time range; a third determination unit 703, used to obtain target data corresponding to the first target timestamp from the scene data corresponding to the preset scene, and determine first annotation information based on the target data; an annotation unit 704, used to annotate the scene data using the first annotation information.

[0107] In some implementations, the scene data includes first data for the target vehicle and second data for at least one target object within a set range from the target vehicle.

[0108] In some implementations, the third determination unit is specifically used to: obtain the first data and the second data corresponding to the first target timestamp in the scene data; determine the first driving state information of the target vehicle based on the first data, and determine the second driving state information of the target object based on the second data; and use the first driving state information and / or the second driving state information as the first annotation information.

[0109] In some implementations, the third determination unit is specifically used to: determine the probability of collision between the target vehicle and each target object within a first set time range after the first target timestamp based on the first driving state information of the target vehicle and the second driving state information of each target object; determine the target object whose collision probability is greater than a set probability threshold as the first object; determine a first risk coefficient of the first object based on the probability of collision between the target vehicle and the first object, the first risk coefficient indicating the degree of danger of the first object to the target vehicle; determine a second risk coefficient of the target vehicle based on the probability of collision between the target vehicle and each first object and the number of the first objects, the second risk coefficient indicating the degree of danger of the target vehicle itself; use the first risk coefficient and / or the second risk coefficient as the first annotation information.

[0110] In some implementations, the device also includes an anomaly labeling unit, which is used to: obtain standard data corresponding to the preset scene; obtain second labeling information based on a comparison result of the scene data and the standard data, wherein the second labeling information indicates an anomaly type in the preset scene, and the anomaly type includes at least one of positioning anomaly, predicting anomaly, and controlling anomaly; and label the scene data using the second labeling information.

[0111] In some implementations, the device further includes a category labeling unit, which is used to: obtain a plurality of labels set for the category according to the category of the preset scene; obtain at least one target label from the plurality of labels, and label the scene data.

[0112] In some implementations, the device also includes a real scene labeling unit, which is used to: in response to the occurrence of the preset scene, obtain third labeling information according to the category of the preset scene and the real scene corresponding to the preset scene, wherein the third labeling information indicates the reason for the occurrence of the preset scene and / or the target object related to the preset scene; and label the scene data according to the third labeling information.

[0113] In some implementations, the autonomous driving data includes multiple data frames; when the first determination unit is used to determine a first time range in which a preset scene appears in the autonomous driving data of a target vehicle, it is specifically used to: obtain a first data sequence for the target vehicle from multiple data frames contained in the autonomous driving data; for each target object, obtain a second data sequence of the target object from multiple data frames contained in the autonomous driving data; determine a target data frame that meets the preset scene standard based on the first data sequence and the second data sequence of each target object; label the target data frame with a label corresponding to the preset scene; and determine the first time range in which the preset scene appears in the autonomous driving data of the target vehicle based on the data frame labeled with the label corresponding to the preset scene.

[0114] At least one embodiment of the present disclosure further provides an electronic device, such as Figure 8 As shown, the device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method described in any embodiment of the present disclosure when executing the computer instructions.

[0115] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0116] At least one embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the method described in any embodiment of the present disclosure when the program is executed by a processor.

[0117] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0119] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0120] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0121] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0122] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0123] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0124] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0125] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0126] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data labeling method, characterized in that: The method comprises: Determine a first time range in which a preset scenario appears in the autonomous driving data of the target vehicle; the autonomous driving data is data generated by the autonomous driving system set by the target vehicle during an actual autonomous driving process or an autonomous driving test process; Determining a first target timestamp of a target event occurring in the preset scenario according to the driving information of the target vehicle within the first time range; Obtaining target data corresponding to the first target timestamp from the scene data corresponding to the preset scene, and determining first annotation information according to the target data; wherein the scene data includes data in the autonomous driving data that meets the pre-set standard of the preset scene; the target data corresponding to the first target timestamp is used to represent information about the target vehicle and target objects around the target vehicle within the first target timestamp; The scene data is labeled using the first labeling information.

2. The method according to claim 1, characterized in that The scene data includes first data for the target vehicle and second data for at least one target object within a set range from the target vehicle.

3. The method according to claim 2, characterized in that The obtaining, from the scene data corresponding to the preset scene, target data corresponding to the first target timestamp, and determining first annotation information according to the target data, includes: Acquire first data and second data corresponding to the first target timestamp in the scene data; Determine first driving state information of the target vehicle according to the first data, and determine second driving state information of the target object according to the second data; The first driving state information and / or the second driving state information is used as first marking information.

4. The method according to claim 2, characterized in that: The obtaining, from the scene data corresponding to the preset scene, target data corresponding to the first target timestamp, and determining to obtain first annotation information according to the target data, includes: Determining, according to the first driving state information of the target vehicle and the second driving state information of each target object, a probability of collision between the target vehicle and each target object within a first set time range after the first target timestamp; Determine a target object whose collision probability is greater than a set probability threshold as a first object; Determining a first risk factor of the first object according to a probability of a collision between the target vehicle and the first object, the first risk factor indicating a degree of danger of the first object to the target vehicle; Determining a second risk factor of the target vehicle according to the probability of the target vehicle colliding with each first object and the number of the first objects, the second risk factor indicating the risk level of the target vehicle itself; The first risk factor and / or the second risk factor is used as the first labeling information.

5. The method according to claim 1, characterized in that The method further comprises: Acquire standard data corresponding to the preset scene; According to the comparison result of the scene data and the standard data, second annotation information is obtained, where the second annotation information indicates an abnormality type in the preset scene, and the abnormality type includes at least one of positioning abnormality, predicting abnormality, and controlling abnormality; The scene data is labeled using the second labeling information.

6. The method according to claim 1, characterized in that The method further comprises: According to the category of the preset scene, a plurality of tags set for the category are acquired; At least one target tag is obtained from the multiple tags to label the scene data.

7. The method according to claim 1, characterized in that The method further comprises: In response to the occurrence of the preset scene, obtaining third annotation information according to the category of the preset scene and the real scene corresponding to the preset scene, wherein the third annotation information is used to indicate the reason for the occurrence of the preset scene and / or the target object related to the preset scene; The scene data is labeled according to the third labeling information.

8. The method according to claim 1, characterized in that The autonomous driving data includes a plurality of data frames; Determine the first time range in which the preset scenario appears in the autonomous driving data of the target vehicle, including: Obtaining a first data sequence for the target vehicle from a plurality of data frames included in the autonomous driving data; For each target object, obtaining a second data sequence of the target object from a plurality of data frames included in the autonomous driving data; Determining a target data frame that meets a preset scene standard according to the first data sequence and the second data sequence of each target object; Marking the target data frame with a label corresponding to the preset scene; According to the data frame marked with a label corresponding to the preset scene, a first time range in which the preset scene appears in the automatic driving data of the target vehicle is determined.

9. A data labeling device, characterized in that: The device comprises: A first determining unit is used to determine a first time range in which a preset scene appears in the autonomous driving data of the target vehicle; the autonomous driving data is data generated by the autonomous driving system set by the target vehicle during an actual autonomous driving process or an autonomous driving test process; A second determining unit, configured to determine a first target timestamp of a target event occurring in the preset scenario according to the driving information of the target vehicle within the first time range; a third determination unit, configured to obtain target data corresponding to the first target timestamp from the scene data corresponding to the preset scene, and determine first annotation information according to the target data; wherein the scene data includes data in the autonomous driving data that meets the pre-set standard of the preset scene; and the target data corresponding to the first target timestamp is used to represent information about the target vehicle and target objects around the target vehicle within the first target timestamp; A labeling unit is used to label the scene data using the first labeling information.

10. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 8 when executing the computer instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images