Data labeling method, device, equipment and storage medium

Abstract

Disclosed are a data labeling method, apparatus, device and storage medium, the method comprising: acquiring autonomous driving data of a target vehicle; obtaining first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data; determining target data that meets the standards of a preset scenario based on the first data and the second data; and labeling the target data with a label corresponding to the preset scenario.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data labeling method, device, equipment and storage medium. Background Art

[0002] Autonomous driving usually requires a large amount of labeled data for simulation testing or field testing, such as scene labeled data, event labeled data, etc., to ensure the safety of autonomous driving. Currently, driving data is usually labeled manually, which is not only inefficient, but also needs to be done during the autonomous driving process. How to automatically and effectively label autonomous driving data and mine scenes is a difficult problem that needs to be solved in the development of autonomous driving technology. Summary of the invention

[0003] The present invention discloses a data annotation solution.

[0004] According to one aspect of the present disclosure, a data labeling method is provided, the method comprising: acquiring autonomous driving data of a target vehicle; obtaining first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data; determining target data that meets the standards of a preset scenario based on the first data and the second data; and labeling the target data with a label corresponding to the preset scenario.

[0005] In combination with any embodiment provided by the present disclosure, the autonomous driving data includes multiple data frames, and obtaining first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data includes: obtaining a first data sequence for the target vehicle from the multiple data frames contained in the autonomous driving data; for each target object, obtaining a second data sequence for the target object from the multiple data frames contained in the autonomous driving data; and determining target data that meets preset scenario standards from the autonomous driving data based on the first data and the second data, includes: determining target data frames that meet preset scenario standards based on the first data sequence and the second data sequences of each target object.

[0006] In combination with any embodiment provided by the present disclosure, the data frames in the autonomous driving data carry a timestamp; determining the target data frames that meet the preset scene standards based on the first data sequence and the second data sequence of each target object includes: determining the target data that meets the standards of the preset scene based on the first data sequence and the second data sequence carrying the same timestamp.

[0007] In combination with any embodiment provided by the present disclosure, the multiple data frames contained in the autonomous driving data are divided into multiple periods, each period containing multiple data frames; the target data frames that meet the preset scene standards are determined based on the first data sequence and the second data sequence of each target object, including: determining the probability of the target vehicle colliding with each target object based on the first data sequence and the second data sequence of each target object in at least one period; determining the target object whose probability of collision is greater than a first set probability threshold as the first object; in response to the number of the first objects being greater than a first set number in the at least one period, determining the data frames in the at least one period as the target data frames; labeling the target data with a label corresponding to the preset scene, including: labeling the target data frame with a label corresponding to the potential dangerous scene.

[0008] In combination with any embodiment provided by the present disclosure, the method also includes: determining the first driving state information of the target vehicle and the second driving state information of each target object based on a data frame marked with a label corresponding to a potential dangerous scene; for each frame in the data frame marked with a label corresponding to the potential dangerous scene, expanding the size of the target vehicle and each target object in the frame at a set ratio; determining the overlap between the expanded target vehicle and each expanded target object based on the first driving state information of the target vehicle and the second driving state information of each target object; determining the target object whose overlap is greater than a second set threshold as the second object; in response to the continuous sequence formed by the continuously collected data frames in the data frame whose overlap with the target vehicle is greater than the second set threshold meeting the set condition, modifying the label of the data frame containing the continuous sequence and marked with the label corresponding to the potential dangerous scene to the label corresponding to the dangerous scene.

[0009] In combination with any embodiment provided by the present disclosure, the continuous sequence satisfies the set conditions, including: the duration of the continuous sequence is greater than a first time threshold; and / or the target count value is greater than a first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than a second time threshold and the number of times is greater than a second set number, and the initial value of the target count value is 0; wherein the first time threshold is greater than the second time threshold.

[0010] In combination with any embodiment provided in the present disclosure, continuously collected data frames marked with labels corresponding to dangerous scenes constitute a dangerous scene, and the method also includes: in response to the time interval between two adjacent dangerous scenes being less than a third time threshold, modifying the labels of the data frames constituting the two adjacent dangerous scenes to labels corresponding to the same dangerous scene.

[0011] In combination with any embodiment provided in the present disclosure, the method further includes: in response to the occurrence of a preset event, marking the autonomous driving data within a set time range when the preset event occurs with a label corresponding to the preset event.

[0012] According to one aspect of the present disclosure, a data labeling device is provided, comprising: a first acquisition unit, configured to acquire autonomous driving data of a target vehicle; a second acquisition unit, configured to obtain first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data; a determination unit, configured to determine target data that meets the criteria of a preset scenario based on the first data and the second data; and a first labeling unit, configured to label the target data with a label corresponding to the preset scenario.

[0013] In combination with any embodiment provided by the present disclosure, the autonomous driving data includes multiple data frames, and the second acquisition unit is specifically used to: obtain a first data sequence for the target vehicle from the multiple data frames contained in the autonomous driving data; for each target object, obtain a second data sequence of the target object from the multiple data frames contained in the autonomous driving data; and determine the target data that meets the preset scene standards from the autonomous driving data based on the first data and the second data, including: determining the target data frame that meets the preset scene standards based on the first data sequence and the second data sequence of each target object.

[0014] In combination with any embodiment provided in the present disclosure, the data frame in the autonomous driving data carries a timestamp; the determination unit is specifically used to determine target data that meets the standards of a preset scenario based on a first data sequence and a second data sequence carrying the same timestamp.

[0015] In combination with any embodiment provided in the present disclosure, the multiple data frames contained in the autonomous driving data are divided into multiple cycles, each cycle containing multiple data frames; the determination unit is specifically used to: determine the probability of the target vehicle colliding with each target object based on a first data sequence in at least one cycle and a second data sequence of each target object; determine the target object whose probability of collision is greater than a first set probability threshold as the first object; in response to the number of the first objects being greater than a first set number in the at least one cycle, determine the data frame in the at least one cycle as the target data frame; labeling the target data with a label corresponding to the preset scene, including: labeling the target data frame with a label corresponding to the potential dangerous scene.

[0016] In combination with any embodiment provided by the present disclosure, the device also includes a dangerous scene labeling unit, which is used to: determine the first driving state information of the target vehicle and the second driving state information of each target object based on the data frame labeled with the label corresponding to the potential dangerous scene; for each frame in the data frame labeled with the label corresponding to the potential dangerous scene, expand the size of the target vehicle and each target object in the frame at a set ratio; determine the overlap between the expanded target vehicle and each expanded target object based on the first driving state information of the target vehicle and the second driving state information of each target object; determine the target object whose overlap is greater than a second set threshold as the second object; in response to the continuous sequence formed by the continuously collected data frames in the data frame whose overlap with the target vehicle is greater than the second set threshold meeting the set condition, modify the label of the data frame containing the continuous sequence and labeled with the label corresponding to the potential dangerous scene to the label corresponding to the dangerous scene.

[0017] In combination with any embodiment provided by the present disclosure, the continuous sequence satisfies the set conditions, including: the duration of the continuous sequence is greater than a first time threshold; and / or the target count value is greater than a first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than a second time threshold and the number of times is greater than a second set number, and the initial value of the target count value is 0; wherein the first time threshold is greater than the second time threshold.

[0018] In combination with any embodiment provided in the present disclosure, continuously collected data frames marked with labels corresponding to dangerous scenes constitute a dangerous scene, and the device also includes a merging unit for modifying the labels of the data frames constituting two adjacent dangerous scenes to labels corresponding to the same dangerous scene in response to the time interval between two adjacent dangerous scenes being less than a third time threshold.

[0019] In combination with any embodiment provided in the present disclosure, the device also includes a second labeling unit, which is used to: in response to the occurrence of a preset event, label the autonomous driving data within a set time range when the preset event occurs with a label corresponding to the preset event.

[0020] According to one aspect of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the data labeling method described in any embodiment of the present disclosure when executing the computer instructions.

[0021] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data labeling method described in any embodiment of the present disclosure is implemented.

[0022] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the program is executed by a processor, the data labeling method described in any embodiment of the present disclosure is implemented.

[0023] The autonomous driving device, method, vehicle and storage medium of one or more embodiments of the present disclosure perform labeling processing on autonomous driving data generated by a target vehicle during autonomous driving, obtain first data of the target vehicle itself and second data of at least one target object within a set range of the target vehicle from the autonomous driving data, determine target data that meets the standards of a preset scenario from the autonomous driving data based on the first data and the second data, and label the target data with a label corresponding to the preset scenario, so that data that meets the required scenario can be mined and labeled in the existing autonomous driving data, which improves the labeling efficiency on the one hand and can provide more autonomous driving data labeled with scenarios for autonomous driving simulation tests or field tests on the other. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A flowchart of a data annotation method provided for at least one embodiment of the present disclosure;

[0026] Figure 2 A flowchart of a potential danger scenario mining method proposed by at least one embodiment of the present disclosure is shown;

[0027] Figure 3 A flow chart showing a method for determining a dangerous scenario proposed by at least one embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram showing a data annotation device proposed by at least one embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram of the structure of an electronic device proposed by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0033] The data labeling method provided by the embodiment of the present disclosure can be executed by a terminal device, a server or other types of electronic devices, wherein the terminal device can be a vehicle-mounted device installed in a vehicle; the server can be a local server or a cloud server.

[0034] Figure 1 is a flow chart of a data labeling method proposed by at least one embodiment of the present disclosure. Figure 1 As shown, the method includes steps 101-104.

[0035] In step 101, the automatic driving data of the target vehicle is obtained.

[0036] In the disclosed embodiment, the target vehicle is a vehicle equipped with an automatic driving system to control the vehicle to drive automatically without human drivers. The automatic driving data of the target vehicle is data generated by the automatic driving system during actual automatic driving or automatic driving testing.

[0037] In step 102, first data for the target vehicle and second data for at least one target object within a set range from the target vehicle are obtained from the autonomous driving data.

[0038] For the autonomous driving data, the data is divided according to the object from which the data originates, including first data of the target vehicle itself (self-vehicle) and second data of at least one target object (surrounding object) within a set range of the target vehicle.

[0039] Among them, the first data of the target vehicle itself includes the vehicle status data of the target vehicle, such as the vehicle status data such as vehicle position, speed, driving direction, etc.; the second data of the target object is obtained from the environmental perception data detected by the sensor arranged in the vehicle, including one or more of the distance data, speed data, and status data of the target object, wherein the sensor is such as a camera, a lidar, a millimeter wave radar, etc.

[0040] Assuming that within the set range of the target vehicle, for example, 100 meters, there are a first vehicle, a second vehicle, and a traffic light, the second data of the first vehicle, the second vehicle, and the traffic light are respectively obtained from the automatic driving data. The second data of the first vehicle and the second vehicle respectively include the distance from the target vehicle, the vehicle speed, etc., and the second data of the traffic light includes the distance from the target vehicle and the state of the light.

[0041] In step 103, target data meeting the standards of a preset scenario is determined from the autonomous driving data based on the first data and the second data.

[0042] For the scene to be mined from the autonomous driving data, the standard of the scene can be pre-set. For example, for the scene of following a vehicle at an intersection, the corresponding scene standard is that at the intersection, there is at least one vehicle in front of the target vehicle, the vehicle is in the same lane as the target vehicle, and both vehicles are not stationary; for the scene of free lane change, the corresponding scene standard is that the target vehicle goes straight, there is at least one vehicle in front of the target vehicle, the vehicle is stationary or has a lower speed than the target vehicle, and the lane ID of the target vehicle changes. Among them, the lane ID comes from the annotated high-precision map. In the high-precision map, in addition to the lane, intersections, traffic lights, etc. have unique IDs.

[0043] According to the vehicle status data of the target vehicle and the environmental perception data of the surrounding objects of the target vehicle, the target data that meets the preset scenario standard can be determined from the automatic driving data.

[0044] In step 104, the target data is labeled with a label corresponding to the preset scenario.

[0045] The autonomous driving data marked with labels corresponding to the preset scenarios can be used for simulation tests or field tests. When the data marked with labels corresponding to the preset scenarios are run, they are processed according to the process corresponding to the preset scenarios, that is, the control signals of the target vehicle are output according to the preset scenarios, and the control signals include one or more of the following control signals: throttle, brake, gear, steering wheel angle, etc.

[0046] For example, if the preset scenario indicates "following a vehicle at an intersection", the target data that meets the standard of the preset scenario is labeled with a "following a vehicle at an intersection" label. When the target vehicle is running to the data labeled with the "following a vehicle at an intersection" label, the target vehicle is controlled to maintain a low speed and go straight to avoid collision with the vehicle in front at the intersection.

[0047] In an embodiment of the present disclosure, first data of the target vehicle itself and second data of at least one target object within a set range of the target vehicle are obtained from the autonomous driving data. Based on the first data and the second data, target data that meets the standards of a preset scenario is determined from the autonomous driving data, and a label corresponding to the preset scenario is annotated for the target data. This allows data that meets the required scenario to be mined from the existing autonomous driving data and annotated. This improves the efficiency of annotation on the one hand, and can provide more autonomous driving data annotated with scenarios on the other hand for use in autonomous driving simulation tests or field tests.

[0048] In some implementations, the autonomous driving data may be annotated in an offline state. For example, the data generated by the target vehicle during the actual autonomous driving process or the autonomous driving test process may be uploaded to the server in real time during the driving process, or may be first stored in the vehicle-mounted device and then uploaded to the server after the driving is completed. The autonomous driving data uploaded to the server may be annotated in an offline state.

[0049] In some implementations, autonomous driving data may be acquired in real time during an actual autonomous driving process or an autonomous driving test, and the autonomous driving data may be labeled online.

[0050] In some implementations, the autonomous driving data includes multiple data frames, each of which has a timestamp, and the multiple data frames can be stored in the form of logs, for example. In an offline state, the log of the autonomous driving data can be played back in time sequence to annotate, analyze or otherwise process the autonomous driving data.

[0051] Each data frame in the autonomous driving data may include the vehicle state data of the target vehicle and the environmental perception data detected by the sensor set in the target vehicle. According to the multiple data frames included in the autonomous driving data, a first data sequence for the target vehicle can be obtained, that is, the first data sequence of the target vehicle can be obtained according to the vehicle state data of the target vehicle contained in each data frame; for each target object, the second data sequence of the target object can be obtained from the multiple data frames included in the autonomous driving data, that is, the second data sequence of the target object can be obtained from the environmental perception data contained in each data frame. Next, according to the first data sequence and the second data sequence of each target object, a target data frame that meets the preset scene standard is determined. That is, for each frame scene, the sequence of the self-vehicle and the surrounding objects is maintained. For each type of scene to be mined, the frame is judged whether it meets the standard according to the standard corresponding to the scene, and the data frame that meets the standard is determined as the target data frame. Finally, the target data frame is labeled according to the label corresponding to the preset scene.

[0052] By establishing data sequences for the target vehicle and each object around the target vehicle respectively, the relative relationship between the target vehicle and the surrounding objects and the changing state of the relative relationship can be displayed, so that the target data frame that meets the preset scene standards can be accurately judged, thereby improving the efficiency and accuracy of the autonomous driving data standards.

[0053] In the disclosed embodiment, the data frames in the autonomous driving data have timestamps, so each frame in the first data sequence and the second data sequence obtained based on the multiple data frames contained in the autonomous driving data also has timestamps. The first data sequence and the second data sequence are aligned according to the timestamps, so as to more clearly show the relative relationship between the target vehicle and the surrounding objects and the changing state of the relative relationship. Among them, the relative relationship includes the relative position, distance, etc. of the target vehicle and the surrounding objects. In some implementations, target data that meets the standards of the preset scene can be determined based on the first data sequence and the second data sequence carrying the same timestamp.

[0054] Figure 2 A flow chart of a potential danger scene mining method proposed in at least one embodiment of the present disclosure is shown, wherein the method is used to analyze existing autonomous driving data and mark the data corresponding to the potential danger scenes contained therein. In the method, the multiple data frames contained in the autonomous driving data are divided into multiple periods, and each period contains multiple data frames. The number of periods contained in the autonomous driving data and the number of data frames contained in each period are related to the total number of data frames contained in the autonomous driving data. Figure 2As shown, the method includes steps 201 to 203.

[0055] In step 201 , the probability of the target vehicle colliding with each target object is determined based on a first data sequence in at least one cycle and a second data sequence of each target object.

[0056] In this step, the probability of collision between the vehicle and the surrounding objects is determined by calculating the overlap between the vehicle and the surrounding objects within a certain period, such as within each period or within every two periods. The overlap can be determined based on the position, speed, and direction of the vehicle, and the position and speed of the surrounding objects.

[0057] In step 202, a target object whose collision probability is greater than a first set probability threshold is determined as a first object. The first object is a dangerous obstacle, that is, surrounding objects whose collision probability is higher than a certain standard are classified as dangerous obstacles.

[0058] In step 203, in response to the number of the first objects being greater than a first set number within the at least one period, the data frames within the at least one period are determined to be target data frames.

[0059] In step 204, the target data frame is labeled with a label corresponding to the potential danger scene. That is, the N data frames in the at least one cycle are labeled with the label corresponding to the potential danger scene.

[0060] If the number of dangerous obstacles is greater than the first set number, it indicates that the scene may be a potential dangerous scene. In this case, if normal driving is still maintained, the possibility of a collision is high, so when a potential dangerous scene is detected, the process corresponding to the dangerous scene can be called for processing, that is, the control signals of the throttle, brake, gear, steering wheel angle, etc. of the target vehicle in the dangerous scene are output.

[0061] In the disclosed embodiment, by mining the data of potential dangerous scenes in the driving data, potential dangerous scenes can be discovered in a timely manner during the autonomous driving process, and response plans for the dangerous scenes can be taken in a timely manner to ensure the safety of the vehicle's autonomous driving.

[0062] Figure 3 A flow chart of a method for determining a dangerous scene proposed by at least one embodiment of the present disclosure is shown, wherein the method is used to further analyze the mined potential dangerous scene to determine that the potential dangerous scene is indeed a dangerous scene. Figure 3 As shown, the method includes steps 301-305.

[0063] In step 301, first driving state information of the target vehicle and second driving state information of each target object are determined based on a data frame marked with a label corresponding to a potential danger scene.

[0064] For data frames marked as potentially dangerous scenes, the first driving state information of the target vehicle, such as at least one of the position, speed, direction, etc. of the target vehicle, and the second driving state information of the target object within a set range from the target vehicle, that is, the surrounding objects, such as at least one of the position, speed, etc. of the target object, can be determined from the vehicle state information and environmental perception information contained in the data frames.

[0065] In one example, the position, speed, direction, etc. of the target vehicle in the ENU coordinate system (Local Cartesian Coordinats coordinate system, station center coordinate system or station coordinate system) can be obtained based on the data frames in the at least one period marked with tags corresponding to the potential danger scene. Since the ENU coordinate system is used to understand the motion law of other objects centered on the observer, the relative relationship between the target vehicle and the target object can be determined by obtaining the speed and position of the target object in the coordinate system of the target vehicle.

[0066] In step 302, for each frame in the data frame marked with a label corresponding to a potential dangerous scene, the size of the target vehicle and each target object in the frame is expanded at a set ratio.

[0067] For example, the shapes of the target vehicle and each target object can be simplified to a rectangle, and the rectangular area corresponding to the target vehicle and the rectangular area corresponding to each target object can be expanded by a set ratio, for example, by 1.2 times. Those skilled in the art should understand that the set ratio can be set according to actual conditions, and the present disclosure does not limit the specific value of the set ratio.

[0068] In step 303, the overlap degree between the expanded target vehicle and each expanded target object is determined according to the first driving state information of the target vehicle and the second driving state information of each target object.

[0069] For example, based on the position information of the target vehicle and the position information of any one of the target objects, the relative position relationship between the two can be determined, so that the overlap degree between the expanded target vehicle and the expanded target object can be determined.

[0070] In step 304, the target object whose overlap degree is greater than a second set threshold is determined to be the second object.

[0071] In response to the fact that the overlap between the expanded target vehicle and any one of the target objects in one of the data frames marked with a label corresponding to the potential dangerous scene is greater than the second set threshold, the target object is determined as a second object. The second object is a dangerous obstacle, that is, surrounding objects with a probability of collision higher than a certain standard are listed as dangerous obstacles.

[0072] The second object determined in this step may be one target object, or may include two or more target objects. In the case where the second object includes two or more target objects, subsequent judgments are performed on the target objects included in the second object respectively.

[0073] In step 305, in response to the continuous sequence of data frames continuously collected from the data frames in which the overlap between the second object and the target vehicle is greater than the second set threshold value, satisfying the set condition, the label of the data frame containing the continuous sequence and marked with a label corresponding to the potential dangerous scene is modified to a label corresponding to the dangerous scene. In one example, for a determined second object, in a plurality of data frames in which the overlap between the second object and the target vehicle is greater than the second set threshold value, the duration of the continuous sequence of data frames continuously collected is the duration of the overlap between the second object and the target vehicle being greater than the second set threshold value.

[0074] In one example, the set conditions satisfied by the continuous sequence include: the duration of the continuous sequence is greater than the first time threshold; and / or the target count value is greater than the first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than the second time threshold for more than the second set number of times, and the initial value of the target count value is 0, wherein the first time threshold is greater than the second time threshold. That is, when the duration of the dangerous obstacle is long, or the duration of the dangerous obstacle is too long, the potential dangerous scene is determined as a dangerous scene. For example, when the duration of the continuous sequence is greater than the first time threshold (for example, 1s), the label corresponding to the potential dangerous scene is modified to the label corresponding to the dangerous scene; for another example, in the 1st frame to the nth frame, the overlap between the second object and the target vehicle is greater than the second set threshold, and the target count value is increased by 1; in the n+1th frame, the overlap between the second object and the target vehicle is not greater than the second set threshold; in the n+2th to 2n+1th frames, the overlap between the second object and the target vehicle is greater than the second set threshold, and the target count value is increased by 1, and so on. When the target count value is greater than the first set number (for example, 2 times), it is determined to be a dangerous scene.

[0075] In the disclosed embodiments, by analyzing potential dangerous scenarios and further determining them as dangerous scenarios, dangerous scenarios can be detected more accurately during the autonomous driving process, and response plans for dangerous scenarios can be taken in a timely manner to ensure the safety of the vehicle's autonomous driving.

[0076] In some implementations, continuously collected data frames marked with labels corresponding to dangerous scenes constitute a dangerous scene. The dangerous scenes in the autonomous driving data can be sorted according to the timestamps carried by the data frames in each dangerous scene. In response to the time interval between two adjacent dangerous scenes of the same type being less than a third time threshold, the labels of the data frames constituting the two adjacent dangerous scenes are modified to labels corresponding to the same dangerous scene, that is, the autonomous driving data corresponding to the adjacent dangerous scenes are merged. Among them, the time interval between adjacent dangerous scenes can be determined based on the time interval between the timestamp of the last frame of the previous dangerous scene and the timestamp of the first frame of the next dangerous scene, or it can be determined in other ways. Through the above method, dangerous scenes can be processed continuously and stably.

[0077] In some implementations, the autonomous driving data generated may also be labeled online during the autonomous driving process of the target vehicle.

[0078] In one example, when a preset event occurs, such as entering a takeover state from an automatic driving state, or an emergency braking event occurs, the automatic driving data within a set time range when the preset event occurs can be annotated with third annotation information, and the third annotation information indicates the preset event.

[0079] For example, in the event of an emergency braking event, the autonomous driving data within 5 seconds of the emergency braking event may be labeled as an "emergency braking event."

[0080] In the disclosed embodiment, by automatically labeling the autonomous driving data online during the autonomous driving process, more autonomous driving data labeled with events can be provided for autonomous driving simulation testing or field testing while improving standard efficiency.

[0081] Figure 4 A schematic diagram of a data annotation device proposed in at least one embodiment of the present disclosure is shown. Figure 4As shown, the device includes: a first acquisition unit 401, used to acquire the automatic driving data of the target vehicle; a second acquisition unit 402, used to obtain first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the automatic driving data; a determination unit 403, used to determine target data that meets the standards of a preset scenario based on the first data and the second data; a first labeling unit 404, used to label the target data with a label corresponding to the preset scenario.

[0082] In some implementations, the autonomous driving data includes multiple data frames, and the second acquisition unit is specifically used to: obtain a first data sequence for the target vehicle from the multiple data frames included in the autonomous driving data; for each target object, obtain a second data sequence of the target object from the multiple data frames included in the autonomous driving data; and determine the target data that meets the preset scene standards from the autonomous driving data based on the first data and the second data, including: determining the target data frame that meets the preset scene standards based on the first data sequence and the second data sequence of each target object.

[0083] In some implementations, the data frames in the autonomous driving data carry a timestamp; the determination unit is specifically used to determine target data that meets the standards of a preset scenario based on a first data sequence and a second data sequence that carry the same timestamp.

[0084] In some implementations, the multiple data frames included in the autonomous driving data are divided into multiple cycles, each cycle containing multiple data frames; the determination unit is specifically used to: determine the probability of the target vehicle colliding with each target object based on a first data sequence in at least one cycle and a second data sequence of each target object; determine the target object whose probability of collision is greater than a first set probability threshold as the first object; in response to the number of the first objects being greater than a first set number in the at least one cycle, determine the data frame in the at least one cycle as the target data frame; labeling the target data with a label corresponding to the preset scene, including: labeling the target data frame with a label corresponding to the potential danger scene.

[0085] In some implementations, the device also includes a dangerous scene labeling unit, which is used to: determine the first driving state information of the target vehicle and the second driving state information of each target object based on the data frame labeled with the label corresponding to the potential dangerous scene; for each frame in the data frame labeled with the label corresponding to the potential dangerous scene, expand the size of the target vehicle and each target object in the frame at a set ratio; determine the overlap between the expanded target vehicle and each expanded target object based on the first driving state information of the target vehicle and the second driving state information of each target object; determine the target object whose overlap is greater than a second set threshold as the second object; in response to the continuous sequence composed of continuously collected data frames in the data frame whose overlap with the target vehicle is greater than the second set threshold satisfying the set condition, modify the label of the data frame containing the continuous sequence and labeled with the label corresponding to the potential dangerous scene to the label corresponding to the dangerous scene.

[0086] In some implementations, the continuous sequence satisfies set conditions, including: the duration of the continuous sequence is greater than a first time threshold; and / or a target count value is greater than a first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than a second time threshold for a number of times greater than a second set number, and the initial value of the target count value is 0; wherein the first time threshold is greater than the second time threshold.

[0087] In some implementations, continuously collected data frames marked with labels corresponding to dangerous scenes constitute a dangerous scene, and the device also includes a merging unit for modifying the labels of the data frames constituting two adjacent dangerous scenes to labels corresponding to the same dangerous scene in response to the time interval between two adjacent dangerous scenes being less than a third time threshold.

[0088] In some implementations, the device also includes a second labeling unit, which is used to: in response to the occurrence of a preset event, label the autonomous driving data within a set time range when the preset event occurs with a label corresponding to the preset event.

[0089] At least one embodiment of the present disclosure further provides an electronic device, such as Figure 5 As shown, the device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method described in any embodiment of the present disclosure when executing the computer instructions.

[0090] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0091] At least one embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the method described in any embodiment of the present disclosure when the program is executed by a processor.

[0092] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0095] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0096] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0097] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0099] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0100] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0101] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data labeling method, characterized in that: The method comprises: Obtain the autonomous driving data of the target vehicle; Obtaining first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data, wherein the first data includes vehicle state data of the target vehicle, and the second data is obtained from environmental perception data detected by a sensor disposed in the target vehicle; Determining target data that meets the criteria of a preset scenario based on the first data and the second data; Labeling the target data with a label corresponding to the preset scenario; When the preset scene is a potential danger scene, determining a second object from each target object according to a data frame marked with a label corresponding to the potential danger scene, the second object being a target object whose overlap between the target vehicle and each target object after size expansion is greater than a second set threshold, the size expansion being determined based on driving state information of the target vehicle and each target object in the data frame; In response to a continuous sequence of continuously collected data frames corresponding to the second object satisfying a set condition, a label of a data frame containing the continuous sequence and marked with a label corresponding to a potential dangerous scene is modified to a label corresponding to a dangerous scene.

2. The method according to claim 1, characterized in that The autonomous driving data includes a plurality of data frames, and obtaining first data for the target vehicle and second data of at least one target object within a set range from the target vehicle from the autonomous driving data includes: Obtaining a first data sequence for the target vehicle from a plurality of data frames included in the autonomous driving data; For each target object, obtaining a second data sequence of the target object from a plurality of data frames included in the autonomous driving data; The determining, according to the first data and the second data, target data meeting a preset scenario standard from the autonomous driving data includes: A target data frame that meets a preset scene standard is determined according to the first data sequence and the second data sequence of each target object.

3. The method according to claim 2, characterized in that The data frames in the autonomous driving data carry timestamps; determining the target data frames that meet the preset scene standards based on the first data sequence and the second data sequence of each target object includes: determining the target data that meets the preset scene standards based on the first data sequence and the second data sequence carrying the same timestamp.

4. The method according to claim 2 or 3, characterized in that: The multiple data frames included in the autonomous driving data are divided into multiple periods, each period including multiple data frames; The step of determining a target data frame that meets a preset scene standard according to the first data sequence and the second data sequence of each target object includes: Determining the probability of the target vehicle colliding with each target object based on the first data sequence in at least one cycle and the second data sequence of each target object; Determine a target object whose collision probability is greater than a first set probability threshold as a first object; In response to the number of the first objects being greater than a first set number within the at least one period, determining the data frames within the at least one period as target data frames; The step of labeling the target data with a label corresponding to the preset scenario includes: The target data frame is labeled with a label corresponding to the potential danger scenario.

5. The method according to claim 1, characterized in that The continuous sequence satisfies the set conditions, including: The duration of the continuous sequence is greater than the first time threshold; and / or, The target count value is greater than a first set number, wherein the target count value is increased by 1 when the duration of the continuous sequence is greater than the second time threshold and the number of times is greater than the second set number, and the initial value of the target count value is 0; The first time threshold is greater than the second time threshold.

6. The method according to claim 1 or 5, characterized in that: The continuously collected data frames marked with labels corresponding to the dangerous scene constitute a dangerous scene, and the method further includes: In response to the time interval between two adjacent dangerous scenes being less than a third time threshold, the labels of the data frames constituting the two adjacent dangerous scenes are modified to a label corresponding to the same dangerous scene.

7. The method according to claim 1, characterized in that The method further comprises: In response to a preset event occurring, the autonomous driving data within a set time range when the preset event occurs is marked with a label corresponding to the preset event.

8. A data labeling device, characterized in that: The device comprises: A first acquisition unit, used to acquire the automatic driving data of the target vehicle; a second acquisition unit, configured to obtain, from the autonomous driving data, first data for the target vehicle and second data of at least one target object within a set range from the target vehicle, wherein the first data includes vehicle state data of the target vehicle, and the second data is obtained from environmental perception data detected by a sensor disposed in the target vehicle; a determining unit, configured to determine target data that meets a standard of a preset scenario according to the first data and the second data; A first labeling unit, configured to label the target data with a label corresponding to the preset scenario; a dangerous scene labeling unit, configured to, when the preset scene is a potential dangerous scene, determine a second object from each target object according to a data frame labeled with a label corresponding to the potential dangerous scene, wherein the second object is a target object whose overlap between the target vehicle and each target object after size expansion is greater than a second set threshold, wherein the size expansion is determined based on driving state information of the target vehicle and each target object in the data frame; In response to a continuous sequence of continuously collected data frames corresponding to the second object satisfying a set condition, a label of a data frame containing the continuous sequence and marked with a label corresponding to a potential dangerous scene is modified to a label corresponding to a dangerous scene.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images