Method, apparatus, and electronic device for processing driving data

Abstract

The present invention provides a method, apparatus and electronic device for processing driving data. Driving data of a vehicle is acquired through a preset sensor installed on the vehicle; a target scenario element included in the driving data is determined from a variety of preset scenario elements; wherein, each preset scenario element corresponds to a label category, each preset scenario element corresponds to a variety of scenario information, each label category includes multiple status labels, and one scenario information corresponds to one status label; based on the information screening method corresponding to the target scenario element, the scenario information included in the target scenario element is determined; according to the status label corresponding to the scenario information included in the target scenario element, the driving data is labeled to obtain labeled data. The present invention labels and stores valid data, greatly reducing the data storage amount. The stored driving data contains hundreds of status label combinations, facilitating data search, and data details can be retrieved and mined through the status labels.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, device, and electronic device for processing driving data. Background Art

[0002] Only by the dual drive of algorithms and data can more mature autonomous driving technologies be achieved, which is also an inevitable choice for the implementation of autonomous driving technologies. The occasionally occurring corner cases (extreme situations) are one of the sources for upgrading the data-driven algorithm models. As time goes by, the gap in algorithms among various companies will gradually narrow. What really affects the level of autonomous driving technology is actually the data. Massive data will help the autonomous driving technology reach a new level. In the prior art, massive data is collected by continuously collecting data using in-vehicle sensors to form a data closed-loop.

[0003] However, the amount of data collected by the prior art is huge, which brings great pressure to data storage and transmission. In addition, a lot of the collected data is invalid data. When it is necessary to find and apply a small amount of valuable data scenarios from the massive data, it is necessary to manually record the situation at that time during the test process or the collection process. After the data is collected, it is also necessary to rely on manual playback of the data to identify it during the process and label it. This process is not only inefficient but also prone to missing many details, resulting in the inability to fully reproduce all the details when viewing the data again. Summary of the Invention

[0004] The purpose of the present invention is to provide a driving data method, device, and processing method to solve the problem that all details cannot be reproduced when viewing the data again.

[0005] In a first aspect, the present invention provides a method for processing driving data, the method comprising: obtaining driving data of a vehicle through a preset sensor installed on the vehicle; determining a target scenario element included in the driving data from a variety of preset scenario elements; wherein, each preset scenario element corresponds to a label category, each preset scenario element corresponds to a variety of scenario information, each label category includes a plurality of status labels, and one scenario information corresponds to one status label; determining the scenario information included in the target scenario element based on the information screening method corresponding to the target scenario element; and annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element to obtain annotated data.

[0006] In an optional embodiment, after the step of annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element, the method further comprises: deleting the driving data other than the annotated data corresponding to the target scenario element in the driving data.

[0007] In an alternative embodiment, the step of obtaining the driving data of the vehicle through the preset sensors installed on the vehicle includes: determining the driving data of the vehicle through a plurality of laser point clouds of a lidar; the step of determining the scene information included in the target scene element based on the information screening method corresponding to the target scene element includes: performing time synchronization on the plurality of laser point clouds to obtain spliced point cloud data; determining scene parameter information from the spliced point cloud data; obtaining scene information according to the spliced point cloud data and the scene parameter information; wherein the scene information includes the speed magnitude and direction of the scene object.

[0008] In an alternative embodiment, the step of labeling the driving data according to the status label corresponding to the scene information included in the target scene element includes: determining whether the speed magnitude and direction of the scene object meet the preset status threshold; if the speed magnitude and direction of the scene object meet the preset status threshold, respectively label the multiple scene information corresponding to each preset scene element in the driving data.

[0009] In an alternative embodiment, the step of determining the scene parameter information from the spliced point cloud data includes: detecting the scene status information indicated by the target scene element from the spliced point cloud data by using a first preset algorithm; filtering the point cloud data of the scene status information according to the first preset algorithm to obtain target point cloud data; using a second preset algorithm to segment the target point cloud data to determine the scene parameter information in the target point cloud data.

[0010] In an alternative embodiment, the step of obtaining scene information according to the spliced point cloud data and the scene parameter information includes: obtaining a plurality of scene parameter information according to the preset tracking time threshold and the scene parameter information in the spliced point cloud data; obtaining scene information according to the change mode of the plurality of scene parameter information.

[0011] In an alternative embodiment, the driving data of the vehicle is determined through image information; the step of determining the scene information included in the target scene element based on the information screening method corresponding to the target scene element; and labeling the driving data according to the status label corresponding to the scene information included in the target scene element includes: determining scene information based on the image information; wherein the scene information includes scene color information and scene shape information; classifying the scene information according to the scene color information to obtain a classification result; determining the scene shape information based on the classification result; when the classification result and the scene shape information meet the preset labeling conditions, respectively label the multiple scene information corresponding to each preset scene element in the driving data.

[0012] In an alternative embodiment, the method further includes: associating the labeled data with the driving data.

[0013] Second aspect, the present invention provides a processing device for driving data, which is applied to the above method; the device includes: a driving data acquisition module for acquiring driving data of a vehicle through a preset sensor installed on the vehicle; a scene element determination module for determining a target scene element included in the driving data from a variety of preset scene elements; wherein, each preset scene element corresponds to a label category, each preset scene element corresponds to a variety of scene information, each label category includes multiple status labels, and one scene information corresponds to one status label; a scene information determination module for determining the scene information included in the target scene element based on the information screening method corresponding to the target scene element; a labeling module for labeling the driving data according to the status label corresponding to the scene information included in the target scene element.

[0014] Third aspect, the present invention provides an electronic device, which includes a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above driving data processing method.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] The present invention provides a method, device and electronic device for processing driving data, which use the existing sensors and data acquisition equipment of the vehicle to identify driving data, and perform data screening and label recognition according to algorithms. When the status label is triggered, the corresponding driving data is labeled and stored in the form of a label. All the labels after scene understanding, feature extraction, data mining, and unified format arrangement are implemented at the vehicle end, and the data is no longer recorded in full volume, greatly reducing the data storage volume. Therefore, the data transmission efficiency can be greatly improved. For dedicated data acquisition, the data transmission period can be changed from once a day to once a week or once a month. And because only the required valid data is retained, the storage space of the data server can use a limited data space to store the database for data mining, data retrieval, and data simulation.

[0017] In addition, the embodiments of the present invention also associate and save the labeled data with the driving data. The stored driving data contains hundreds of status label combinations of multiple label categories. Through the hundreds of status label combinations of multiple label categories, the current scene of the data can be vividly described, which facilitates data search. In addition, because the driving data is labeled and stored, when mining the driving data, relying on the label, the subsequent driving data, due to the existence of the label, is not just a combination of some information, but three-dimensional information with scene descriptions. The data details can be retrieved and mined through the feature descriptions of the scene information.

[0018] Other features and advantages of the present invention will be set forth in the following description, or can be learned by inference from the description, or can be learned by implementing the above technologies of the present invention without any doubt.

[0019] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. Description of the Drawings

[0020] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of a method for processing driving data provided by an embodiment of the present invention;

[0022] Figure 2 It is a flowchart of another method for processing driving data provided by an embodiment of the present invention;

[0023] Figure 3 It is a flowchart of another method for processing driving data provided by an embodiment of the present invention;

[0024] Figure 4 It is a schematic structural diagram of a device for processing driving data provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Specific Embodiments

[0026] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present invention claimed, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0028] Only by the dual drive of algorithms and data can more mature autonomous driving technologies be achieved, which is also an inevitable choice for the implementation of autonomous driving technologies. The occasional corner cases are one of the sources for upgrading the data-driven algorithm models. As time goes by, the gap in algorithms among companies will gradually narrow. What really affects the level of autonomous driving technology is data. Massive data will help autonomous driving technology reach a new level. In the existing technology, massive data is collected by continuously collecting data through in-vehicle sensors to form a data closed-loop.

[0029] However, the amount of data collected by the existing technology is huge, bringing considerable pressure to data storage and transmission. In addition, a lot of the collected data is invalid data. When it is necessary to find and apply a small amount of valuable data scenarios from massive data, it is necessary to manually record the situation at that time during the test process or the collection process. After the data is collected, it is also necessary to rely on manual playback of the data to identify and label it during the process. This process is not only inefficient but also prone to missing many details, resulting in the inability to fully reproduce all the details when viewing the data again for data mining, data retrieval, and data simulation.

[0030] Based on the above problems, the embodiments of the present invention provide a method, device, and electronic device for processing driving data. This technology can be applied to scenarios of obtaining driving data. For the convenience of understanding this embodiment, first, a method for processing driving data disclosed in the embodiments of the present invention will be introduced in detail; as Figure 1 shown, the method includes the following steps:

[0031] Step S102, obtain the driving data of the vehicle through preset sensors installed on the vehicle.

[0032] The key environmental perception of autonomous driving is used to collect the basic information of the surrounding environment and is also the basis of autonomous driving. An autonomous driving vehicle perceives the environment through preset sensors. The sensors are like the eyes of the vehicle, and there are many types of sensors, such as cameras, ultrasonic radars, millimeter-wave radars, and lidar. Due to different autonomous driving routes and different levels of autonomous driving achieved, there are slight differences in the types of sensors deployed.

[0033] Cameras can collect image information and are closest to human vision. Through the collected images, rich environmental information can be identified through computer algorithm analysis, such as pedestrians, bicycles, motor vehicles, road trajectory lines, curbs, road signs, traffic lights, etc. With the addition of algorithms, vehicle distance measurement and road tracking can also be achieved, thereby realizing forward collision warning (abbreviation: FCW) and lane departure warning (abbreviation: LDW).

[0034] An ultrasonic radar, also known as an ultrasonic sensor, is developed using ultrasonic characteristics. It is an energy conversion device that converts an alternating electrical signal into a sound signal within the ultrasonic frequency range or converts a sound signal in the external sound field into an electrical signal. It is usually used to obtain the distance between the object to be measured and the ultrasonic radar.

[0035] Millimeter waves are actually electromagnetic waves. The distance to an object is determined by transmitting radio signals and receiving the reflected signals, and its frequency is usually between 10 and 300 GHz.

[0036] A lidar is a radar system that detects information such as the position and speed of a target by emitting laser beams. Its working principle is to emit a laser beam towards the target, and then compare the received echo reflected from the target with the transmitted signal. After calculation and analysis, relevant information about the target can be obtained, such as parameters like the target distance, azimuth, altitude, speed, attitude, and even shape.

[0037] In this embodiment, the above-mentioned preset sensors mainly obtain driving data corresponding to extreme situations. Therefore, in this embodiment, the specific preset sensors to be used are determined according to extreme situations.

[0038] Step S104: Determine the target scenario elements included in the driving data from various preset scenario elements; among them, each preset scenario element corresponds to a label category, each preset scenario element corresponds to multiple scenario information, each label category contains multiple state labels, and one scenario information corresponds to one state label.

[0039] In this embodiment, the driving data of the vehicle comes from the real scenario, and the real scenario contains various preset scenario elements. Among them, the preset scenario elements can be divided into elements of the test vehicle itself and external environment elements. The external environment elements include: static environment elements, dynamic environment elements, traffic participant elements, meteorological elements, etc. Specifically, the above-mentioned preset scenario elements are sensed by the existing sensors and data acquisition equipment of the vehicle.

[0040] Among them, each different preset scenario element contains different label categories. For example, the elements of the test vehicle itself contain multiple label categories, such as time, location, weight, geometric information, performance information, position status information, motion status information, sensor status, vehicle actuator status, driving task information, etc.; the static environment elements in the external environment elements include static obstacles, surrounding landscapes, traffic facilities, roads, etc.; the dynamic environment elements in the external environment elements include dynamic indication facilities, communication environment information, etc.; the traffic participant elements in the external environment include motor vehicles, non-motor vehicles, pedestrians, animals, etc.; the meteorological element environment temperature information, light condition information, weather condition information, etc. in the external environment. The preset scenario elements in the above-mentioned real scenarios and the label categories included in the above-mentioned preset scenario elements are only used to illustrate the technical solution of the present invention, rather than limiting it.

[0041] Each of the above-mentioned preset scenario elements corresponds to multiple scenario information, each label category contains multiple status labels, and one scenario information corresponds to one status label. The above-mentioned scenario information refers to the scenario details included in each preset scenario element, such as specific information such as color, shape or motion status. When the scenario information meets the preset data collection rules, it indicates that the corresponding preset scenario element is the target scenario element. The data collection rules refer to whether the scenario information belongs to an extreme situation.

[0042] Step S106, based on the information screening method corresponding to the target scenario element, determine the scenario information included in the target scenario element.

[0043] After determining the target scenario element, according to the scenario information included in the target scenario element, determine the preset information screening method, and then based on this information screening method, obtain the scenario information included in the target scenario element.

[0044] Specifically, the above-mentioned status labels can also be divided into labels used to trigger effective data recording and labels only used to describe data characteristics. The above-mentioned labels used to trigger effective data recording are determined by the user according to different strategies and requirements. After perceiving the above-mentioned preset scenario elements, according to the different label categories of the above-mentioned preset scenario elements and the status labels corresponding to the label categories, algorithm recognition is performed. The label used to trigger effective data recording is recognized through algorithms such as perception, prediction, decision-making, planning, and control, and it is determined whether the label used to trigger effective data recording is triggered to screen the above-mentioned scenario information.

[0045] Step S108, label the driving data according to the status label corresponding to the scenario information included in the target scenario element to obtain labeled data.

[0046] After filtering out the above scene information, label the scene information according to the status label corresponding to the scene information to obtain labeled data, where the labeled data includes labels used to trigger valid data recording in the driving data and labels only used to describe data characteristics.

[0047] Step S110: Delete the driving data other than the labeled data corresponding to the target scene elements in the driving data.

[0048] Specifically, when the above label used to trigger valid data recording is triggered, not only is the scene information labeled, but also the driving data is saved. Among them, the driving data within a predefined time threshold range can be saved. For example, when the label used to trigger valid data recording is the downgrade of the autonomous driving trigger function, when this label is triggered, the data 10 seconds before and 20 seconds after the function downgrade needs to be recorded, and the driving data outside this time threshold range is deleted, and the data is no longer recorded in full volume.

[0049] In addition, after obtaining the labeled data, the embodiments of the present invention also associate the labeled data with the acquired driving data in a predefined format and save them together. The labels used to trigger valid data recording and the labels only used to describe data characteristics in the labeled data are both recorded in a Json file. After the above labeled data is associated and saved with the driving data, the stored driving data contains hundreds of status label combinations of multiple label categories, and the current scene of the data can be vividly described through the hundreds of status label combinations of multiple label categories.

[0050] A method for processing driving data provided by the embodiments of the present invention uses the existing sensors and data acquisition equipment of the vehicle to identify the driving data, and filters and identifies the data according to an algorithm. When the status label is triggered, the corresponding driving data is labeled and stored in the form of a label. All the labels after scene understanding, feature extraction, data mining, and unified format sorting are implemented on the vehicle side, and the data is no longer recorded in full volume, greatly reducing the data storage volume. Therefore, the efficiency of data transmission can be greatly improved. For dedicated data acquisition, the data transmission cycle can be changed from once a day to once a week or once a month. And because only the required valid data is retained, the storage space of the data server can use a limited data space to store databases for data mining, data retrieval, and data simulation.

[0051] In addition, the embodiments of the present invention also associate and store the labeled data with the driving data. The stored driving data contains hundreds of status label combinations of multiple label categories. Through the hundreds of status label combinations of multiple label categories, the current data scenario can be vividly described, facilitating data search. In addition, due to the labeled storage of driving data, when mining driving data, relying on labels, subsequent driving data, due to the existence of labels, is not just a combination of some information, but three-dimensional information with scene descriptions. Data details can be retrieved and mined through the feature descriptions of scene information.

[0052] For the above embodiment, the present invention also provides another method for processing driving data, which is implemented on the basis of the above method; as Figure 2 shown, the method includes the following steps:

[0053] Step S202, determine the driving data of the vehicle through multiple laser point clouds of the lidar.

[0054] Step S204, determine the target scene elements included in the driving data from multiple preset scene elements.

[0055] In this embodiment, the above target scene elements are different obstacles determined by the user according to different strategies and requirements, and the labels used to trigger the recording of valid data are the status details of different obstacles determined by the user according to different strategies and requirements. Specifically, when obtaining the driving data corresponding to different obstacles in the real environment, it is obtained through the recognition of multiple lidars. Among them, multiple lidars detect and identify the obstacles in the driving data. When an obstacle in the driving data is recognized, multiple laser point clouds indicated by the obstacle are obtained.

[0056] Step S206, perform time synchronization on the multiple laser point clouds to obtain the spliced point cloud data.

[0057] After obtaining the multiple laser point clouds indicated by the obstacle, first input the laser point cloud data driven by each lidar, and then perform time synchronization on each laser point cloud data. After time synchronization, the coordinates of each laser point cloud can be obtained. Then, each laser point cloud coordinate is converted to the ego-vehicle coordinate system according to the external parameters. Finally, the laser point clouds of multiple lidars are spliced into one frame of point cloud in the ego-vehicle coordinate system.

[0058] Specifically, in the above time synchronization method, one of the multiple lidars is used as the main sensor, and the rest of the lidars are secondary sensors. The main sensor is a sensor capable of sensing the driving data directly in front of the vehicle (if there is only one lidar, the current step of stitching point cloud data is skipped). The lidar point cloud sensed by the main sensor is the main lidar point cloud, and the lidar point cloud sensed by the secondary sensors is the secondary lidar point cloud. Starting from any moment, the lidar point cloud data is received. If the secondary lidar point cloud is received, it is cached in the corresponding buffer. If the main lidar point cloud is received, after waiting for 50 ms, the frame with the closest timestamp is searched for in each secondary lidar point cloud. If the timestamp difference between this frame and the main lidar point cloud timestamp exceeds 50 ms, it is discarded; otherwise, it enters the coordinate transformation step. After performing coordinate transformation on the above multiple lidar point cloud data, the stitched point cloud data is obtained.

[0059] Step S208: Determine the scene parameter information from the stitched point cloud data.

[0060] Among them, the above scene parameter information is determined through the following steps 20 - 22:

[0061] Step 20: Use the first preset algorithm to detect the scene state information indicated by the target scene elements from the stitched point cloud data.

[0062] Step 21: Filter the point cloud data of the scene state information according to the first preset algorithm to obtain the target point cloud data.

[0063] Step 22: Use the second preset algorithm to segment the target point cloud data to determine the scene parameter information in the target point cloud data.

[0064] After obtaining the above stitched point cloud data, input the stitched point cloud data, and then use the CNNSeg algorithm to detect the obstacle information, including position, orientation, size, and type. Then, remove the ground points and obstacle points in the stitched point cloud data to filter the stitched point cloud data and obtain the target point cloud data. Among them, the target point cloud data also needs to be detected and verified; specifically, after obtaining the target point cloud data, the NCut algorithm is called, which can be selected not to be called through the configuration file. The NCut algorithm segments the target point cloud data, detects the above-filtered target point cloud data, and returns the detection result. The returned detection result includes the obstacle position, orientation, and size. The obstacle position, orientation, size, and type information in the detected target point cloud data is the scene parameter information in the target point cloud data.

[0065] Step S210: Obtain the scene information according to the stitched point cloud data and the scene parameter information; among them, the scene information includes the speed magnitude and direction of the scene objects.

[0066] Specifically, the above scene information is determined through the following steps 30-31:

[0067] In step 30, according to a preset tracking time threshold and the scene parameter information in the spliced point cloud data, multiple pieces of scene parameter information are obtained.

[0068] In step 31, according to the change mode of the multiple pieces of scene parameter information, the scene information is obtained.

[0069] After obtaining the scene parameter information through the above steps, Kalman filtering and Hungarian matching are adopted. According to the preset tracking time threshold, the obstacles among multiple frames of driving data are tracked and matched. At this time, the scene parameter information of each frame of the obstacle can be obtained. According to the change mode of the scene parameter information of each frame of the obstacle, the speed magnitude and direction of the obstacle can be calculated and output.

[0070] Step S212, determine whether the speed magnitude and direction of the scene object satisfy the preset state threshold.

[0071] Step S214, if the speed magnitude and direction of the scene object satisfy the preset state threshold, label the multiple pieces of scene information corresponding to each preset scene element in the driving data respectively.

[0072] In this embodiment, the label used to trigger the effective data record is determined according to whether the speed magnitude and direction of the obstacle are extreme cases. When the speed magnitude and direction of the above obstacle satisfy the preset state threshold, it means that the label used to trigger the effective data record is triggered. At this time, the multiple pieces of scene information corresponding to each preset scene element in the driving data are labeled respectively to obtain the labeled data. The multiple pieces of scene information include, but are not limited to, the speed magnitude and direction of the obstacle indicated by the triggered label.

[0073] In addition, after obtaining the labeled data, the embodiment of the present invention also associates the labeled data with the acquired driving data in a conventional format and saves them together. The labels used to trigger the effective data record and the labels only used to describe the data characteristics in the labeled data are both recorded in a Json file.

[0074] In addition, the driving data corresponding to different obstacles in the real environment can also be obtained through visual perception. The method for determining whether the above state label is triggered is similar to the determination method obtained by the above lidar, except that the format of the acquired driving data is different.

[0075] Another method for processing driving data provided by an embodiment of the present invention obtains obstacle information in driving data through lidar or visual perception, and judges the obstacle information according to an algorithm. When the obtained obstacle information matches a label defined to trigger the recording of valid data, the data recording is triggered, and only the corresponding driving data is recorded, instead of recording all driving data in full, which greatly reduces the data storage amount. In addition, the embodiment of the present invention also associates and stores the labeled data with the driving data. The stored driving data contains hundreds of status label combinations of multiple label categories. Through the hundreds of status label combinations of multiple label categories, the current data scene can be vividly described, which facilitates data search. In addition, due to the labeled storage of driving data, when mining driving data, relying on labels, subsequent driving data, due to the existence of labels, is not just a combination of some information, but three-dimensional information with scene descriptions. Data details can be retrieved and mined through the feature descriptions of scene information.

[0076] Another method for processing driving data provided by an embodiment of the present invention is implemented on the basis of the above method; as Figure 3 shown, the method includes:

[0077] Step S302, obtain the driving data of the vehicle through a preset sensor installed on the vehicle.

[0078] Step S304, determine the target scene elements included in the driving data from a variety of preset scene elements.

[0079] In this embodiment, the above target scene elements are traffic signals determined by the user according to different strategies and requirements, and the above labels used to trigger the recording of valid data are the status details of traffic signals determined by the user according to different strategies and requirements, such as the color and shape of traffic signals. Specifically, when obtaining the driving data corresponding to traffic signals in the real environment, it is determined through the image information obtained by the camera.

[0080] Step S306, based on the above image information, determine the scene information indicated by the target scene elements.

[0081] Step S308, classify the scene information according to the scene color information to obtain a classification result.

[0082] Step S310, based on the classification result, determine the scene shape information.

[0083] Specifically, the above scene information includes scene color information and scene shape information. In specific implementation, driving data corresponding to road traffic signals is obtained through a camera to obtain image information. Then, through a 2D detection task, traffic signals in the image information are detected, and pixel positions of traffic signals with high recall are given. At the same time, an image bounding box of the traffic signal can be output, and coordinate information of the traffic signal on the image information is provided.

[0084] The neural network loss function of the 2D detection task is shown in the following formula:

[0085]

[0086] Among them, Loss in the above formula represents the detection loss value. Specifically, when performing the 2D detection task, there are a total of S*S detection grids, and each grid generates B candidate boxes (anchor boxes). Each candidate box will finally obtain a corresponding bounding box through the network. Finally, S*S*B bounding boxes will be obtained.

[0087] When the j-th anchor box of the i-th grid is responsible for a certain object, the bounding box generated by the anchor box is compared with the box of the real target to calculate the center coordinate error; when the j-th anchor box of the i-th grid is responsible for a certain real target, the bounding box generated by the anchor box is compared with the box of the real target to calculate the width and height error. Among them, x, y, w, and h represent the x, y, w, and h direction coordinates of the real box of the target in a certain picture; represents the center point coordinates of the bounding box.

[0088] The confidence level that there is indeed an object in the boxed box and the confidence level that the boxed box includes all the features of the entire object. Regardless of whether the anchor box is responsible for a certain target, the confidence error will be calculated. represents whether the j-th anchor box of the i-th grid is responsible for this object. If so, the value is 1, otherwise it is 0; represents that the j-th anchor box of the i-th grid is not responsible for this object; the parameter confidence represents the true value, The value of is determined by whether the bounding box of the grid cell is responsible for predicting a certain object. If it is responsible, then Otherwise,

[0089] When the j-th anchor box of the i-th grid is responsible for a certain real target, the bounding box generated by the anchor box calculates the classification loss function; where P i represents the distribution probability.

[0090] Extract the traffic signal rect from the bounding box of the traffic signal output by the 2D detection task, that is, the upper left corner coordinates, width, and height of the rectangular box range of the traffic signal. Then send the traffic signal into the traffic signal classifier to output the traffic signal color category and obtain the classification result. Among them, the states of the traffic signal include four categories: red category, yellow category, green category, and unknown category; specifically, the above-mentioned unknown category is the color category that is similar to the three color categories of red, yellow, and green detected by the detection network from the image information, and the unknown category contains non-traffic signal features. The traffic light classification task uses the resnet18 network for classification.

[0091] Among them, the network loss function of the traffic signal classifier is where P j is the j-th value of the input probability vector P, and y i is the i-th value of the label y of the sample.

[0092] After determining the color of the above traffic signal, classify the shape of the traffic signal to obtain the scene shape information, and interpret the more specific semantic information of the traffic signal according to the coordinate information of the traffic signal on the image information and the color state of each traffic signal, such as straight red light, straight green light, straight yellow light, left red light, left green light, left yellow light, right red light, right green light, right yellow light.

[0093] Step S312, when the classification result and the scene shape information meet the preset annotation conditions, annotate the multiple scene information corresponding to each preset scene element in the driving data respectively to obtain the annotation data.

[0094] In this embodiment, when the classification result and the scene shape information meet the preset annotation conditions, various scene information corresponding to each preset scene element in the driving data is respectively annotated. Specifically, the label used to trigger the recording of valid data is determined according to whether the color and shape of the above traffic signal are extreme cases. For example, when the color indicated by the traffic signal in the image information is of the unknown class, it means that the detected traffic signal in the image information is a misdetection result similar to the traffic signal color. If the autonomous vehicle follows this misdetected traffic signal for autonomous driving, situations such as running a red light and violating traffic rules may occur. Therefore, the color classification of the above traffic signal being of the unknown class can be defined as an extreme case, that is, the color classification of the above traffic signal being of the unknown class can be the label used to trigger the recording of valid data. When the color classification of the above traffic signal is of the unknown class, the data annotation condition can be triggered. At this time, various scene information corresponding to each preset scene element in the driving data is respectively annotated to obtain annotated data. The various scene information includes, but is not limited to, the color classification of the above traffic signal being of the unknown class indicated by the triggered label.

[0095] In addition, traffic signals of the arrow type that are relatively far away are relatively blurred in image display and it is not easy to distinguish whether it is an arrow light or a round light, which will affect the traffic signal classification accuracy and may also cause situations that affect autonomous driving and lead to violations of traffic rules such as running a red light. Then, at this time, the state of this traffic signal can also be defined as an extreme case, that is, the above traffic signal being of a blurred shape is the label used to trigger the recording of valid data. When this label is triggered, various scene information corresponding to each preset scene element in the driving data is respectively annotated to obtain annotated data. The various scene information includes, but is not limited to, the above traffic signal being of a blurred shape indicated by the triggered label.

[0096] In addition, after obtaining the annotated data, the embodiment of the present invention also associates the annotated data with the obtained driving data in a conventional format and saves them together. The labels used to trigger the recording of valid data and the labels only used to describe data characteristics in the annotated data are both recorded in a Json file.

[0097] Another method for processing driving data provided by an embodiment of the present invention obtains driving data by acquiring image information through a camera, and judges and classifies the results output in the image information according to an algorithm, and matches the classification results with tags defined for triggering the recording of valid data. Once the defined tags are matched, the data recording is triggered, and only the corresponding driving data is recorded, and the driving data is no longer recorded in full volume, greatly reducing the data storage amount. In addition, the embodiment of the present invention also associates and stores the labeled data with the driving data. The stored driving data contains hundreds of status tag combinations of multiple tag categories. Through the hundreds of status tag combinations of multiple tag categories, the current data location scenario can be vividly described, facilitating data search. In addition, due to the labeled storage of driving data and relying on tags when mining driving data, subsequent driving data, due to the existence of tags, is not just a combination of some information, but three-dimensional information with scene descriptions, and data details can be retrieved and mined through the feature descriptions of the scene information.

[0098] Based on the above system embodiment, an embodiment of the present invention further provides a driving data processing device, and this device is applied to the above method; as Figure 4 shown, this device includes:

[0099] A driving data acquisition module 401, configured to acquire driving data of a vehicle through a preset sensor installed on the vehicle.

[0100] A scene element determination module 402, configured to determine target scene elements included in the driving data from a variety of preset scene elements; wherein, each preset scene element corresponds to a tag category, each preset scene element corresponds to a variety of scene information, each tag category includes multiple status tags, and one scene information corresponds to one status tag.

[0101] A scene information determination module 403, configured to determine the scene information included in the target scene elements based on the information screening method corresponding to the target scene elements.

[0102] A labeling module 404, configured to label the driving data according to the status tags corresponding to the scene information included in the target scene elements.

[0103] A processing device for driving data provided by an embodiment of the present invention identifies driving data by using existing vehicle sensors and data acquisition equipment, screens and identifies tags for the data according to an algorithm. When a status tag is triggered, the corresponding driving data is marked and stored in the form of a tag. All tags after scene understanding, feature extraction, data mining, and unified format collation are implemented on the vehicle side, and the data is no longer recorded in full volume, greatly reducing the data storage volume. Therefore, the data transmission efficiency can be greatly improved. For dedicated data acquisition, the data transmission cycle can be changed from once a day to once a week or once a month. And because only the required valid data is retained, the storage space of the data server can use limited data space to store databases for data mining, data retrieval, and data simulation.

[0104] In addition, the embodiment of the present invention also associates and stores the marked data with the driving data. The stored driving data contains hundreds of status tag combinations of multiple tag categories. Through the hundreds of status tag combinations of multiple tag categories, the current data scene can be vividly described, facilitating data search. In addition, because the driving data is marked and stored, when mining the driving data, relying on the tags, subsequent driving data, due to the existence of the tags, is not just a combination of some information, but three-dimensional information with scene descriptions. Data details can be retrieved and mined through the feature descriptions of the scene information.

[0105] In addition, by marking and storing the valid data through the embodiment of the present invention, only the algorithm module and the marking module need to be defined, which has better applicability and portability, does not increase costs, and can be continuously iterated and upgraded.

[0106] Specifically, the above device further includes: a data deletion module 405, configured to delete the driving data other than the marked data corresponding to the target scene elements in the driving data.

[0107] In some embodiments, the above driving data acquisition module 401 is further configured to determine the driving data of the vehicle through multiple laser point clouds of a lidar. The above scene information determination module 403 is further configured to perform time synchronization on the multiple laser point clouds to obtain spliced point cloud data; determine scene parameter information from the spliced point cloud data; and obtain scene information according to the spliced point cloud data and the scene parameter information; wherein the scene information includes the speed magnitude and direction of scene objects.

[0108] The above marking module 404 is further configured to determine whether the speed magnitude and direction of the scene object satisfy a preset status threshold; if the speed magnitude and direction of the scene object satisfy the preset status threshold, mark multiple pieces of scene information corresponding to each preset scene element in the driving data respectively.

[0109] The above-mentioned scene information determination module 403 is further configured to detect the scene state information indicated by the target scene elements from the spliced point cloud data by using a first preset algorithm; filter the point cloud data of the scene state information according to the first preset algorithm to obtain target point cloud data; and segment the target point cloud data by using a second preset algorithm to determine the scene parameter information in the target point cloud data.

[0110] The above-mentioned scene information determination module 403 is further configured to obtain multiple pieces of scene parameter information according to a preset tracking time threshold and the scene parameter information in the spliced point cloud data; and obtain scene information according to the change mode of the multiple pieces of scene parameter information.

[0111] In some embodiments, the above-mentioned annotation module 404 is further configured to determine scene information based on image information; where the scene information includes scene color information and scene shape information; classify the scene information according to the scene color information to obtain a classification result; determine the scene shape information based on the classification result; and when the classification result and the scene shape information meet the preset annotation conditions, respectively annotate multiple pieces of scene information corresponding to each preset scene element in the driving data.

[0112] Specifically, the above-mentioned device further includes a data association module 406, configured to associate the annotation data with the driving data.

[0113] An embodiment of the present invention further provides an electronic device, as Figure 5 shown, the electronic device includes a processor 101 and a memory 100, the memory 100 stores machine-executable instructions that can be executed by the processor 101, and the processor 101 executes the machine-executable instructions to implement the above-mentioned driving data processing method.

[0114] Further, Figure 5 the shown terminal device further includes a bus 102 and a communication interface 103, and the processor 101, the communication interface 103, and the memory 100 are connected through the bus 102.

[0115] Among them, the memory 100 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 103 (which can be wired or wireless), a communication connection between the system network element and at least one other network element is realized, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5It is represented only by a bidirectional arrow, but it does not mean that there is only one bus or one type of bus.

[0116] The processor 101 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 101 or the instructions in the form of software. The above-mentioned processor 101 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and combines its hardware to complete the steps of the method in the foregoing embodiments.

[0117] The embodiments of the present invention also provide a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above-mentioned driving data processing method. For the specific implementation, reference can be made to the method embodiments and will not be elaborated here.

[0118] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0119] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of various embodiments of the present invention.

Claims

1. A method for processing driving data, characterized in that, The method includes: Obtaining driving data of the vehicle through preset sensors installed on the vehicle; Determining a target scenario element included in the driving data from multiple preset scenario elements; wherein, each of the preset scenario elements corresponds to a label category, each of the preset scenario elements corresponds to multiple scenario information, each of the label categories includes multiple status labels, and one piece of scenario information corresponds to one status label; Determining the scenario information included in the target scenario element based on the information screening method corresponding to the target scenario element; Annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element to obtain annotated data; Wherein, the step of obtaining driving data of the vehicle through preset sensors installed on the vehicle includes: determining the driving data of the vehicle through multiple laser point clouds of a lidar; The step of determining the scenario information included in the target scenario element based on the information screening method corresponding to the target scenario element includes: Performing time synchronization on the multiple laser point clouds to obtain spliced point cloud data; Determining scenario parameter information from the spliced point cloud data; Obtaining the scenario information according to the spliced point cloud data and the scenario parameter information; wherein, the scenario information includes the speed magnitude and direction of a scenario object; The step of annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element includes: Judging whether the speed magnitude and direction of the scenario object meet a preset status threshold; If the speed magnitude and direction of the scenario object meet the preset status threshold, respectively annotating multiple pieces of scenario information corresponding to each of the preset scenario elements in the driving data; Wherein, the driving data of the vehicle is also determined through image information; The step of determining the scenario information included in the target scenario element based on the information screening method corresponding to the target scenario element; and annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element includes: Determining the scenario information indicated by the target scenario element based on the image information; wherein, the scenario information includes scenario color information and scenario shape information; Classifying the scenario information according to the scenario color information to obtain a classification result; Determining the scenario shape information based on the classification result; When the classification result and the scenario shape information meet a preset annotation condition, respectively annotating multiple pieces of scenario information corresponding to each of the preset scenario elements in the driving data.

2. The method according to claim 1, characterized in that, After the step of annotating the driving data according to the status label corresponding to the scenario information included in the target scenario element, the method further includes: Deleting the driving data other than the annotated data corresponding to the target scenario element in the driving data.

3. The method according to claim 1, characterized in that, The step of determining scenario parameter information from the spliced point cloud data includes: Detecting the scenario state information indicated by the target scenario element from the spliced point cloud data by using a first preset algorithm; Filter the point cloud data of the scene state information according to a first preset algorithm to obtain target point cloud data; Use a second preset algorithm to segment the target point cloud data to determine the scene parameter information in the target point cloud data.

4. The method according to claim 1, characterized in that The step of obtaining the scene information according to the spliced point cloud data and the scene parameter information includes: Obtain a plurality of pieces of the scene parameter information according to a preset tracking time threshold and the scene parameter information in the spliced point cloud data; Obtain the scene information according to the change mode of the plurality of pieces of the scene parameter information.

5. The method according to claim 1, wherein The method further includes: Associate the annotation data with the driving data.

6. A processing device for driving data, the device being applied to the method according to any one of the above-mentioned claims 1 to 5; characterized in that, The device includes: A driving data acquisition module, which acquires the driving data of the vehicle through a preset sensor installed on the vehicle; A scene element determination module, configured to determine a target scene element included in the driving data from a variety of preset scene elements; wherein, each of the preset scene elements corresponds to a label category, each of the preset scene elements corresponds to a variety of scene information, each of the label categories includes a plurality of state labels, and one piece of the scene information corresponds to one of the state labels; A scene information determination module, configured to determine the scene information included in the target scene element based on the information screening method corresponding to the target scene element; An annotation module, configured to annotate the driving data according to the state label corresponding to the scene information included in the target scene element; Wherein, the driving data acquisition module is further configured to determine the driving data of the vehicle through a plurality of laser point clouds of a lidar; the scene information determination module is further configured to perform time synchronization on the plurality of laser point clouds to obtain spliced point cloud data; determine scene parameter information from the spliced point cloud data; obtain the scene information according to the spliced point cloud data and the scene parameter information; the scene information includes the speed magnitude and direction of a scene object; The annotation module is further configured to: determine whether the speed magnitude and direction of the scene object satisfy a preset state threshold; if the speed magnitude and direction of the scene object satisfy the preset state threshold, respectively annotate the variety of scene information corresponding to each of the preset scene elements in the driving data; The driving data of the vehicle is further determined through image information; the annotation module is further configured to: determine the scene information indicated by the target scene element based on the image information; wherein, the scene information includes scene color information and scene shape information; classify the scene information according to the scene color information to obtain a classification result; determine the scene shape information based on the classification result; when the classification result and the scene shape information satisfy a preset annotation condition, respectively annotate the variety of scene information corresponding to each of the preset scene elements in the driving data.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method for processing driving data according to any one of claims 1 to 5.

Images