Driving data generation method for training decision planning model and related device

Abstract

The invention discloses a driving data generation method for training a decision planning model and a related device, and the method comprises the steps: building a simulation scene library, the simulation scene library comprises an expert driving data packet set and a first driving data packet set which is recorded in a real vehicle automatic driving process based on a first decision planning model and does not meet the driving requirement of a target driving scene; at least one of a second driving data packet set obtained after other vehicle information of each driving data packet in the expert driving data packet set is deleted and a third driving data packet set which is screened out from the first training sample set and does not meet the driving requirement of the target driving scene; performing automatic driving simulation by using the first decision planning model and the simulation scene library to obtain an abnormal data packet set; and performing automatic driving simulation based on the decision planning rule and the abnormal data packet set, and obtaining a target data packet set after the driving data which does not meet the driving requirement of the target driving scene in the abnormal data packet set is corrected.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a driving data generation method and related devices for training a decision-making planning model. Background Art

[0002] Autonomous driving technology refers to the technology that enables cars to drive autonomously through computer vision, sensors, control systems and other technical means. It can help vehicles perceive the surrounding environment information, including road conditions, traffic signals, other vehicles and pedestrians, and then automatically adjust the driving direction, speed, distance and safety distance parameters based on this information, so that the vehicle can drive more safely.

[0003] AI (Artificial Intelligence) is one of the key factors in realizing autonomous driving technology. Vehicles must be able to perceive, analyze and predict various road conditions in order to make the best decisions and ensure driving safety. In order to achieve automatic decision planning, it is usually necessary to collect driving data of experts in various driving scenarios as training sample sets for model training to obtain a decision planning model. However, if you want to obtain a high-quality decision planning model, you may need to upgrade multiple versions. Each version requires re-finding experts with better driving skills to specialize in driving scenarios where the model does not work well, and using the re-collected driving data to upgrade the previous version of the decision planning model, which requires high labor and time costs. Summary of the invention

[0004] The present application provides a driving data generation method and related devices for training a decision planning model, which can reduce the labor cost and time cost of obtaining high-quality driving data.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating driving data for training a decision planning model, the method comprising:

[0007] Establishing a simulation scenario library, wherein the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set, wherein the first driving data packet set includes driving data packets that do not meet the driving requirements of a target driving scenario and are recorded during the automatic driving of a real vehicle based on a first decision planning model, the second driving data packet set includes driving data packets after deleting other vehicle information from each driving data packet in the expert driving data packet set, the third driving data packet set includes driving data packets that do not meet the driving requirements of the target driving scenario and are screened from a first training sample set, and the first decision planning model is a decision planning model trained using the first training sample set;

[0008] Performing an autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, wherein the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scenario;

[0009] An autonomous driving simulation is performed based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, wherein the target data packet set includes a driving data packet after correcting the driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

[0010] It can be seen from the above scheme that the embodiment of the present application can first use at least one of the expert driving data packet set, the first driving data packet set that does not meet the driving requirements of the target driving scenario recorded during the automatic driving of the actual vehicle based on the first decision planning model, the second driving data packet set after deleting the other vehicle information of each driving data packet in the expert driving data packet set, and the third driving data packet set that does not meet the driving requirements of the target driving scenario screened out from the first training sample set to establish a simulation scenario library, and then use the first decision planning model and the simulation scenario library to perform automatic driving simulation to obtain an abnormal data packet set, and finally perform automatic driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, so as to use the target data packet set to correct the first decision planning model and obtain a higher quality second decision planning model. It can be seen from this that when upgrading the old version of the decision planning model, the embodiment of the present application does not look for experts with better driving skills to perform manual driving, but instead uses the collected driving data packets to screen abnormal driving data packets, and uses the traditional rule algorithm (i.e., decision planning rules) to simulate the abnormal driving data packets to correct the abnormal driving data packets, and obtain simulated high-quality driving data packets for optimizing the deep learning model (i.e., decision planning model). The entire process does not require manual time, thereby reducing the labor cost and time cost of obtaining high-quality driving data.

[0011] In a first possible implementation manner of the first aspect, after performing autonomous driving simulation based on the decision planning rule and the abnormal data packet set to obtain the target data packet set, the method further includes:

[0012] The target data packet set is converted into a second training sample set that conforms to a model training format, so as to train the first decision planning model using the second training sample set to obtain a second decision planning model.

[0013] It can be seen from the above scheme that the embodiment of the present application performs autonomous driving simulation based on decision planning rules and abnormal data packet sets. After obtaining the target data packet set, the target data packet set is first converted into a second training sample set that conforms to the model training format, and then the first decision planning model is trained using the second training sample set to obtain the second decision planning model, thereby improving the training efficiency of the first decision planning model.

[0014] In a second possible implementation manner of the first aspect, when the target driving scenario includes cruise control, before training the first decision planning model using the second training sample set to obtain the second decision planning model, the method further includes:

[0015] Filtering driving data that does not meet the cruise control driving requirement in the second training sample set;

[0016] The driving data that does not meet the cruise control driving requirement includes at least one of the following:

[0017] There is a diverging flow scenario in the two adjacent lanes of the ego vehicle;

[0018] There is another vehicle within a preset distance in front of the vehicle;

[0019] The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold;

[0020] The difference between the speed of the vehicle ahead of the own vehicle in the lane and the speed of the own vehicle is less than a preset speed threshold.

[0021] It can be seen from the above scheme that the embodiment of the present application can improve the accuracy of the decision planning model for the cruise control scenario by filtering the driving data in the second training sample set that will have a collision risk when accelerating before using the second training sample set to train the first decision planning model.

[0022] In a third possible implementation manner of the first aspect, each of the driving data packets in the simulation scenario library includes map information, and the method for acquiring the map information includes:

[0023] For each driving data packet without map information added, input data of a landmark perception model is obtained from the driving data packet without map information added, wherein the input data includes a road environment image at a target time and a local posture at the target time, the local posture includes an offset of a global posture of the vehicle when the road environment image is collected relative to a global posture at a target starting point, the target time is any time when the vehicle collects the road environment image on an extended navigation path, and the extended navigation path includes an extended path of the vehicle navigation path;

[0024] Processing the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information;

[0025] Based on the landmark information included in each of the road environment images in the driving data packet without map information added thereto, map information corresponding to the driving data packet without map information added thereto is generated.

[0026] It can be seen from the above scheme that the embodiment of the present application can not only automatically perceive the landmark information contained in the road environment image at any time based on the pre-trained landmark perception model, but also the input data of the landmark perception model only includes the road environment image and local posture, and has nothing to do with the high-precision map. Therefore, local map information composed of landmark information can be obtained without getting rid of the high-precision map.

[0027] In a fourth possible implementation manner of the first aspect, when the target driving scenario includes cruise control, performing the autonomous driving simulation based on the decision planning rule and the abnormal data packet set to obtain the target data packet set includes:

[0028] Performing an autonomous driving simulation based on the decision-making planning rule and the abnormal data packet set to obtain a driving trajectory that meets the cruise control driving requirements;

[0029] The driving trajectory that meets the cruise control driving requirement is used to replace the corresponding driving trajectory that does not meet the cruise control driving requirement in the abnormal data packet set to obtain the target data packet set.

[0030] In a second aspect, an embodiment of the present application provides a driving data generating device for training a decision planning model, the device comprising:

[0031] an establishing unit, configured to establish a simulation scenario library, wherein the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set, wherein the first driving data packet set includes driving data packets that do not meet driving requirements of a target driving scenario and are recorded during an actual vehicle automatic driving process based on a first decision planning model, the second driving data packet set includes driving data packets after other vehicle information of each driving data packet in the expert driving data packet set is deleted, the third driving data packet set includes driving data packets that do not meet driving requirements of the target driving scenario and are screened from a first training sample set, and the first decision planning model is a decision planning model trained using the first training sample set;

[0032] A first simulation unit is used to perform autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, wherein the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scenario;

[0033] The second simulation unit is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, wherein the target data packet set includes a driving data packet after correcting the driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

[0034] In a first possible implementation manner of the second aspect, the apparatus further includes:

[0035] A conversion unit is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, and then convert the target data packet set into a second training sample set that conforms to the model training format, so as to use the second training sample set to train the first decision planning model and obtain a second decision planning model.

[0036] In a second possible implementation manner of the second aspect, the device further includes:

[0037] a filtering unit, configured to filter driving data that does not meet the cruise control driving requirement in the second training sample set before training the first decision planning model using the second training sample set to obtain the second decision planning model when the target driving scenario includes cruise control;

[0038] The driving data that does not meet the cruise control driving requirement includes at least one of the following:

[0039] There is a diverging flow scenario in the two adjacent lanes of the ego vehicle;

[0040] There is another vehicle within a preset distance in front of the vehicle;

[0041] The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold;

[0042] The difference between the speed of the vehicle ahead of the own vehicle in the lane and the speed of the own vehicle is less than a preset speed threshold.

[0043] In a third possible implementation manner of the second aspect, the device further includes:

[0044] an acquisition unit, for acquiring input data of a landmark perception model from the driving data packet without adding map information for each driving data packet without adding map information when each driving data packet in the simulation scenario library includes map information, wherein the input data includes a road environment image at a target time and a local posture at the target time, the local posture includes an offset of a global posture of the ego vehicle when collecting the road environment image relative to a global posture at a target starting point, the target time is any time when the ego vehicle collects the road environment image on an extended navigation path, and the extended navigation path includes an extended path of a vehicle-mounted navigation path;

[0045] A perception unit, configured to process the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information;

[0046] A generating unit is used to generate map information corresponding to the driving data packet without map information added based on the landmark information contained in each of the road environment images in the driving data packet without map information added.

[0047] In a fourth possible implementation manner of the second aspect, the second simulation unit includes:

[0048] A simulation module, for performing an automatic driving simulation based on the decision planning rule and the abnormal data packet set when the target driving scenario includes cruise control, so as to obtain a driving trajectory that meets the cruise control driving requirement;

[0049] The replacement module is used to replace the driving trajectory that does not meet the cruise control driving requirement in the abnormal data packet set with the driving trajectory that meets the cruise control driving requirement to obtain the target data packet set.

[0050] It can be seen from the above scheme that the embodiment of the present application can first use at least one of the expert driving data packet set, the first driving data packet set that does not meet the driving requirements of the target driving scenario recorded during the automatic driving of the actual vehicle based on the first decision planning model, the second driving data packet set after deleting the other vehicle information of each driving data packet in the expert driving data packet set, and the third driving data packet set that does not meet the driving requirements of the target driving scenario screened out from the first training sample set to establish a simulation scenario library, and then use the first decision planning model and the simulation scenario library to perform automatic driving simulation to obtain an abnormal data packet set, and finally perform automatic driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, so as to use the target data packet set to correct the first decision planning model and obtain a higher quality second decision planning model. It can be seen from this that when upgrading the old version of the decision planning model, the embodiment of the present application does not look for experts with better driving skills to perform manual driving, but instead uses the collected driving data packets to screen abnormal driving data packets, and uses the traditional rule algorithm (i.e., decision planning rules) to simulate the abnormal driving data packets to correct the abnormal driving data packets, and obtain simulated high-quality driving data packets for optimizing the deep learning model (i.e., decision planning model). The entire process does not require manual time, thereby reducing the labor cost and time cost of obtaining high-quality driving data.

[0051] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any possible implementation manner of the first aspect.

[0052] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0053] one or more processors;

[0054] The processor is coupled to a storage device, and the storage device is used to store one or more programs;

[0055] When one or more programs are executed by one or more processors, the electronic device implements the method described in any possible implementation manner of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a vehicle, which includes a device as described in any implementation of the second aspect, or includes an electronic device as described in the fourth aspect.

[0057] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the method described in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0059] Figure 1 A flowchart of a method for generating driving data for training a decision planning model provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of another method for generating driving data for training a decision planning model provided in an embodiment of the present application;

[0061] Figure 3 A driving data generation system architecture diagram for training a decision planning model provided in an embodiment of the present application;

[0062] Figure 4 A block diagram of a driving data device for training a decision planning model provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of the structure of an electronic device or computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of the present application and the accompanying drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0066] Figure 1The present invention is a flowchart of a method for generating driving data for training a decision planning model. The method can be applied to an electronic device or a computer device, and can be applied to a terminal or a server. Specifically, the method can include the following steps:

[0067] S110: Establish a simulation scenario library.

[0068] Among them, Figure 2 As shown, the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set. The expert driving data packet set includes a driving data packet generated by an expert driving vehicle. The first driving data packet set includes driving data packets that do not meet the driving requirements of the target driving scenario and are recorded during the automatic driving of the real vehicle based on the first decision planning model. For example, when the target driving scenario includes cruise control, the first driving data packet set includes driving data packets that do not reach the speed limit. The second driving data packet set includes driving data packets after deleting the other vehicle information of each driving data packet in the expert driving data packet set.

[0069] The third driving data packet set includes driving data packets that are screened out from the first training sample set and do not meet the driving requirements of the target driving scenario, and the first decision planning model is a decision planning model trained using the first training sample set. The first training sample set is a large-scale set of driving data packets collected for training the decision planning model. After the first decision planning model is obtained by training based on the first training sample set, the first training sample set can be input into the first decision planning model again in an offline manner to obtain driving data packets that do not meet the driving requirements of the target driving scenario. The so-called offline method refers to simply inputting data into the first decision planning model to obtain model output data without simulating the driving scenario or running the actual vehicle.

[0070] The target driving scenarios include cruise control, no parking on highways, overtaking and lane changing, and other scenarios that exist in vehicle driving.

[0071] S120: Perform autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set.

[0072] Among them, the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scene, and each data packet that does not meet the driving requirements of the target driving scene can be called an abnormal driving data packet. After establishing the simulation scenario library, the first decision planning model obtained by pre-training and the simulation scenario library can be used to perform automatic driving simulation, that is, in the simulation of various driving scenarios, the first decision planning model is used to perform automatic driving, so as to obtain a data packet set that does not meet the driving requirements of the target driving scene. For example, a cruise control scene can be simulated, and the first decision planning model can be used to perform cruise control to determine whether the driving trajectory corresponding to each driving data packet can reach the speed limit value and drive stably at the speed limit value. If not, the driving data packet can be determined as an abnormal driving data packet. For another example, a high-speed parking prohibition scene can also be simulated, and the first decision planning model can be used for automatic driving to determine whether the driving trajectory corresponding to each driving data packet has a high-speed parking situation. If so, the driving data packet is determined to be an abnormal driving data packet.

[0073] S130: Perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set.

[0074] The target data packet set includes driving data packets obtained by correcting driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

[0075] In the related art, decision planning can be performed in a manner based on matching decision planning rules based on autonomous driving. "Decision planning rules" refer to traditional planning algorithms based on optimization, which are rules pre-set one by one based on driving experience. During autonomous driving, decision planning can be performed in a manner that matches the decision planning rules, and the pre-set decision planning rules themselves will not change as the actual driving scenes are automatically learned. Therefore, the decision planning rules are different from the planning algorithms based on deep learning, that is, they are different from the decision planning models trained based on driving data in the embodiments of the present application. The embodiments of the present application can simulate autonomous driving on an abnormal data packet set based on the decision planning rules, using the decision planning rules as the learning standard, and by continuously adjusting the abnormal driving data (such as driving trajectory, driving speed, etc.) in the abnormal data packet set, so that the adjusted abnormal driving data meets the decision planning rules, thereby achieving a correction effect on the abnormal driving data.

[0076] In an embodiment of the present application, an autonomous driving simulation is performed based on decision planning rules and an exception data packet set to obtain a driving trajectory that meets the driving requirements of a target driving scenario; the driving trajectory that meets the driving requirements of the target driving scenario replaces the corresponding driving trajectory in the exception data packet set that does not meet the driving requirements of the target driving scenario to obtain a target data packet set.

[0077] When the target driving scenario includes cruise control, an automatic driving simulation is performed based on the decision-making planning rules and the abnormal data packet set to obtain a driving trajectory that meets the cruise control driving requirements; the driving trajectory that meets the cruise control driving requirements replaces the corresponding driving trajectory in the abnormal data packet set that does not meet the cruise control driving requirements to obtain the target data packet set. When the target driving scenario includes no parking on highways, an automatic driving simulation is performed based on the decision-making planning rules and the abnormal data packet set to obtain a driving trajectory that meets the no parking on highways; the driving trajectory that meets the no parking on highways replaces the corresponding driving trajectory in the abnormal data packet set that does not meet the no parking on highways to obtain the target data packet set.

[0078] The driving data generation method for training a decision planning model provided in an embodiment of the present application can first establish a simulation scenario library using at least one of an expert driving data packet set, a first driving data packet set that does not meet the driving requirements of a target driving scenario and is recorded during the automatic driving of a real vehicle based on a first decision planning model, a second driving data packet set after deleting other vehicle information from each driving data packet in the expert driving data packet set, and a third driving data packet set that does not meet the driving requirements of the target driving scenario and is screened out from a first training sample set. Then, the first decision planning model and the simulation scenario library are used to perform automatic driving simulation to obtain an abnormal data packet set. Finally, the automatic driving simulation is performed based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, so as to use the target data packet set to correct the first decision planning model and obtain a higher quality second decision planning model. It can be seen from this that when upgrading the old version of the decision planning model, the embodiment of the present application does not look for experts with better driving skills to perform manual driving, but instead uses the collected driving data packets to screen abnormal driving data packets, and uses the traditional rule algorithm (i.e., decision planning rules) to simulate the abnormal driving data packets to correct the abnormal driving data packets, and obtain simulated high-quality driving data packets for optimizing the deep learning model (i.e., decision planning model). The entire process does not require manual time, thereby reducing the labor cost and time cost of obtaining high-quality driving data.

[0079] In one embodiment, if Figure 2 As shown, in order to facilitate the training of the first decision planning model and improve the training efficiency of the first decision planning model, after the automatic driving simulation is performed based on the decision planning rules and the abnormal data packet set to obtain the target data packet set, the target data packet set can be first converted into a second training sample set that conforms to the model training format, and then the first decision planning model is trained using the second training sample set to obtain the second decision planning model. For example, each file in the target data packet set can be converted from the .bag format to the .npz format.

[0080] In addition, when there is no map information in the target data packet set, corresponding map information can also be added, such as adding high-precision maps and adding local maps according to image perception methods.

[0081] In the embodiment of the present application, the map information can be added to the target data packet set after the target data packet set is obtained, or the map information can be added when the simulation scenario library is established.

[0082] In one embodiment, the process of obtaining map information according to the image perception method includes steps A1-A3:

[0083] A1: For each driving data packet without map information added, the input data of the landmark perception model is obtained from the driving data packet without map information added.

[0084] Among them, the input data includes the road environment image at the target time and the local posture at the target time. The local posture includes the offset of the global posture of the vehicle when collecting the road environment image relative to the global posture at the target starting point. The target time is any time when the vehicle collects the road environment image on the extended navigation path. The extended navigation path includes the extended path of the vehicle navigation path.

[0085] The pose includes the vehicle position and the vehicle attitude. The global pose includes the vehicle position and the vehicle attitude based on global positioning. When obtaining the global pose, there is no need for high-precision positioning such as RTK (Real-time kinematic, carrier phase differential technology), and only road-level positioning is required, such as navigation positioning. Global positioning can use the earth as a reference system for positioning, that is, the coordinate system of global positioning can be a geodetic coordinate system. The coordinate system where the local pose is located can be a boot coordinate system. In the boot coordinate system, the position point where the vehicle is powered on is the origin of the coordinate system, so the coordinate system is also called the power-on coordinate system. In addition, the local pose can also be positioned based on the navigation and positioning system without the need for high-precision positioning. The target starting point can be any specified position, such as the position when the vehicle is powered on, or any specified position during the driving process of the vehicle.

[0086] When the user enters the starting point (or the default current location) and the end point in the vehicle navigation software, the vehicle navigation software will generate a vehicle navigation path based on the starting point and the end point, and the navigation positioning system will locate the global position of the vehicle in real time. The vehicle navigation software can be software on the vehicle computer or software on a mobile terminal that communicates with the vehicle computer. Therefore, the vehicle navigation path can be a path generated by the vehicle navigation software on the vehicle computer or a path generated by the vehicle navigation software on a mobile terminal that communicates with the vehicle computer. The embodiment of the present application does not limit the source of the vehicle navigation path.

[0087] The electronic device or computer device can first obtain the vehicle navigation path and the global posture of the vehicle when traveling on the vehicle navigation path; then extract POI (Point of Interest) information corresponding to the global posture from the target data; finally, add the POI information to the vehicle navigation path to obtain the extended navigation path.

[0088] The target data includes navigation events and / or in-vehicle navigation maps. Navigation events refer to events reported in the in-vehicle navigation system, such as a ramp about to be entered 1 km ahead, a confluence point 1 km ahead, etc. POI information includes the number of lanes, confluence points, speed limit information, long solid lines, etc. The POI information corresponding to the global posture is the POI information within a preset distance range ahead of the global posture on the in-vehicle navigation path.

[0089] A2: Process the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information.

[0090] The landmark perception model can be a generative model or other neural network models. The method for generating the landmark perception model includes: obtaining a landmark training sample set, wherein each training sample in the landmark training sample set includes: a road environment sample image group, a local pose corresponding to each frame of the road environment sample image in the road environment sample image group, and a landmark true value corresponding to each frame of the road environment sample image; using the landmark training sample set to perform training to obtain a landmark perception model, the landmark perception model is used to perceive and output landmark information in the road environment sample image.

[0091] Among them, the road environment image group includes multiple frames of continuous road environment sample images. The "continuous" here not only includes the literal meaning of continuous, but also includes that the total number of frames of road environment sample images collected within a predetermined time period exceeds a preset value.

[0092] A3: Generate map information corresponding to the driving data package to which map information is not added based on the landmark information included in each road environment image in the driving data package to which map information is not added.

[0093] The landmark information obtained from the road environment image is not the landmark information marked in the road environment image, but the landmark information extracted from the road environment image, and the obtained landmark information is the landmark information that retains the geometric shape and the relative position relationship between each landmark, and its display effect is similar to a simple map. Therefore, the landmark information contained in each road environment image in the driving data package without adding map information can be directly used as the corresponding map information.

[0094] The embodiments of the present application can not only automatically perceive the landmark information contained in the road environment image at any time based on the pre-trained landmark perception model, but also the input data of the landmark perception model only includes the road environment image and the local posture, and has nothing to do with the high-precision map. Therefore, local map information composed of landmark information can be obtained without getting rid of the high-precision map.

[0095] In one embodiment, if Figure 2 As shown, when the target driving scenario includes cruise control, before the first decision planning model is trained using the second training sample set to obtain the second decision planning model, the method further includes: filtering driving data in the second training sample set that does not meet the cruise control driving requirements;

[0096] The driving data that does not meet the cruise control driving requirements includes at least one of the following:

[0097] There is a diverging flow scenario in the two adjacent lanes of the ego vehicle;

[0098] There are other vehicles within a preset distance in front of the vehicle;

[0099] The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold;

[0100] The difference between the speed of the vehicle ahead in the lane and the speed of the vehicle itself is less than a preset speed threshold.

[0101] The values ​​of the preset distance range, preset time interval threshold and preset vehicle speed threshold can be determined based on actual experience. For example, the preset distance range is 80m, the preset time interval threshold is 2s, and the preset vehicle speed threshold is -5km / h.

[0102] When the distance between the vehicle in front of the vehicle is too long, it will not affect the driving speed of the vehicle at all. Therefore, the judgment standard for the vehicle in front of the vehicle can be that the lateral distance from the projection of the lane line of the vehicle is less than 3m.

[0103] In addition, for other target driving scenarios, unsafe driving data in the second training sample set may also be filtered out before model training.

[0104] The embodiment of the present application can improve the accuracy of the decision planning model for the cruise control scenario by filtering the driving data with acceleration and collision risk in the second training sample set before using the second training sample set to train the first decision planning model.

[0105] In one embodiment, in order to prevent the decision planning model from directly learning to deduce the future trajectory based on the speed, resulting in overfitting of the speed, the speed input to the decision planning model has a 50% probability of being enhanced and disturbed. However, this will cause the model input speed to fluctuate, and the model output trajectory cannot be stabilized at the speed limit. In this case, the input speed can be not enhanced when the vehicle speed is greater than the preset speed threshold, and the real speed is used to avoid introducing randomness. The preset speed threshold can be 90km / h.

[0106] In one implementation, in order to implement the above-mentioned driving data generation method for training a decision planning model, another embodiment of the present application provides a driving data generation system for training a decision planning model, such as Figure 3 As shown, the system includes a client, an API (Application Programming Interface), a pipeline configuration, a pipeline server, a pipeline controller, a pipeline UI (User Interface), an additional data server, a data interaction interface, a task server, container orchestration and resource scheduling Kubernetes, underlying containers, object storage, CPU (Central Processing Unit) / GPU (Graphics Processing Unit), TCP (Transmission Control Protocol) / RDMA (Remote Direct Memory Access), NVMe (Non-Volatile Memory express, NVMe storage system) / SSD (Solid State Drives, solid state drives) / HDD (Hard Disk Drive, hard disk drives).

[0107] Among them, users can interact with the server through the client and API, and the server can include Figure 3 The architecture except for the client and API. Pipeline refers to the automated process of the driving data generation method for training the decision planning model. The tasks served by the task server include simulation, format conversion, filtering logic, and model training. In other words, the pipeline includes four stages: simulation, format conversion, filtering logic, and model training.

[0108] The pipeline configuration includes defining a set of configuration parameters that meet the various tasks in the pipeline through unified simulation, format conversion, filtering logic, and model training parameter templates.

[0109] The pipeline server is used to provide client tools or APIs to receive pipeline requests submitted by users for production tasks.

[0110] The pipeline controller is used to define the order of each task in the pipeline, flexibly connect the tasks in the entire data production process, and realize data production automation.

[0111] Pipeline UI is used to output the current pipeline operation status and logs through the UI.

[0112] The additional data server is used to maintain the metadata of each task, submit resource operation status, and facilitate the tracing of historical task information.

[0113] The data interaction interface is used to transfer meta-information between tasks, for example, transferring the simulation task identifier to the format conversion task.

[0114] Kubernetes is used to host the resource scheduling and container environment for a batch of tasks and unify the task scheduling method.

[0115] Object storage is used to manage the processing of simulation, format conversion, filtering logic, and model training, and to coordinate the interactions between various tasks.

[0116] CPU / GPU, TCP / RDMA, NVMe / SSD / HDD are the underlying support such as processor, memory, and protocol required for pipeline processing.

[0117] Corresponding to the above method embodiment, another embodiment of the present application provides a driving data generating device for training a decision planning model, such as Figure 4 As shown, the device comprises:

[0118] The establishment unit 210 is used to establish a simulation scenario library, wherein the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set, wherein the first driving data packet set includes driving data packets that do not meet the driving requirements of the target driving scenario and are recorded during the automatic driving of the real vehicle based on the first decision planning model, the second driving data packet set includes driving data packets after deleting other vehicle information from each driving data packet in the expert driving data packet set, the third driving data packet set includes driving data packets that do not meet the driving requirements of the target driving scenario and are screened from the first training sample set, and the first decision planning model is a decision planning model trained using the first training sample set;

[0119] A first simulation unit 220 is used to perform an autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, wherein the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scenario;

[0120] The second simulation unit 230 is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, wherein the target data packet set includes a driving data packet after correcting the driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

[0121] In a possible implementation manner, the device further includes:

[0122] A conversion unit is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, and then convert the target data packet set into a second training sample set that conforms to the model training format, so as to use the second training sample set to train the first decision planning model and obtain a second decision planning model.

[0123] In a possible implementation manner, the device further includes:

[0124] a filtering unit, configured to filter driving data that does not meet the cruise control driving requirement in the second training sample set before training the first decision planning model using the second training sample set to obtain the second decision planning model when the target driving scenario includes cruise control;

[0125] The driving data that does not meet the cruise control driving requirement includes at least one of the following:

[0126] There is a diverging flow scenario in the two adjacent lanes of the ego vehicle;

[0127] There is another vehicle within a preset distance in front of the vehicle;

[0128] The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold;

[0129] The difference between the speed of the vehicle ahead of the own vehicle in the lane and the speed of the own vehicle is less than a preset speed threshold.

[0130] In a possible implementation manner, the device further includes:

[0131] an acquisition unit, for acquiring input data of a landmark perception model from the driving data packet without adding map information for each driving data packet without adding map information when each driving data packet in the simulation scenario library includes map information, wherein the input data includes a road environment image at a target time and a local posture at the target time, the local posture includes an offset of a global posture of the ego vehicle when collecting the road environment image relative to a global posture at a target starting point, the target time is any time when the ego vehicle collects the road environment image on an extended navigation path, and the extended navigation path includes an extended path of a vehicle-mounted navigation path;

[0132] A perception unit, configured to process the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information;

[0133] A generating unit is used to generate map information corresponding to the driving data packet without map information added based on the landmark information contained in each of the road environment images in the driving data packet without map information added.

[0134] In a possible implementation, the second simulation unit 230 includes:

[0135] A simulation module, for performing an automatic driving simulation based on the decision planning rule and the abnormal data packet set when the target driving scenario includes cruise control, so as to obtain a driving trajectory that meets the cruise control driving requirement;

[0136] The replacement module is used to replace the driving trajectory that does not meet the cruise control driving requirement in the abnormal data packet set with the driving trajectory that meets the cruise control driving requirement to obtain the target data packet set.

[0137] The driving data generating device for training a decision planning model provided in an embodiment of the present application can first establish a simulation scenario library using at least one of an expert driving data packet set, a first driving data packet set that does not meet the driving requirements of a target driving scenario and is recorded during the automatic driving of a real vehicle based on a first decision planning model, a second driving data packet set after deleting other vehicle information from each driving data packet in the expert driving data packet set, and a third driving data packet set that does not meet the driving requirements of the target driving scenario and is screened out from a first training sample set, and then perform automatic driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, and finally perform automatic driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, so as to correct the first decision planning model using the target data packet set to obtain a higher quality second decision planning model. It can be seen from this that when upgrading the old version of the decision planning model, the embodiment of the present application does not look for experts with better driving skills to perform manual driving, but instead uses the collected driving data packets to screen abnormal driving data packets, and uses the traditional rule algorithm (i.e., decision planning rules) to simulate the abnormal driving data packets to correct the abnormal driving data packets, and obtain simulated high-quality driving data packets for optimizing the deep learning model (i.e., decision planning model). The entire process does not require manual time, thereby reducing the labor cost and time cost of obtaining high-quality driving data.

[0138] Based on the above method embodiments, another embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0139] Based on the above method embodiment, another embodiment of the present application provides an electronic device or a computer device, such as Figure 5 As shown, including:

[0140] One or more processors 310;

[0141] The processor 310 is coupled to a storage device 320, and the storage device 320 is used to store one or more programs;

[0142] When the one or more programs are executed by the one or more processors 310, the electronic device or computer device implements the method described in any of the above embodiments.

[0143] Based on the above embodiments, an application embodiment provides a vehicle, which includes a device as described in any of the above embodiments, or includes the electronic device as described above.

[0144] Based on the above embodiments, another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the method described in any of the above embodiments.

[0145] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here. It can be understood by those skilled in the art that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present application.

[0146] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating driving data for training a decision planning model, characterized in that: The method comprises: Establishing a simulation scenario library, wherein the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set, wherein the first driving data packet set includes driving data packets that do not meet the driving requirements of a target driving scenario and are recorded during the automatic driving of a real vehicle based on a first decision planning model, the second driving data packet set includes driving data packets after deleting other vehicle information from each driving data packet in the expert driving data packet set, the third driving data packet set includes driving data packets that do not meet the driving requirements of the target driving scenario and are screened from a first training sample set, and the first decision planning model is a decision planning model trained using the first training sample set; Performing an autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, wherein the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scenario; An autonomous driving simulation is performed based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, wherein the target data packet set includes a driving data packet after correcting the driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

2. The method according to claim 1, characterized in that After performing autonomous driving simulation based on the decision planning rule and the abnormal data packet set to obtain the target data packet set, the method further includes: The target data packet set is converted into a second training sample set that conforms to a model training format, so as to train the first decision planning model using the second training sample set to obtain a second decision planning model.

3. The method according to claim 2, characterized in that When the target driving scenario includes cruise control, before training the first decision planning model using the second training sample set to obtain the second decision planning model, the method further includes: Filtering driving data that does not meet the cruise control driving requirement in the second training sample set; The driving data that does not meet the cruise control driving requirement includes at least one of the following: There is a diverging flow scenario in the two adjacent lanes of the ego vehicle; There is another vehicle within a preset distance in front of the vehicle; The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold; The difference between the speed of the vehicle ahead of the own vehicle in the lane and the speed of the own vehicle is less than a preset speed threshold.

4. The method according to claim 1, characterized in that: Each of the driving data packets in the simulation scenario library includes map information, and the method for obtaining the map information includes: For each driving data packet without map information added, input data of a landmark perception model is obtained from the driving data packet without map information added, wherein the input data includes a road environment image at a target time and a local posture at the target time, the local posture includes an offset of a global posture of the vehicle when the road environment image is collected relative to a global posture at a target starting point, the target time is any time when the vehicle collects the road environment image on an extended navigation path, and the extended navigation path includes an extended path of the vehicle navigation path; Processing the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information; Based on the landmark information included in each of the road environment images in the driving data packet without map information added thereto, map information corresponding to the driving data packet without map information added thereto is generated.

5. The method according to any one of claims 1 to 4, characterized in that When the target driving scenario includes cruise control, the autonomous driving simulation is performed based on the decision planning rule and the abnormal data packet set to obtain the target data packet set, including: Performing an autonomous driving simulation based on the decision-making planning rule and the abnormal data packet set to obtain a driving trajectory that meets the cruise control driving requirements; The driving trajectory that meets the cruise control driving requirement is used to replace the corresponding driving trajectory that does not meet the cruise control driving requirement in the abnormal data packet set to obtain the target data packet set.

6. A driving data generating device for training a decision planning model, characterized in that: The device comprises: an establishing unit, configured to establish a simulation scenario library, wherein the simulation scenario library includes at least one of an expert driving data packet set, a first driving data packet set, a second driving data packet set, and a third driving data packet set, wherein the first driving data packet set includes driving data packets that do not meet driving requirements of a target driving scenario and are recorded during an actual vehicle automatic driving process based on a first decision planning model, the second driving data packet set includes driving data packets after other vehicle information of each driving data packet in the expert driving data packet set is deleted, the third driving data packet set includes driving data packets that do not meet driving requirements of the target driving scenario and are screened from a first training sample set, and the first decision planning model is a decision planning model trained using the first training sample set; A first simulation unit is used to perform an autonomous driving simulation using the first decision planning model and the simulation scenario library to obtain an abnormal data packet set, wherein the abnormal data packet set is a data packet set that does not meet the driving requirements of the target driving scenario; The second simulation unit is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, wherein the target data packet set includes a driving data packet after correcting the driving data in the abnormal data packet set that does not meet the driving requirements of the target driving scenario.

7. The device according to claim 6, characterized in that The device also includes: A conversion unit is used to perform autonomous driving simulation based on the decision planning rules and the abnormal data packet set to obtain a target data packet set, and then convert the target data packet set into a second training sample set that conforms to the model training format, so as to use the second training sample set to train the first decision planning model and obtain a second decision planning model.

8. The device according to claim 7, characterized in that The device also includes: a filtering unit, configured to filter driving data that does not meet the cruise control driving requirement in the second training sample set before training the first decision planning model using the second training sample set to obtain the second decision planning model when the target driving scenario includes cruise control; The driving data that does not meet the cruise control driving requirement includes at least one of the following: There is a diverging flow scenario in the two adjacent lanes of the ego vehicle; There is another vehicle within a preset distance in front of the vehicle; The headway between the vehicle and the vehicle ahead in the lane is less than a preset headway threshold; The difference between the speed of the vehicle ahead of the own vehicle in the lane and the speed of the own vehicle is less than a preset speed threshold.

9. The device according to claim 6, characterized in that The device also includes: an acquisition unit, for acquiring input data of a landmark perception model from the driving data packet without adding map information for each driving data packet without adding map information when each driving data packet in the simulation scenario library includes map information, wherein the input data includes a road environment image at a target time and a local pose at the target time, the local pose includes an offset of a global pose of the ego vehicle when collecting the road environment image relative to a global pose at a target starting point, the target time is any time when the ego vehicle collects the road environment image on an extended navigation path, and the extended navigation path includes an extended path of a vehicle-mounted navigation path; A perception unit, configured to process the input data based on the landmark perception model to obtain the landmark information contained in the road environment image at the target time in the driving data packet without adding map information; A generating unit is used to generate map information corresponding to the driving data packet without map information added based on the landmark information contained in each of the road environment images in the driving data packet without map information added.

10. The device according to any one of claims 6 to 9, characterized in that: The second simulation unit comprises: A simulation module, for performing an automatic driving simulation based on the decision planning rule and the abnormal data packet set when the target driving scenario includes cruise control, so as to obtain a driving trajectory that meets the cruise control driving requirement; The replacement module is used to replace the driving trajectory that does not meet the cruise control driving requirement in the abnormal data packet set with the driving trajectory that meets the cruise control driving requirement to obtain the target data packet set.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; The processor is coupled to a storage device, and the storage device is used to store one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the method according to any one of claims 1 to 5.

13. A vehicle, characterized in that: The vehicle comprises the device according to any one of claims 6 to 10, or comprises the electronic device according to claim 12.

Images