Automatic driving simulation scene generation method and device, equipment and storage medium

Abstract

The invention discloses an automatic driving simulation scene generation method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a vehicle driving video frame; obtaining first speed data of the first vehicle according to the driving track data of the first vehicle in the vehicle driving video frame; obtaining first simulation speed data of the first simulation vehicle based on the first speed data; performing trajectory calculation according to the initial position data of the first simulation vehicle and the first simulation speed data to obtain scene trajectory data of the first simulation vehicle; acquiring initial position data and initial speed data of a second simulation vehicle; and generating an automatic driving simulation scene based on the scene trajectory data and the initial position data and the initial speed data of the second simulation vehicle. Therefore, the simulation trajectory is generated based on the real speed data, and the problem of trajectory data distortion is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving simulation test technology, and in particular to an autonomous driving simulation scene generation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of science and technology, smart vehicles controlled by autonomous driving systems are widely used in people's lives. In order to ensure the safety of smart vehicles, it is necessary to conduct a large number of tests on smart vehicles in autonomous driving simulation scenarios in order to evaluate the autonomous driving capabilities of smart vehicles.

[0003] In the related technologies, machine learning and other methods are mainly used to perform data repair and data generation based on the collected dynamic data set information to generate the corresponding autonomous driving simulation scene. However, there is a problem of trajectory distortion in the related technologies, resulting in the effect of the generated autonomous driving simulation scene to be optimized. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present invention proposes a method, device, equipment and storage medium for generating an autonomous driving simulation scene, which generates a simulation trajectory based on real speed data, effectively overcomes the problem of trajectory data distortion, and optimizes the generation effect of the simulation scene.

[0005] To achieve the above-mentioned purpose, a first aspect of the present invention proposes a method for generating an autonomous driving simulation scene, the method comprising: obtaining a vehicle driving video frame; wherein the vehicle driving video frame includes at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle; obtaining first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame; obtaining first simulation speed data of the first simulated vehicle based on the first speed data; performing trajectory calculation according to the starting position data of the first simulated vehicle and the first simulation speed data to obtain scene trajectory data of the first simulated vehicle; obtaining the starting position data and initial speed data of the second simulated vehicle; wherein the change in the driving state of the first simulated vehicle has an impact on the change in the driving state of the second simulated vehicle; and generating the autonomous driving simulation scene based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

[0006] According to one embodiment of the present invention, the first vehicle includes a first driving state and a second driving state in the vehicle driving video frame; the first speed data of the first vehicle is obtained according to the driving trajectory data of the first vehicle in the vehicle driving video frame, including: determining an interaction event point according to the second driving state; wherein the first driving state is adjacent to the second driving state, and the second driving state is located after the first driving state; determining a trajectory data time range according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point; determining the first trajectory data of the first vehicle in the vehicle driving video frame based on the trajectory data time range; wherein the driving trajectory data of the first vehicle includes the first trajectory data; and obtaining the first speed data according to the first trajectory data.

[0007] According to one embodiment of the present invention, the first vehicle is located in the first lane in the first driving state; the first vehicle is located in the second lane in the second driving state; when the first vehicle changes from the first driving state to the second driving state, the first vehicle drives from the first lane to the second lane; determining the interaction event point according to the second driving state includes: determining the interaction event point according to the moment when the first vehicle drives into the second lane.

[0008] According to one embodiment of the present invention, the first vehicle and the second vehicle are located in the same lane; the first vehicle has a speed state in the vehicle driving video frame; the first speed data of the first vehicle is obtained according to the driving trajectory data of the first vehicle in the vehicle driving video frame, including: if the first vehicle is away from the second vehicle in the speed state, the interaction event point is determined according to the time corresponding to the maximum speed data of the first vehicle in the speed state, or if the first vehicle is approaching the second vehicle in the speed state, the interaction event point is determined according to the time corresponding to the minimum speed data of the first vehicle in the speed state; the trajectory data time range is determined according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point; the second trajectory data of the first vehicle is determined in the vehicle driving video frame based on the trajectory data time range; wherein the driving trajectory data of the first vehicle includes the second trajectory data; the first speed data is obtained according to the second trajectory data.

[0009] According to one embodiment of the present invention, before obtaining the starting position data and initial speed data of the second simulated vehicle, the method also includes: obtaining second speed data of the second vehicle according to the driving trajectory data of the second vehicle in the vehicle driving video frame; obtaining second simulated speed data of the second simulated vehicle based on the second speed data; obtaining the initial speed data of the second simulated vehicle, including: obtaining the initial speed data of the second simulated vehicle according to the starting speed data of the second simulation speed data.

[0010] According to one embodiment of the present invention, the first speed data is the speed sample data of the first simulated vehicle; the second speed data is the speed sample data of the second simulated vehicle; the first simulation speed data is obtained based on the first speed data, and the second simulation speed data is obtained based on the second speed data, including: obtaining combined sample data based on the first speed data and the second speed data for training an initial data generation model to obtain a trained simulation data generation model; generating simulated vehicle data using the simulation data generation model; wherein the simulated vehicle data includes the first simulation speed data and the second simulation speed data.

[0011] According to one embodiment of the present invention, the driving trajectory data of the first vehicle is the trajectory sample data of the first simulated vehicle; the driving trajectory data of the second vehicle is the trajectory sample data of the second simulated vehicle; the combined sample data obtained based on the first speed data and the second speed data includes: combining the first speed data, the driving trajectory data of the first vehicle, the second speed data and the driving trajectory data of the second vehicle to obtain the combined sample data; wherein the simulated vehicle data also includes the first simulated trajectory data of the first simulated vehicle and the second simulated trajectory data of the second simulated vehicle; the starting position data of the first simulated vehicle is obtained according to the starting coordinate data of the first simulation trajectory data; the starting position data of the second simulated vehicle is obtained according to the starting coordinate data of the second simulation trajectory data.

[0012] To achieve the above-mentioned purpose, a second aspect of the present invention proposes an automatic driving simulation scene generation device, the device comprising: a video frame acquisition module, used to acquire a vehicle driving video frame; wherein the vehicle driving video frame includes at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle; a speed data acquisition module, used to obtain first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame; a simulation speed acquisition module, used to obtain first simulation speed data of the first simulated vehicle based on the first speed data; a scene trajectory acquisition module, used to perform trajectory calculation according to the starting position data of the first simulated vehicle and the first simulation speed data to obtain scene trajectory data of the first simulated vehicle; a simulation vehicle data acquisition module, used to acquire the starting position data and initial speed data of the second simulated vehicle; wherein the change in the driving state of the first simulated vehicle has an impact on the change in the driving state of the second simulated vehicle; and a simulation scene generation module, used to generate the automatic driving simulation scene based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

[0013] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for generating an autonomous driving simulation scene described in any of the aforementioned embodiments are implemented.

[0014] To achieve the above-mentioned objectives, the fourth aspect of the present invention proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating an autonomous driving simulation scene described in any of the aforementioned embodiments are implemented.

[0015] According to multiple embodiments provided by the present invention, corresponding simulated speed data is generated based on real speed data to further generate simulated trajectories. Since speed data can directly reflect the time dependence of vehicle motion, by analyzing the changes in speed characteristics, the dynamic changes of trajectory data can be better simulated, and therefore, the problem of trajectory data distortion can be effectively overcome. Generating simulated trajectories based on real speed data (rather than simply extrapolating based on position data or timestamps) can more accurately simulate the position and speed of interactive vehicles at different time points, and can improve the generation efficiency of simulation scenes, optimize the generation effect of simulation scenes, and effectively improve the controllability and generalization of autonomous driving simulation scenes.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for generating an autonomous driving simulation scene provided according to an embodiment of the present specification.

[0018] Figure 2a The figure is a flowchart of a first speed data acquisition method provided according to an embodiment of the present specification.

[0019] Figure 2b A schematic diagram of a lane change intrusion scenario provided according to an embodiment of the present specification.

[0020] Figure 2c A schematic diagram of a lane change cut-out scenario provided according to an embodiment of the present specification.

[0021] Figure 3a The figure is a flowchart of another first speed data acquisition method provided according to an embodiment of the present specification.

[0022] Figure 3b A schematic diagram of a car-following acceleration scenario provided according to an implementation of this specification.

[0023] Figure 3c A schematic diagram of a car-following deceleration scenario provided according to an implementation of this specification.

[0024] Figure 4 A flowchart of another method for generating an autonomous driving simulation scene provided according to an embodiment of the present specification.

[0025] Figure 5 The present invention is a flowchart of a method for obtaining first simulation speed data and second simulation speed data according to an embodiment of the present specification.

[0026] Figure 6 A schematic diagram of the process of model training and data generation provided according to an embodiment of the present specification.

[0027] Figure 7 This is a structural block diagram of an autonomous driving simulation scene generation device provided according to an embodiment of this specification.

[0028] Figure 8 A structural block diagram of a computer device provided according to an embodiment of the present specification. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0030] With the rapid development of science and technology, smart vehicles that rely on intelligent control modules such as automatic driving systems to assist drivers in driving safely or replace drivers in controlling vehicles are widely used in people's lives. In order to ensure the safety of smart vehicles, it is necessary to conduct a large number of tests on smart vehicles in automatic driving simulation scenarios, so as to evaluate the automatic driving capabilities of smart vehicles and their adaptability to traffic environments based on intelligent tests.

[0031] Autonomous driving simulation scenario testing refers to testing the driving behavior of autonomous driving vehicles or autonomous driving algorithms in virtual autonomous driving test scenarios, so as to determine whether the autonomous driving vehicles or autonomous driving algorithms can achieve intelligent autonomous driving effects, such as automatic obstacle avoidance, intelligent response to emergencies, etc. Therefore, the simulation effect of autonomous driving simulation scenarios has an important impact on the evaluation of autonomous driving tests.

[0032] In related technologies, the generation of autonomous driving simulation scenes mainly includes two types of methods. The first type of method is mainly based on the scene characteristics of the real world, extracting dynamic and static data from video image data, and then using dynamic and static data to reconstruct events in a simulation environment to achieve the transformation from real to virtual.

[0033] The second type of method mainly collects some autonomous driving scene data sets, and after scene classification, divides them into static data sets and dynamic data sets, outputs static data set data information and dynamic data set data information. For the dynamic and static data extracted from various scenes, machine learning and deep learning methods are used to perform data repair and data generation based on dynamic data set data information to generate corresponding autonomous driving simulation scenes. For example, dynamic data generation of simulation scenes is performed by constructing a GAN (Generative Adversarial Networks) network, and new simulation scenes are generated in combination with static data sets.

[0034] However, the first type of method mentioned above has problems such as huge engineering workload, high cost, and poor generalization. Specifically, the efficiency of constructing the simulation test scenario library is low, and it is difficult to meet the use case requirements of the expected functional safety test.

[0035] The second type of methods mentioned above currently have few descriptions of the operating procedures, specific implementation methods, and tools used to generate autonomous driving test scenarios for the interaction between two vehicles. For example, the characteristics of scenarios such as the opponent's sudden stop and the opponent's lane change are not clearly defined, and it is impossible to give the operator clearer guidance. In addition, most methods use vehicle trajectory data as input, but because the data points at a certain moment in the trajectory data have a large correlation and dependence with the data at the previous or subsequent moments, and machine learning is usually unable to effectively capture and utilize this time dependency, it is unable to better deal with the temporal characteristics of the trajectory data. Therefore, the trajectory data generated using vehicle trajectory data as input may often be distorted at the starting position or the end position.

[0036] In order to solve the trajectory distortion problem in the generation of autonomous driving scenes, it is necessary to propose a method, device, equipment and storage medium for generating autonomous driving simulation scenes. The corresponding simulated speed data is generated based on the real speed data to further generate the simulation trajectory. Since the speed data can directly reflect the time dependence of the vehicle movement, the dynamic changes of the trajectory data can be better simulated by analyzing the changes in the speed characteristics. Therefore, the problem of trajectory data distortion can be effectively overcome. Generating simulation trajectories based on real speed data (rather than simply extrapolating based on position data or timestamps) can more accurately simulate the position and speed of the interactive vehicles at different time points, and can improve the generation efficiency of the simulation scene, optimize the generation effect of the simulation scene, and effectively improve the controllability and generalization of the autonomous driving simulation scene.

[0037] Furthermore, the method for generating an autonomous driving simulation scene provided in this specification also simulates by fusing the trajectory data and speed data of the vehicle to generate the corresponding simulated trajectory data and simulated speed data of the simulated vehicle. By adding the corresponding speed features on the basis of the trajectory data, the defect of poor learning effect of the machine learning method for the temporal nature of trajectory data in the related technology can be made up, thereby greatly reducing the trajectory distortion problem of other vehicles except the tested vehicle in the autonomous driving simulation test scene, and improving the generation effect of the autonomous driving simulation scene.

[0038] This specification provides a method for generating an autonomous driving simulation scene. Figure 1 As shown, the method for generating an autonomous driving simulation scene may include the following steps.

[0039] S110: Acquire vehicle driving video frames.

[0040] The vehicle driving video frame includes at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle.

[0041] S120. Obtain first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame.

[0042] S130. Obtain first simulation speed data of the first simulation vehicle based on the first speed data.

[0043] S140 , performing trajectory calculation according to the starting position data of the first simulated vehicle and the first simulated speed data to obtain scene trajectory data of the first simulated vehicle.

[0044] S150: Acquire the starting position data and initial speed data of the second simulated vehicle.

[0045] The change of the driving state of the first simulated vehicle has an impact on the change of the driving state of the second simulated vehicle.

[0046] S160: Generate an autonomous driving simulation scene based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

[0047] The vehicle driving video frame is a video frame in a vehicle driving video, and the vehicle driving video is a video clip collected from a real road environment and containing two or more vehicles (at least one pair of vehicles has an interactive relationship).

[0048] If two or more vehicles have an interactive relationship, the changes in driving state such as speed change, lane change, and lane merging generated by one vehicle will directly affect the driving decisions of other vehicles, thereby affecting the changes in driving state of other vehicles. In this specification, two or more vehicles that have an interactive relationship in a vehicle driving video frame are referred to as a group of traffic participants.

[0049] The driving state change may include at least one of a vehicle speed change, a driving lane change, a driving direction change, etc.

[0050] The driving trajectory data of the first vehicle includes coordinate data of the first vehicle in the vehicle driving video frame.

[0051] The first simulated vehicle and the second simulated vehicle are simulation objects in the autonomous driving simulation scenario (autonomous driving simulation test scenario). The second simulated vehicle is used as the main vehicle to be tested in the scenario for deploying autonomous driving algorithms, etc. The first simulated vehicle is used as a background vehicle or interference vehicle in the scenario for simulating real traffic conditions. The change in the driving state of the first simulated vehicle will affect the autonomous driving driving decision of the second simulated vehicle, so that the behavior of the autonomous driving system or algorithm under changes in traffic conditions or road environment can be tested.

[0052] The first simulation speed data is obtained by simulating the first speed data.

[0053] The scene trajectory data is the motion trajectory simulated by the first simulated vehicle in the autonomous driving simulation scene.

[0054] The starting position data of each of the first simulated vehicle and the second simulated vehicle are the starting coordinates of each of the first simulated vehicle and the second simulated vehicle in the automatic driving simulation scene.

[0055] An autonomous driving simulation scenario is a simulation scenario that simulates real traffic environments and scenarios to test and verify autonomous driving algorithms or systems.

[0056] Specifically, a hovering drone can be set above the real scene to be simulated, or a shooting device can be set up at a fixed height to collect vehicle driving video data in the real scene, so as to capture and record the information of all traffic participants in the scene. Based on the collected vehicle driving video data, a vehicle driving video frame containing two or more vehicles with an interactive relationship can be obtained.

[0057] For any vehicle driving video frame, the vehicle that has a direct impact on the driving state change of other vehicles in the interactive relationship can be regarded as the first vehicle, and the affected vehicle can be regarded as the second vehicle. According to the coordinate data of the first vehicle in multiple vehicle driving video frames, the driving trajectory data of the first vehicle can be obtained, and by analyzing the driving trajectory data, the first speed data of the first vehicle can be calculated.

[0058] Based on the real first speed data of the first vehicle, the speed data in the simulation environment can be obtained as the first simulated speed data of the first simulated vehicle. The driving trajectory of the first simulated vehicle in the simulation scene can be generated by combining the starting position data and the first simulated speed data of the first simulated vehicle and using trajectory calculation methods such as kinematic models, so as to obtain the scene trajectory data of the first simulated vehicle.

[0059] For the second simulated vehicle in the autonomous driving simulation scene, the starting position data and initial speed data of the second simulated vehicle can be obtained according to pre-set parameters or through simulation or other methods. The scene trajectory data of the first simulated vehicle, the starting position and initial speed data of the second simulated vehicle, and other possible environmental factors (such as road conditions, traffic signals, etc.) are combined to generate an autonomous driving simulation scene.

[0060] It is understandable that whether there is an interactive relationship between vehicles can be determined based on the vehicle driving video (or multiple vehicle driving video frames).

[0061] In some embodiments, the interaction relationship is an interaction relationship between two vehicles, and the number of the first vehicle is one.

[0062] Exemplarily, multiple vehicle driving video frames are obtained based on the video data collected in the real traffic scene. For any video frame, it is assumed that a group of vehicles (including two vehicles) with a lane change interaction relationship (the lane change behavior of one vehicle directly affects the driving state of another vehicle) is included therein. The vehicle with the lane change behavior can be regarded as the opponent vehicle (corresponding to the first vehicle in this specification), and the vehicle affected by its lane change can be regarded as the main vehicle (corresponding to the second vehicle in this specification).

[0063] According to the coordinate data of the first vehicle in multiple video frames and the time corresponding to each of the multiple video frames, the speed of the first vehicle in each video frame can be calculated to obtain the complete speed of the first vehicle in multiple video frames. By simulating or converting the speed of the first vehicle, the simulated speed in the autonomous driving simulation scene is obtained as the simulated speed data of the first simulated vehicle. According to the starting position data of the first simulated vehicle that is preset or generated by simulation or random generation, the driving trajectory of the first simulated vehicle in the simulation scene is calculated with the simulated speed data to simulate the movement trajectory of the opponent vehicle in the simulation scene. Furthermore, based on the driving trajectory of the first simulated vehicle, the starting position and initial speed of the second simulated vehicle, a dynamic interaction scene in the autonomous driving simulation scene can be generated.

[0064] In some other embodiments, the interaction relationship is an interaction relationship between multiple vehicles, the number of the first vehicles may be multiple, and accordingly, the number of the first simulation vehicles may also be multiple.

[0065] Exemplarily, multiple vehicle driving video frames are obtained based on the video data collected in the real traffic scene. For any video frame, it is assumed that it contains a group of vehicles (including multiple vehicles on multiple lanes) that have a lane change interaction relationship. The vehicle with lane change behavior can be regarded as the opponent vehicle (corresponding to the first vehicle in this specification), and the vehicle affected by the lane change can be regarded as the main vehicle (corresponding to the second vehicle in this specification).

[0066] For example, a video frame includes three adjacent lanes, which are respectively recorded as lane 1, lane 2, and lane 3, and there is a vehicle in each lane. The vehicle in lane 1 is recorded as vehicle 1, the vehicle in lane 2 is recorded as vehicle 2, and the vehicle in lane 3 is recorded as vehicle 3. Assume that vehicle 1 and vehicle 3 change lanes and drive into lane 2 respectively, and drive in front of vehicle 2. Then, vehicle 1 and vehicle 3 can be regarded as opponent vehicles, and vehicle 2 can be regarded as the main vehicle.

[0067] The description of obtaining the simulated speed data of the first simulated vehicle and the driving trajectory in the simulated scene is similar to that in the above process, and will not be repeated here.

[0068] It should be noted that the vehicle in the vehicle driving video frame can be identified by target detection and other methods, and a group of traffic participants with an interactive relationship can be determined from any vehicle driving video frame by analyzing trajectories and other methods, and the first vehicle and the second vehicle can be determined therefrom. In multiple different vehicle driving video frames, multiple groups of different traffic participants can be determined. In this specification, the first vehicle and the second vehicle refer to the same group of traffic participants.

[0069] The starting position data of the first simulated vehicle may be preset or obtained by simulating the trajectory of the first vehicle; the starting position data and initial speed data of the second simulated vehicle may be preset or obtained by simulating the trajectory of the second vehicle.

[0070] Driving status changes may include changing lanes, exiting a ramp, merging lanes, overtaking, etc., which are not specifically limited in this specification.

[0071] The driving track data of the first vehicle in the vehicle driving video frame, the first speed data, the starting position data of the first simulated vehicle, the first simulation speed data, the scene track data, the starting position data and the initial speed data of the second simulated vehicle all include horizontal and vertical data. The horizontal and vertical directions can be determined according to the coordinate system corresponding to the video frame.

[0072] In the above implementation, by generating a simulation trajectory based on real speed data in a real scene (rather than simply extrapolating based on position data or timestamps), the position and speed of the opponent vehicle at a future time point can be more accurately simulated in the autonomous driving simulation scene. Since speed data can directly reflect the time dependency of vehicle motion, by analyzing the change in speed characteristics, the problem of trajectory data distortion in the prior art of directly using trajectory data for simulation can be effectively overcome, thereby optimizing the generation effect of the simulation scene and effectively improving the controllability and generalization of the autonomous driving simulation scene.

[0073] In some implementations, the first vehicle includes a first driving state and a second driving state in the vehicle driving video frame. Figure 2a As shown, obtaining the first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame may include the following steps.

[0074] S210. Determine an interaction event point according to the second driving state; wherein the first driving state is adjacent to the second driving state, and the second driving state is located after the first driving state.

[0075] S220: Determine a trajectory data time range according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point.

[0076] S230. Determine first trajectory data of the first vehicle in the vehicle driving video frame based on a trajectory data time range; wherein the driving trajectory data of the first vehicle includes the first trajectory data.

[0077] S240: Obtain first speed data according to the first trajectory data.

[0078] The first driving state is the driving state of the first vehicle before the driving state changes, and the second driving state is the driving state of the first vehicle after the driving state changes, and the first driving state is different from the second driving state. The driving state change of the vehicle may include a change in the driving lane, a change in driving speed or acceleration, a change in driving direction, etc.

[0079] The interaction event point is the time point when the first vehicle has a driving state change behavior or after the driving state change behavior in the interaction scene between two vehicles (i.e., the main vehicle and the opponent vehicle), which has a major impact on the driving decision of the second vehicle, that is, the vehicle interaction event point. It can be understood that if there are multiple vehicles with an interactive relationship in any vehicle driving video frame, the interaction event point between any two vehicles with an interactive relationship can be determined.

[0080] The trajectory data time range is used to define the desired trajectory position and length in the full length trajectory included in the vehicle driving video.

[0081] Specifically, the time period in which the first vehicle changes from the first driving state to the second driving state in the video frame can be identified based on the vehicle driving video frame, and the time point that has a major impact on the driving decision of the second vehicle can be determined based on the second driving state of the first vehicle as the interaction event point of the vehicle interaction. The trajectory data time range containing the interaction event point is determined based on the interaction event point and the preset duration, so as to intercept the first trajectory data of the first vehicle of the corresponding length in the vehicle driving video frame based on the trajectory data time range. The first speed data of the first vehicle is obtained by calculation based on the first trajectory data and the time of the corresponding video frame.

[0082] For the two-vehicle interaction scenario in the traffic scenario, the interaction event point may have different definition rules according to the difference in the vehicle interaction behavior. In some embodiments, for the lane change scenario, when the main vehicle always keeps driving in the same lane within a certain period of time, if the opponent vehicle changes lanes, the time point when the opponent vehicle crosses the lane boundary line can be defined as the interaction event point in the lane change scenario. The lane change result may include two types: the opponent vehicle invades the lane where the main vehicle is located from other lanes, and the opponent vehicle cuts out from the lane where the main vehicle is located to other lanes. It can be understood that for the lane change scenario, it can be considered that the main vehicle and the opponent vehicle keep driving in the same direction.

[0083] For example, by analyzing multiple vehicle video frames, two vehicles interacting in a lane change scenario can be determined, and the host vehicle and the opponent vehicle can be determined from them. The interaction event point (hereinafter referred to as the event point) is defined as the time point when the front of the opponent vehicle crosses the lane boundary line.

[0084] The trajectory data of the two vehicles are preprocessed. Specifically, from the trajectory data set extracted from the video data, based on the corresponding event point definition, all time points that meet the requirements are screened out, and the preset time length can be set to include the time length Time_s before the event point and the time length Time_e after the event point, to obtain the trajectory data time range T containing the event point. Based on the time range, the time series data is determined to be combined into a single group of time series data with a total time length T in the trajectory data set. A new data set is formed by extracting the time series data corresponding to the event points of different opponent vehicles in all lane change scenarios.

[0085] For example, taking the above scenario as an example, the interaction event point can also be defined as the time point at which the center point of the opponent car crosses the lane boundary line. The description of determining the time range of the trajectory data is similar to the above description, and will not be repeated here.

[0086] In other embodiments, for the following scenario (the main vehicle is behind the opponent vehicle), when the main vehicle and the opponent vehicle maintain driving in the same lane within a certain period of time, and the opponent vehicle in front generates an acceleration increase or acceleration decrease behavior, the first driving state is the driving state before the acceleration change, and the second driving state is the driving state after the acceleration change. The interaction event point can be defined as the time point when the opponent vehicle is farthest from the main vehicle after the acceleration increase behavior, or the time point when the opponent vehicle is closest to the main vehicle after the acceleration decrease behavior.

[0087] In some other embodiments, for turning scenarios, when the host car and the opponent car always keep driving in the same lane for a certain period of time, if the opponent car turns, the time point when the opponent car cuts out of the host car's lane to another direction lane can be defined as the interaction event point. Alternatively, when the host car always keeps driving in the same lane for a certain period of time, and the host car and the opponent car are driving in different directions, if the opponent car turns and needs to cross the host car's lane, the time point when the opponent car crosses the boundary line of the host car's lane can be defined as the interaction event point.

[0088] It should be noted that the definition of the interaction event point can be determined according to actual test requirements, etc., and is not specifically limited in this specification. The vehicle driving video data can be analyzed to identify that the first vehicle changes from the first driving state to the second driving state.

[0089] In the above implementation, the concept of "interaction event point" (abbreviated as "event point") is proposed for the vehicle interaction behavior scenario in the traffic scenario. Based on the event point, a certain time length is extended forward and backward to obtain the trajectory data time range, and the time range is used as the preset event duration to extract the trajectory data of the opponent vehicle before and after the event point from the real trajectory data. In this way, the feature definition of the autonomous driving test scenario can be further clarified, and the length of the real trajectory data used for simulation can be standardized to a certain extent, thereby improving the efficiency and stability of scenario generation.

[0090] In some embodiments, the first vehicle is located in the first lane in the first driving state; the first vehicle is located in the second lane in the second driving state; when the first vehicle changes from the first driving state to the second driving state, the first vehicle enters the second lane from the first lane. Determining the interaction event point according to the second driving state may include: determining the interaction event point according to the moment when the first vehicle enters the second lane.

[0091] The first lane and the second lane are different, and the second vehicle is located in the first lane or the second lane.

[0092] In this embodiment, the first vehicle and the second vehicle travel in the same direction. The first driving state indicates that the first vehicle is traveling in the first lane before the lane change occurs, and the second driving state indicates that the first vehicle is traveling in the second lane after the lane change occurs.

[0093] The moment when the first vehicle enters the second lane may be a time point when the front or center point of the first vehicle crosses or is located at a lane boundary line between the first lane and the second lane.

[0094] Specifically, during a certain period of time, the first vehicle is traveling in the first lane, which is recorded as the first vehicle being in the first driving state. In the lane-changing scenario, the first vehicle changes from the first driving state to the second driving state, indicating that the first vehicle switches from the first lane to the second lane. Since in the autonomous driving scenario, the lane-changing of the opponent vehicle has a direct impact on the driving decision of the main vehicle, it is considered that the lane-changing of the first vehicle has a direct impact on the driving decision of the second vehicle. Then, the time point when the front or center point of the first vehicle crosses the lane boundary line between the first lane and the second lane can be regarded as the moment when the first vehicle enters the second lane, thereby determining the moment as the interaction event point.

[0095] In some embodiments, the second vehicle is located in the second lane. When the first vehicle changes from the first driving state to the second driving state, the first vehicle invades the lane where the second vehicle is located. This scenario is called a lane change intrusion scenario.

[0096] For example, reference Figure 2bAs shown, the first vehicle is defined as the opponent vehicle, and the second vehicle is defined as the host vehicle. The opponent vehicle is located in the host lane (corresponding to the first lane) before the driving state changes (i.e., in the first driving state), and the host vehicle is driving in the target lane (corresponding to the second lane). The opponent vehicle changes lanes and enters the target lane, that is, the opponent vehicle changes its driving state and is located in the target lane in the second driving state. The time point when the center point of the opponent vehicle is located at or crosses the lane boundary line between the host lane and the target lane is taken as the moment when the opponent vehicle enters the target lane from the host lane, so that this moment is determined as the interaction event point in this scenario.

[0097] In other embodiments, the second vehicle is located in the first lane. When the first vehicle changes from the first driving state to the second driving state, the first vehicle cuts out of the lane where the second vehicle is located. This scenario is called a lane change scenario.

[0098] For example, reference Figure 2c As shown, the first vehicle is defined as the opponent vehicle, and the second vehicle is defined as the host vehicle. The opponent vehicle is located in the host lane (corresponding to the first lane) in the first driving state, and the host vehicle is driving in the host lane. The opponent vehicle changes lanes and drives into the target lane (corresponding to the second lane), that is, the opponent vehicle changes its driving state and is located in the target lane in the second driving state. The time point when the center point of the opponent vehicle is located at or crosses the lane boundary line between the host lane and the target lane is defined as the interaction event point in this scenario.

[0099] It can be understood that the moment when the first vehicle drives from the first lane into the second lane can also be regarded as the moment when the first vehicle drives out of the first lane.

[0100] In some implementations, the first vehicle and the second vehicle are located in the same lane; the first vehicle has a speed state in the vehicle driving video frame. Figure 3a As shown, obtaining the first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame may include the following steps.

[0101] S310. If the first vehicle is moving away from the second vehicle in a speed state, an interaction event point is determined according to a time corresponding to the maximum speed data of the first vehicle in the speed state; or, if the first vehicle is approaching the second vehicle in a speed state, an interaction event point is determined according to a time corresponding to the minimum speed data of the first vehicle in the speed state.

[0102] S320: Determine a trajectory data time range according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point.

[0103] S330. Determine second trajectory data of the first vehicle in the vehicle driving video frame based on a time range of the trajectory data; wherein the driving trajectory data of the first vehicle includes the second trajectory data.

[0104] S340: Obtain first speed data according to the second trajectory data.

[0105] The speed state may be either an acceleration state or a deceleration state.

[0106] In this embodiment, the second vehicle travels in the same direction as the first vehicle, and the second vehicle is located behind the first vehicle.

[0107] Specifically, within a certain period of time, the first vehicle and the second vehicle are traveling in the same lane, and the second vehicle is located behind the first vehicle. This scenario is defined as a following scenario. In the following scenario, the first vehicle has a certain speed state. If the first vehicle moves away from the second vehicle in the speed state, indicating that the speed state is an accelerated state, the moment when the first vehicle reaches the maximum speed can be determined based on the maximum speed data of the first vehicle in the accelerated state, and the moment is used as the interaction event point.

[0108] Alternatively, if the first vehicle approaches the second vehicle in a speed state, indicating that the speed state is a deceleration state, the moment when the first vehicle reaches the minimum speed can be determined based on the minimum speed data of the first vehicle in the speed state, and the moment can be used as the interaction event point.

[0109] The time range of the trajectory data including the interaction event point is determined according to the interaction event point and the preset duration, so as to intercept the second trajectory data of the first vehicle of the corresponding length in the vehicle driving video frame based on the trajectory data time range. The first speed data of the first vehicle is obtained by calculating according to the second trajectory data and the time of the corresponding video frame.

[0110] It should be noted that, in this specification, the scenario in which the first vehicle moves away from the second vehicle at a speed state is called a following acceleration scenario, and the scenario in which the first vehicle approaches or gets close to the second vehicle at a speed state is called a following deceleration scenario.

[0111] For example, reference Figure 3b As shown in the figure, the first vehicle is defined as the opponent vehicle, the second vehicle is defined as the main vehicle, the main vehicle and the opponent vehicle are traveling in the same lane, and the main vehicle is located behind the opponent vehicle, and the lane is recorded as the host lane. By analyzing the trajectory of the opponent vehicle in the vehicle driving video frame, it can be known that the opponent vehicle is in an accelerating state and gradually moves away from the main vehicle, so this scene is a following acceleration scene. The corresponding speed is calculated according to the trajectory of the opponent vehicle, and the moment when the opponent vehicle reaches the maximum speed can be determined, so as to use this moment as the interaction event point.

[0112] For example, reference Figure 3c As shown in the figure, the first vehicle is defined as the opponent vehicle, the second vehicle is defined as the main vehicle, the main vehicle and the opponent vehicle are traveling in the same lane, and the main vehicle is located behind the opponent vehicle, and the lane is recorded as the host lane. By analyzing the trajectory of the opponent vehicle in the vehicle driving video frame, it can be known that the opponent vehicle is in a deceleration state and is gradually approaching the main vehicle, so this scene is a following deceleration scene. The corresponding speed is calculated according to the trajectory of the opponent vehicle, and the moment when the opponent vehicle reaches the minimum speed can be determined, so that this moment is used as the interaction event point.

[0113] It should be noted that in the above embodiment, the trajectory and speed state analysis is performed based on the center point of the first vehicle, and the trajectory and speed state analysis can also be performed based on the front position point or parking position point of the first vehicle, etc., which is not specifically limited in this specification.

[0114] In some embodiments, reference Figure 4 As shown, before obtaining the starting position data and initial speed data of the second simulated vehicle, the automatic driving simulation scene generation method may also include the following steps.

[0115] S410. Obtain second speed data of the second vehicle according to the driving trajectory data of the second vehicle in the vehicle driving video frame.

[0116] S420: Obtain second simulated speed data of a second simulated vehicle based on the second speed data.

[0117] Accordingly, obtaining the initial speed data of the second simulated vehicle may include:

[0118] S430, obtaining initial speed data of the second simulated vehicle according to the starting speed data of the second simulated speed data.

[0119] The driving trajectory data of the second vehicle includes coordinate data of the second vehicle in the vehicle driving video frame.

[0120] The second simulation speed data is obtained by simulating the second speed data.

[0121] Specifically, based on the collected vehicle driving video data, a vehicle driving video frame containing two or more vehicles having an interactive relationship can be obtained. Based on the second vehicle determined in the vehicle driving video frame, the coordinate data of the second vehicle in multiple video frames is obtained, thereby obtaining the driving trajectory data of the second vehicle. By analyzing the driving trajectory data, the second speed data of the second vehicle can be calculated.

[0122] Based on the simulation of the real second speed data of the second vehicle, the speed data in the simulation environment can be obtained as the second simulated speed data of the second simulated vehicle. Since the second simulated vehicle is a simulated vehicle to be tested for deploying the autonomous driving algorithm, its real-time speed in the test scene needs to be obtained by the autonomous driving algorithm in response to changes in the traffic scene. Therefore, the starting speed data in the second simulated speed data can be used as the initial speed data of the second simulated vehicle. Thus, the initial speed obtained based on the real speed simulation can be set for the second simulated vehicle, so that the simulation scene is closer to the real scene, so as to simulate the real traffic scene more realistically, and the generalization ability of the scene library can be improved.

[0123] It should be noted that, for the description of the method for obtaining the second speed data in the above-mentioned implementation mode, please refer to the description of the method for obtaining the first speed data in this specification, and for the description of the method for obtaining the second simulation speed data, please refer to the description of the method for obtaining the first simulation speed data in this specification, and the details will not be repeated here.

[0124] In some implementations, the first speed data is speed sample data of a first simulated vehicle; and the second speed data is speed sample data of a second simulated vehicle. Figure 5 As shown, obtaining the first simulation speed data based on the first speed data, and obtaining the second simulation speed data based on the second speed data may include the following steps.

[0125] S510, obtaining combined sample data based on the first speed data and the second speed data for training an initial data generation model to obtain a trained simulation data generation model.

[0126] S520. Generate simulated vehicle data using a simulation data generation model; wherein the simulated vehicle data includes first simulated speed data and second simulated speed data.

[0127] Among them, the initial data generation model is a generative adversarial network model (Generative Adversarial Networks, GAN) that can be used to generate simulated speed data.

[0128] Specifically, the first speed data obtained above can be used as speed sample data of the first simulated vehicle, and the second speed data can be used as speed sample data of the second simulated vehicle, so as to obtain combined sample data based on the combination of the first speed data and the second speed data.

[0129] The initial data generation model is trained using the combined sample data to obtain a simulation data generation model, so that the simulation data generation model can simulate and generate first simulation speed data corresponding to the first speed data and second simulation speed data corresponding to the second speed data. The simulation data generation model can eventually output simulated vehicle data including the first simulation speed data and the second simulation speed data.

[0130] In some embodiments, the initial data generation model may adopt a TimeGAN (Time Series Generative Adversarial Networks) model.

[0131] Exemplarily, a time series data set of a group of traffic participants can be obtained based on the first speed data of the first vehicle and the second speed data of the second vehicle obtained by the aforementioned method. Based on the data information of the time series data sets of multiple groups of traffic participants, a TimeGAN network is constructed to obtain a TimeGAN model for generating autonomous driving simulation scenes, and these time series data sets are used to train the TimeGAN model for autonomous driving scene generation. During the training process, optimized hyperparameters are set, and the optimized TimeGAN model data information is output. Among them, the hyperparameter optimization method can include a grid search method, a random search method, and the like.

[0132] Based on the optimized TimeGAN model data information, the TimeGAN generator is used to automatically generate similar scene data, wherein the generated scene feature data may include the vehicle speed data (corresponding to the second simulation speed data) and the opponent vehicle speed data (corresponding to the first simulation speed data).

[0133] It should be noted that the initial data generation model may also adopt other model structures, which can be determined according to actual application requirements, etc., and is not specifically limited in this specification.

[0134] In this implementation, the first speed data and the second speed data may be combined to obtain combined sample data.

[0135] In some embodiments, the driving trajectory data of the first vehicle is the trajectory sample data of the first simulated vehicle; the driving trajectory data of the second vehicle is the trajectory sample data of the second simulated vehicle. Obtaining the combined sample data based on the first speed data and the second speed data may include: combining the first speed data, the driving trajectory data of the first vehicle, the second speed data, and the driving trajectory data of the second vehicle to obtain the combined sample data.

[0136] Among them, the simulated vehicle data also includes first simulated trajectory data of the first simulated vehicle and second simulated trajectory data of the second simulated vehicle; the starting position data of the first simulated vehicle is obtained according to the starting coordinate data of the first simulation trajectory data; the starting position data of the second simulated vehicle is obtained according to the starting coordinate data of the second simulation trajectory data.

[0137] Specifically, the first speed data obtained above can be used as the speed sample data of the first simulated vehicle, the driving trajectory data of the first vehicle can be used as the trajectory sample data of the first simulated vehicle, the second speed data can be used as the speed sample data of the second simulated vehicle, and the driving trajectory data of the second vehicle can be used as the trajectory sample data of the second simulated vehicle. The first speed data, the driving trajectory data of the first vehicle, the second speed data and the driving trajectory data of the second vehicle are combined, that is, the speed sample data and the trajectory sample data of the first simulated vehicle, and the speed sample data and the trajectory sample data of the second simulated vehicle are combined to obtain the combined sample data for training the initial data generation model.

[0138] Further, the initial data generation model is trained using the combined sample data to obtain a simulation data generation model, so that the simulation data generation model can simulate and generate first simulation speed data corresponding to the first speed data, first simulation trajectory data corresponding to the driving trajectory data of the first vehicle, second simulation speed data corresponding to the second speed data, and second simulation trajectory data corresponding to the driving trajectory data of the second vehicle. The simulation data generation model can ultimately output simulated vehicle data including the first simulation speed data, the first simulation trajectory data, the second simulation speed data, and the second simulation trajectory data.

[0139] According to the simulated vehicle data output by the model, the starting coordinate data in the first simulated trajectory data can be used as the starting position data of the first simulated vehicle, and the starting coordinate data in the second simulated trajectory data can be used as the starting position data of the second simulated vehicle.

[0140] It should be noted that, in this embodiment, the initial data generation model is used to generate simulated speed data and corresponding simulated trajectory data. In this specification, the initial data generation model / simulation data generation model may also be referred to as a motion trajectory generation model.

[0141] For example, the main process of the method for generating an autonomous driving simulation scene for vehicle interaction behavior provided in this specification may include:

[0142] The first step is to collect data. Set up a hovering drone above the real scene that needs to be simulated or set up equipment at a fixed height to collect video data of vehicle driving and capture and record information of all traffic participants in the scene.

[0143] The second step is data extraction. Feature extraction is performed on the above-collected video data. For dynamic traffic participants, the corresponding motion trajectories are extracted to obtain trajectory data sets. For static scene individuals, the corresponding position, shape, size and other information are extracted.

[0144] The third step is rule definition. For the interaction scenario between two vehicles (i.e., the main vehicle and the opponent vehicle) in the traffic scenario (Note: the main vehicle corresponds to the vehicle under test that needs to be connected to the autonomous driving algorithm in the autonomous driving test scenario, and the opponent vehicle is the background vehicle that interacts with the main vehicle), the concept of "event point" is proposed. According to the differences in vehicle interaction behaviors, event points have different definition rules. Specifically, for the lane changing scenario, when the main vehicle always keeps driving in the same lane for a certain period of time, and the opponent vehicle changes lanes at this time, the time point when the center point of the opponent vehicle crosses the lane boundary line can be defined as the event point in the lane changing scenario. Among them, the lane changing results in the lane changing scenario mainly include two types: the opponent vehicle invades the lane where the main vehicle is located from other lanes, and the opponent vehicle cuts out from the lane where the main vehicle is located to other lanes.

[0145] For the following car scenario (where the main car is behind the opponent car), when the main car and the opponent car keep driving in the same lane for a certain period of time, and the opponent car in front accelerates or decelerates, the event point in this scenario can be defined as the time point when the opponent car's speed is the highest or lowest. The following car results in the following car scenario mainly include two types: the opponent car accelerates away from the main car and the opponent car decelerates and approaches the main car.

[0146] The fourth step is to expand the data set. According to the test task requirements, the trajectory data of the two vehicles that generate interaction are preprocessed. The specific operations are as follows: In the trajectory data set extracted from the video data, based on the event point definitions corresponding to different scenes, all time points that meet the requirements are screened out. Set the time length before the event point to Time_s and the time length after the event point to Time_e, and extract the corresponding time series data with a time length of Time_s before the event point and a time series data with a time length of Time_e after the event point from the trajectory data set, and combine them into a single set of time series data with a total time length of T corresponding to the corresponding event point. Extract the time series data corresponding to all event points to form a new time series data set. It should be noted that all the above data sets only contain the lateral and longitudinal trajectory data of the vehicle.

[0147] The horizontal trajectory data is converted into the horizontal speed data, and the vertical trajectory data is converted into the vertical speed data. The specific conversion method is as follows:

[0148]

[0149] Where V L (t) represents the lateral velocity of a vehicle at time t; XL (t-1) represents the lateral position (lateral coordinate) of the vehicle at time t-1, X L (t) represents the lateral position of the vehicle at time t; Δt represents a single time step; V P (t) represents the longitudinal velocity of the vehicle at time t; X P (t-1) represents the longitudinal position (longitudinal coordinate) of the vehicle at time t-1, X P (t) represents the longitudinal position of the vehicle at time t.

[0150] It is understandable that, assuming that the main vehicle trajectory time series data containing N frames of position coordinates with a total duration of T and the opponent vehicle trajectory time series data containing N frames of position coordinates are extracted from the trajectory data set. The speed data corresponding to the trajectory of each vehicle obtained by the above method is added to the time series data set as a new feature, and data expansion is performed to obtain an expanded time series data set. Among them, it is understandable that the N frames of trajectory data of the main vehicle and the opponent vehicle respectively correspond to the corresponding N-1 frames of speed data.

[0151] The fifth step is to train the model. Based on the expanded time series data set, the TimeGAN network is constructed to obtain a vehicle motion trajectory generation model based on TimeGAN, and the model is trained. During the training process, the optimized hyperparameters are set and the optimized TimeGAN model data information is output.

[0152] refer to Figure 6 As shown in Figure 1, in the vehicle motion trajectory generation model based on TimeGAN, the input real data sequence usually contains static features that do not change with time and dynamic features that change with time. Generally, S is used to represent the static feature vector at the encoder input, and X is used to represent the time feature vector. In the model, the real trajectory sequence X is obtained through the encoder to obtain the hidden code H, and the decoder decodes the hidden code H into the restored sequence and X to calculate the loss function G vloss The encoding and decoding process provides a reversible mapping between the dynamic features of the trajectory sequence and the latent code, achieving dimensionality reduction of the temporal dynamic characteristics of the sequence, allowing the adversarial network to learn the potential temporal dynamic characteristics of the data through low-dimensional representation. The supervisor receives the latent code H output by the encoder and generator, and outputs By calculation and H 1:n-1 The loss function captures the time-varying stepwise conditional distribution p(H n |H 1:n-1 ). The generator inputs a random vector Z to generate a hidden code Obtain output with dynamic changing characteristics through the supervisor and The common input discriminator combines the label C to calculate the loss function G uloss With D loss , the label C is the category or attribute of the generated time series, which can help the generator and discriminator network to better generate time series data with specific attributes or categories. When generating trajectory data, the random vector Z passes through the generator from the sequence space to the latent space, and then passes through the supervisor and then the decoder to obtain the final generated sequence

[0153] The sixth step is to use the model to generate data. Based on the optimized TimeGAN model data information, the TimeGAN generator is used to automatically generate similar scene data. The generated scene feature data includes the vehicle position data (the simulation trajectory data of the second simulated vehicle), the vehicle speed data (the simulation speed data of the second simulated vehicle), the opponent vehicle position data (the simulation trajectory data of the first simulated vehicle) and the opponent vehicle speed data (the simulation speed data of the first simulated vehicle), and each feature data contains data in both the horizontal and vertical directions.

[0154] Continue to refer Figure 6 As shown, according to the main vehicle trajectory (also called the vehicle trajectory) containing N frames of position coordinates with a total duration of T, and the opponent vehicle trajectory containing N frames of position coordinates, the data set is expanded to obtain the main vehicle speed (also called the vehicle speed) containing N-1 frames of speed data and the opponent vehicle speed containing N-1 frames of speed data based on these trajectory data. Based on the trajectory data and speed data obtained by the data set expansion, the trained TimeGAN-based motion trajectory generation model is used to generate the data set to obtain the simulated N-frame vehicle trajectory, N-frame opponent vehicle trajectory, N-1 frame vehicle speed, and N-1 frame opponent vehicle speed.

[0155] Step 7: Dynamic scene modeling. According to the dynamic model, the opponent car speed data in the above-generated feature data and the first frame position data (i.e., starting coordinate data) in the opponent car position data are combined and calculated to convert them into the position trajectory of the opponent car in the simulation scene. The specific calculation method is as follows:

[0156] X L '(t) = X L '(t-1)+V L '(t)*Δt

[0157] X P '(t) = X P '(t-1)+V P '(t)*Δt

[0158] In the formula, XL '(t) represents the lateral position of a simulated vehicle at time t; X L '(t-1) represents the lateral position of the simulated vehicle at time t-1; V L '(t) represents the lateral velocity of the simulated vehicle at time t; X P '(t) represents the longitudinal position of the simulated vehicle at time t; X P '(t-1) represents the longitudinal position of the simulated vehicle at time t-1; V P '(t) represents the generated longitudinal velocity of the simulated vehicle at time t.

[0159] Step 8: The first frame trajectory (starting coordinate data) in the generated main vehicle trajectory data and the first frame speed (starting speed data) in the main vehicle speed data are used as the initial state data of the main vehicle. The opponent vehicle trajectory and the main vehicle initial state data obtained by the above calculations are combined to output the same dynamic scene of the autonomous driving simulation.

[0160] In the ninth step, the generated data information of similar dynamic scenes is coupled with the data information of the static data set extracted in the second step to output the autonomous driving simulation scenario library, which is mainly used for testing the interactive behavior between two vehicles and can cover two scenarios: following and lane changing.

[0161] Through the above method, data is generated based on the data collected in reality, which has good scene generalization ability, can improve the efficiency of scene library construction, and make up for the lack of test cases. First, through the interactive behavior scene of two vehicles, the concept of event point is proposed. Based on the event point and the preset event duration, the complete trajectory data of the opponent car and the vehicle before and after the event is extracted, the vehicle speed data is calculated using the trajectory data, and the TimeGAN network is trained using the speed data to generate similar dynamic scene data through the TimeGAN network. Secondly, the data of the first frame of the generated dynamic data of the vehicle and the opponent car (including vehicle speed and position data) is used as the static information of the test case, and the generated complete opponent car speed data is used as the basis for updating the trajectory of the opponent car in the test case to construct a single complete test case. Then, all the data generated by the TimeGAN network are constructed one by one to generate a simulation scene library. This method clearly describes the scene definition tool and operation process, and proposes to use the generated opponent car speed data to support trajectory generation, which can effectively overcome the problem of trajectory data distortion in related technologies.

[0162] The above method can effectively improve the efficiency of scene library generation, and the scene library has good generalization ability and can supplement data other than the scene library based on real data. At the same time, the usage scenario corresponding to the above generation method is clearly defined, and the method and process are clear, which can greatly reduce the trajectory distortion problem of vehicles other than the tested vehicle in the generated autonomous driving simulation scene.

[0163] This specification provides an automatic driving simulation scene generation device, referring to Figure 7 As shown, the autonomous driving simulation scene generation device 700 may include: a video frame acquisition module 710, a speed data acquisition module 720, a simulation speed acquisition module 730, a scene trajectory acquisition module 740, a simulation vehicle data acquisition module 750, and a simulation scene generation module 760.

[0164] The video frame acquisition module 710 is used to acquire vehicle driving video frames; wherein the vehicle driving video frames include at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle.

[0165] The speed data acquisition module 720 is used to obtain first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame.

[0166] The simulation speed acquisition module 730 is used to obtain first simulation speed data of the first simulation vehicle based on the first speed data.

[0167] The scene trajectory acquisition module 740 is used to perform trajectory calculation according to the starting position data and the first simulation speed data of the first simulation vehicle to obtain the scene trajectory data of the first simulation vehicle.

[0168] The simulated vehicle data acquisition module 750 is used to acquire the starting position data and initial speed data of the second simulated vehicle; wherein the change in the driving state of the first simulated vehicle has an impact on the change in the driving state of the second simulated vehicle.

[0169] The simulation scene generation module 760 is used to generate an automatic driving simulation scene based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

[0170] For the specific definition of the automatic driving simulation scene generation device, please refer to the definition of the automatic driving simulation scene generation method above, which will not be repeated here. Each module in the above-mentioned automatic driving simulation scene generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0171] The present specification also provides a computer device, referring to Figure 8 As shown, the computer device 800 may include a memory 810, a processor 820, and a computer program 830 stored in the memory 810 and executable on the processor 820. When the processor 820 executes the computer program 830, the method for generating an autonomous driving simulation scene in any of the aforementioned embodiments is implemented.

[0172] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating an autonomous driving simulation scene in any of the aforementioned embodiments.

[0173] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0174] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0176] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0177] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0178] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for generating an autonomous driving simulation scene, characterized in that: The method comprises: Acquire a vehicle driving video frame; wherein the vehicle driving video frame includes at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle; Obtaining first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame; Obtaining first simulated speed data of a first simulated vehicle based on the first speed data; Performing trajectory calculation according to the starting position data of the first simulated vehicle and the first simulated speed data to obtain scene trajectory data of the first simulated vehicle; Acquire the starting position data and initial speed data of the second simulated vehicle; wherein the change of the driving state of the first simulated vehicle has an impact on the change of the driving state of the second simulated vehicle; The automatic driving simulation scene is generated based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

2. The method according to claim 1, characterized in that The first vehicle includes a first driving state and a second driving state in the vehicle driving video frame; and obtaining first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame includes: Determining an interaction event point according to the second driving state; wherein the first driving state is adjacent to the second driving state, and the second driving state is located after the first driving state; Determine a trajectory data time range according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point; Determining first trajectory data of the first vehicle in the vehicle driving video frame based on the trajectory data time range; wherein the driving trajectory data of the first vehicle includes the first trajectory data; The first speed data is obtained according to the first trajectory data.

3. The method according to claim 2, characterized in that The first vehicle is located in the first lane in the first driving state; the first vehicle is located in the second lane in the second driving state; when the first vehicle changes from the first driving state to the second driving state, the first vehicle drives from the first lane to the second lane; The determining the interaction event point according to the second driving state includes: The interaction event point is determined according to the moment when the first vehicle enters the second lane.

4. The method according to claim 1, characterized in that The first vehicle and the second vehicle are located in the same lane; the first vehicle has a speed state in the vehicle driving video frame; and obtaining first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame includes: If the first vehicle is moving away from the second vehicle in the speed state, the interaction event point is determined according to the time corresponding to the maximum speed data of the first vehicle in the speed state; or, if the first vehicle is approaching the second vehicle in the speed state, the interaction event point is determined according to the time corresponding to the minimum speed data of the first vehicle in the speed state; Determine a trajectory data time range according to the interaction event point and a preset duration; wherein the trajectory data time range includes the interaction event point; Determining second trajectory data of the first vehicle in the vehicle driving video frame based on the trajectory data time range; wherein the driving trajectory data of the first vehicle includes the second trajectory data; The first speed data is obtained according to the second trajectory data.

5. The method according to claim 1, characterized in that Before obtaining the starting position data and initial speed data of the second simulated vehicle, the method further includes: Obtaining second speed data of the second vehicle according to the driving trajectory data of the second vehicle in the vehicle driving video frame; obtaining second simulated speed data of a second simulated vehicle based on the second speed data; Acquiring initial speed data of the second simulated vehicle, comprising: The initial speed data of the second simulated vehicle is obtained according to the starting speed data of the second simulated speed data.

6. The method according to claim 5, characterized in that The first speed data is speed sample data of the first simulated vehicle; the second speed data is speed sample data of the second simulated vehicle; obtaining the first simulation speed data based on the first speed data, and obtaining the second simulation speed data based on the second speed data, comprises: Based on the first speed data and the second speed data, combined sample data is obtained for training an initial data generation model to obtain a trained simulation data generation model; The simulation data generation model is used to generate simulated vehicle data; wherein the simulated vehicle data includes the first simulation speed data and the second simulation speed data.

7. The method according to claim 6, characterized in that The driving trajectory data of the first vehicle is the trajectory sample data of the first simulated vehicle; the driving trajectory data of the second vehicle is the trajectory sample data of the second simulated vehicle; the combined sample data obtained based on the first speed data and the second speed data includes: The first speed data, the driving trajectory data of the first vehicle, the second speed data and the driving trajectory data of the second vehicle are combined to obtain the combined sample data; wherein the simulated vehicle data also includes the first simulated trajectory data of the first simulated vehicle and the second simulated trajectory data of the second simulated vehicle; the starting position data of the first simulated vehicle is obtained according to the starting coordinate data of the first simulation trajectory data; the starting position data of the second simulated vehicle is obtained according to the starting coordinate data of the second simulation trajectory data.

8. An automatic driving simulation scene generation device, characterized in that: The device comprises: A video frame acquisition module, used to acquire a vehicle driving video frame; wherein the vehicle driving video frame includes at least a first vehicle and a second vehicle having an interactive relationship; the interactive relationship indicates that a change in the driving state of the first vehicle has a direct impact on a change in the driving state of the second vehicle; A speed data acquisition module, configured to obtain first speed data of the first vehicle according to the driving trajectory data of the first vehicle in the vehicle driving video frame; A simulation speed acquisition module, used for obtaining first simulation speed data of a first simulation vehicle based on the first speed data; A scene trajectory acquisition module, used to perform trajectory calculation according to the starting position data of the first simulated vehicle and the first simulated speed data to obtain scene trajectory data of the first simulated vehicle; A simulated vehicle data acquisition module, used to acquire the starting position data and initial speed data of the second simulated vehicle; wherein the change of the driving state of the first simulated vehicle has an impact on the change of the driving state of the second simulated vehicle; A simulation scene generation module is used to generate the automatic driving simulation scene based on the scene trajectory data, the starting position data and the initial speed data of the second simulated vehicle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images