Scenario generation method and related apparatus

Abstract

A scenario generation method and a related apparatus, applied to the field of intelligent driving. According to the embodiments of the present application, motion information of obstacles can be obtained on the basis of scenario data of a first driving scenario; for a first obstacle in the first driving scenario, motion information of the first obstacle is adjusted by means of controllable modification; for a second obstacle, motion information of the second obstacle is jointly optimized and updated by means of the motion information of the first obstacle and original motion information of the second obstacle; and a target driving scenario can be generated on the basis of the updated motion information of the first obstacle and the updated motion information of the second obstacle. In conclusion, according to the solution of the present application, some obstacles in the first driving scenario can be controllably adjusted, and joint optimization is carried out on some obstacles for adaptive adjustment, such that controllable modification of the scenario is realized, and the interactivity between obstacles in the scenario is improved while the authenticity of the generated scenario is ensured.

Description

A scene generation method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 26, 2023, with application number 202311251719.8 and application name “A scene generation method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of intelligent driving technology, and in particular to a scene generation method and related devices. Background Art

[0003] To ensure safety, vehicles must be tested before they are released to the market. Simulation testing offers advantages such as high coverage, process safety, and high efficiency, making it widely used for testing intelligent driving devices. Currently, various manufacturers have released a large number of simulators for both private and commercial use.

[0004] The ability to construct test scenarios is an important factor in measuring the quality of simulators. When constructing test scenarios, some simulators generate driving scenarios by manually editing objects in the scenarios, while some simulators collect real road test data and convert it into driving scenarios. However, the scenarios edited by the former may be difficult to appear in real life and lack authenticity. While converting scenes by road test data can ensure the authenticity of the scenarios, it takes a lot of time to collect real road test scenarios. This method of generating driving scenarios lacks controllability of the scenarios, and when new interactive scenarios need to be designed, new scenarios need to be collected again to obtain new scenarios. For example, when a new action (such as braking or changing lanes) needs to be added to the vehicle in front to test the intelligent driving function of the vehicle itself, the actor car often needs to be driven again to collect its corresponding road test data to form a new scenario.

[0005] In summary, how to improve the controllability of the scene without significantly reducing the realism of the scene is a hot issue that technicians in this field are currently studying.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a scene generation method and related devices, which can achieve controllable modification of the scene and improve the interactivity between obstacles in the scene while ensuring the authenticity of the generated scene.

[0008] In a first aspect, an embodiment of the present application provides a scene generation method, the method comprising:

[0009] Acquiring scene data of a first driving scene, where the scene data is used to indicate initial motion information of obstacles in the first driving scene, where the obstacles in the first driving scene include a first obstacle and a second obstacle;

[0010] Adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle;

[0011] Obtaining second motion information of the second obstacle based on the initial motion information of the second obstacle and the first motion information of the first obstacle;

[0012] A target driving scene is generated according to the first motion information of the first obstacle and the second motion information of the second obstacle.

[0013] The embodiment of the present application obtains the motion information of multiple obstacles based on the scene data of the first driving scene, and one or more new scenes can be constructed by updating the motion information of the obstacles. However, different updating methods can be used for different obstacles. Take the example of an obstacle that includes at least a first obstacle and a second obstacle. For the first obstacle, the embodiment of the present application adjusts its motion information through controllable modification, that is, the motion information of the first obstacle can be modified manually, automatically, or even based on the modification of an artificial intelligence (AI) model to adjust the motion information of the first obstacle. For the second obstacle, the embodiment of the present application updates the motion information of the second obstacle through the initial motion information of the second obstacle and the information of other objects in the scene. This updating process can also be called joint optimization. Among them, the information of other objects in the first driving scene can include the updated motion information of the first obstacle, and can also include the motion information of the vehicle itself.

[0014] On the one hand, the process of updating the second obstacle's motion information takes into account its original trajectory, making the updated motion information of the second obstacle (i.e., the second motion information) more realistic. On the other hand, the process of updating the second obstacle's motion information takes into account the impact of changes in information of other objects in the scene, thus establishing a connection between the second obstacle and other objects in the scene, making the scene update process more interactive. In short, this solution can achieve controllable modification of the scene and improve the interactivity between obstacles in the scene while ensuring the authenticity of the generated scene.

[0015] Furthermore, the number of first motion information can be multiple. That is, the motion information of the first obstacle can be adjusted to generate multiple first motion information pieces, each of which is used to describe multiple possible motion processes of the first obstacle. Accordingly, the number of second motion information pieces can also be multiple groups. For example, when generating a target driving scenario, the scenario construction device can obtain a corresponding set of second motion operation information for each first motion information of the first obstacle. This allows for the rapid construction of multiple new driving scenarios (i.e., there can also be multiple target driving scenarios), significantly improving the efficiency of scenario construction. Furthermore, these multiple driving scenarios can meet the user's requirements for scenario realism, controllability, and interactivity.

[0016] In conjunction with the first aspect, in one possible implementation of the first aspect, the scene data further includes initial motion information of the ego vehicle. For example, the initial motion information of the ego vehicle further includes an initial motion trajectory of the ego vehicle. Optionally, the initial motion information of the ego vehicle can be collected by sensors of the ego vehicle, such as one or more of a wheel speedometer, an inertial measurement unit, a positioning system, or the like.

[0017] In combination with the first aspect, in a possible implementation of the first aspect, obtaining second motion information of the second obstacle according to the initial motion information of the second obstacle and the first motion information of the first obstacle includes:

[0018] Second motion information of the second obstacle is obtained according to the initial motion information of the vehicle, the initial motion information of the second obstacle, and the first motion information of the first obstacle.

[0019] In this embodiment, the motion information of the second obstacle can also be optimized in combination with the motion information of the vehicle, further improving the interactivity and authenticity of the scene process.

[0020] In conjunction with the first aspect, in another possible implementation of the first aspect, the degree of interaction between the first obstacle and the ego vehicle is no less than the degree of interaction between the second obstacle and the ego vehicle. The degree of interaction refers to the degree of temporal and spatial overlap between the motion trajectories of obstacles or the ego vehicle. For example, if, at a certain moment or time period, an obstacle and the ego vehicle have a trajectory interaction point, when the obstacle or the ego vehicle is at the trajectory interaction point, the closer the obstacle's position is to the ego vehicle's position, the higher the degree of interaction between the obstacle and the ego vehicle. If the obstacle and the ego vehicle do not have a trajectory interaction point, the shorter the distance between the obstacle and the ego vehicle at their closest point, the higher the degree of interaction.

[0021] Understandably, users have higher design requirements for the motion of obstacles that strongly interact with the ego vehicle. Therefore, the first obstacle, which has a high degree of interaction with the ego vehicle, has its motion information independently and controllably adjusted through controllable modification. Meanwhile, the second obstacle has its motion information updated through joint optimization, combining the updated motion information of the first obstacle. This approach better meets the demand for flexible motion design for highly interactive obstacles and enhances the user experience.

[0022] In conjunction with the first aspect, in another possible implementation of the first aspect, adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes: adjusting the initial motion information of the first obstacle based on user input to generate the first motion information of the first obstacle. In this implementation, the first obstacle can be an obstacle selected by the user for adjustment. That is, this solution supports user-controllable modification of designated obstacles, increasing the flexibility of scene updates and further enhancing the user experience.

[0023] In conjunction with the first aspect, in another possible implementation of the first aspect, adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes: adjusting the initial motion information of the first obstacle using an artificial intelligence model to generate the first motion information of the first obstacle. In this implementation, the first obstacle can be automatically selected by the artificial intelligence model, and the artificial intelligence model automatically adjusts the motion information of the obstacle. This not only improves the efficiency of updating motion information and obtaining the target scene, but also enhances the flexibility and intelligence of scene construction, thereby improving the user experience.

[0024] In conjunction with the first aspect, in another possible implementation of the first aspect, the second obstacle is an obstacle that interacts with the updated first obstacle. In this case, the updated motion information of the first obstacle can be used when updating the motion information of the second obstacle, thereby more accurately and reasonably updating the motion information of the second obstacle.

[0025] In conjunction with the first aspect, in another possible implementation of the first aspect, the second obstacle is an obstacle other than the first obstacle that interacts with the ego vehicle. In this case, the ego vehicle's motion information can be used when updating the second obstacle's motion information, thereby more accurately and reasonably updating the second obstacle's motion information.

[0026] In combination with the first aspect, in another possible implementation of the first aspect, the method further includes:

[0027] Determining obstacle categories based on interaction levels between the obstacles and the ego-vehicle and between obstacles in the first driving scene, where the obstacle categories include first-category obstacles and second-category obstacles;

[0028] The interaction degree between the first type of obstacles and the ego vehicle is higher than that between the second type of obstacles and the ego vehicle.

[0029] As such, since users have a greater need for controlled modification of more interactive obstacles, the above implementation enables controlled modification of the first type of obstacles and joint optimization of the second type of obstacles, meeting this need for controlled modification. Especially when obstacles are modified through user operation or AI models, distinguishing between the first and second types of obstacles helps users or AI models select obstacles for modification and joint optimization, thereby improving the user experience.

[0030] Optionally, the first obstacle belongs to the first category of obstacles, and the second obstacle includes at least one obstacle from the second category and / or includes at least one obstacle from the first category other than the first obstacle. In some embodiments, the first category of obstacles may be referred to as strongly interactive obstacles, and the second category of obstacles may be referred to as weakly interactive obstacles.

[0031] In combination with the first aspect, in yet another possible implementation of the first aspect, the obstacles in the first driving scene include first-category obstacles and second-category obstacles, wherein:

[0032] The first type of obstacle is an obstacle whose interaction with the vehicle meets the first condition.

[0033] The second category of obstacles includes obstacles whose interaction with the ego-vehicle satisfies the second condition and / or obstacles that interact with the third obstacle. The third obstacle is at least one obstacle in the first category, where the first condition differs from the second condition. Furthermore, the first obstacle belongs to the first category, and the second obstacle includes at least one obstacle in the second category and / or at least one obstacle in the first category other than the first obstacle.

[0034] In combination with the first aspect, in another possible implementation of the first aspect, the first condition includes: a time interval between the obstacle and the ego-vehicle passing through the trajectory interaction point falls within a first time range;

[0035] The second condition includes: the time interval between the obstacle and the ego-vehicle passing the trajectory interaction point falls within a second time range, and the lower bound of the second time range is greater than the upper bound of the first time range;

[0036] There is an interaction between the obstacle and the third obstacle when they meet the following conditions:

[0037] There is a trajectory interaction point between the obstacle and the third obstacle, and the time interval between the two passing through the trajectory interaction point falls within the third time range.

[0038] The above implementation provides characteristics for first- and second-type obstacles. It can be seen that first-type obstacles and the ego vehicle's trajectory interaction points are closer in time, while second-type obstacles are farther away. Since closer time intervals increase the impact of obstacles on the ego vehicle's behavior, obstacles with closer time intervals have a higher degree of interaction with the ego vehicle. This allows for quantification of the degree of interaction between the ego vehicle and obstacles, distinguishing between different obstacle types, and helping to select obstacles requiring modification and joint optimization, thereby improving the user experience.

[0039] In conjunction with the first aspect, in another possible implementation of the first aspect, the first condition includes: a trajectory of the obstacle and the trajectory of the ego-vehicle have a trajectory interaction point, and a first time interval between the two passing through the trajectory interaction point is the first N time intervals in the time interval sequence, where N is an integer greater than 0, and the first N time intervals in the time interval sequence are less than or equal to the time intervals other than the first N time intervals in the time interval sequence;

[0040] The second condition includes: there is a trajectory interaction point between the obstacle's trajectory and the vehicle's trajectory, and the first time interval between the two passing the trajectory interaction point falls within the first N+1 to K intervals of the time interval sequence, where K is an integer and K>N;

[0041] The time interval sequence includes M time intervals arranged in order of duration. Each of the M time intervals is the interval between the moment when an ego-vehicle interaction obstacle passes through the first trajectory interaction point with the ego-vehicle and the moment when the ego-vehicle passes through the first trajectory interaction point. The ego-vehicle interaction obstacle is an obstacle that has a trajectory interaction point with the ego-vehicle in the first driving scenario. M is an integer and M ≥ K.

[0042] The above embodiment provides another feature of the first and second categories of obstacles. The first and second categories of interactive obstacles are determined by sorting the degree of interaction. In this way, relatively strong and relatively weak obstacles can be obtained in any scenario without being restricted by a fixed time range. This can improve the compatibility and flexibility of the obstacle classification scheme, facilitate the selection of obstacles that need to be modified and obstacles that need to be jointly optimized, and enhance the user experience.

[0043] In conjunction with the first aspect, in another possible implementation of the first aspect, the initial motion information of the first obstacle includes multiple behavior parameters of the first obstacle, where the multiple behavior parameters are used to describe the behavior of the first obstacle. Adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes adjusting values ​​or thresholds of the multiple behavior parameters of the first obstacle to generate the first motion information of the first obstacle.

[0044] It should be understood that behavior here is broadly defined, encompassing the behavior and state of an entity at a specific moment, or actions performed over a sustained period of time. This implementation supports flexible updating of the first obstacle's motion process by updating the values ​​or thresholds of behavior parameters, and supports determining behavior parameters at a specific moment or stage of motion. This improves the precision of obstacle information editing and enables scenarios that better meet user needs.

[0045] In conjunction with the first aspect, in another possible implementation of the first aspect, the initial motion information of the first obstacle includes a first action set and a first trigger set of the first obstacle, the first action set being used to describe a first action of the first obstacle, the first action set including values ​​of first behavior parameters of the first action, and the first trigger set being used to describe conditions for executing the first action;

[0046] Adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes:

[0047] The first action set and / or the first trigger set of the first obstacle are adjusted to generate the first motion information of the first obstacle. In the above embodiment, the behavior parameters of the first obstacle can be integrated into the action set and the trigger set, which can support the modification of the behavior parameter set within a certain stage and improve the user experience.

[0048] In combination with the first aspect, in another possible implementation manner of the first aspect, the initial motion information of the first obstacle includes an initial motion trajectory of the first obstacle;

[0049] Adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes:

[0050] The initial motion trajectory of the first obstacle is adjusted to obtain a first motion trajectory, and the first motion information includes the first motion trajectory.

[0051] The initial motion trajectory may include multiple trajectory points, which may indicate a first position change of the first obstacle within a first time period. Similarly, the first motion trajectory may also include multiple trajectory points, which may indicate that the first motion trajectory is used to indicate a second position change of the first obstacle within the first time period.

[0052] In the above embodiment, the motion information of the obstacle can be updated by editing the motion trajectory. Especially when only the trajectory needs to be adjusted to generate a new scene, the efficiency of updating the motion information can be significantly improved, thereby enhancing the user experience.

[0053] In combination with the first aspect, in another possible implementation of the first aspect, the initial motion trajectory of the first obstacle includes multiple trajectory points, and adjusting the initial motion trajectory of the first obstacle to obtain the first motion trajectory includes:

[0054] receiving a user input for an update operation on some or all of the plurality of track points;

[0055] According to the updating operation, a first motion trajectory is obtained.

[0056] This implementation supports user input operations to update the motion information of the first obstacle, improves the flexibility of obstacle updates, and enhances user experience.

[0057] In conjunction with the first aspect, in another possible implementation of the first aspect, the scene data further includes an initial motion trajectory of the vehicle. The first motion trajectory is obtained according to an update instruction for the initial motion trajectory of the first obstacle, including:

[0058] Determining a trajectory intersection point between the initial motion trajectory of the first obstacle and the initial motion trajectory of the ego vehicle;

[0059] According to the trajectory interaction points, derived trajectory points are obtained. The derived trajectory points are located on the initial motion trajectory of the vehicle and are different from the positions of the trajectory interaction points.

[0060] According to the derived trajectory points, a first motion trajectory is obtained.

[0061] This implementation method can quickly obtain a new first motion trajectory by derivation of trajectory interaction points, which can significantly improve the efficiency of updating motion information, enhance the user experience, and improve the generalization of generated scenarios.

[0062] In combination with the first aspect, in another possible implementation of the first aspect, obtaining second motion information of the second obstacle based on the initial motion information of the second obstacle and the first motion information of the first obstacle includes:

[0063] Second motion information of the second obstacle is determined based on the initial motion information of the second obstacle, the cost of the second motion information of the second obstacle, and the first motion information of the first obstacle.

[0064] In the above embodiment, the cost can be evaluated when the motion information of the second obstacle is updated, so that the motion information with the lowest adjustment cost can be selected as the second motion information of the second obstacle as much as possible, thereby improving the rationality of the generated scene.

[0065] Optionally, the cost of the second motion information includes at least one of a authenticity cost, a smoothness cost, an obstacle avoidance cost, and a scene design cost.

[0066] Specifically, the second motion information includes a second motion trajectory of the second obstacle, and the second motion trajectory includes multiple trajectory points. The authenticity cost is used to indicate the difference between the second motion information of the second obstacle and the initial motion information of the second obstacle, and the authenticity cost is related to the coordinates, orientation, curvature and speed of the multiple trajectory points. The smoothness cost is used to indicate the smoothness of the position change of multiple trajectory points, and the smoothness cost is related to the curvature and acceleration of the second motion trajectory. The obstacle avoidance cost is used to indicate the obstacle avoidance amplitude of the second obstacle. The scenario design cost is a predefined scenario cost design. Exemplarily, the scenario design cost is used to flexibly adjust the cost of making decisions for the vehicle in different test scenarios. For example, corresponding scenario design costs can be predefined for difficult scenarios, special weather scenarios, special terrain scenarios, etc.

[0067] In combination with the first aspect, in another possible implementation of the first aspect, the cost J of the second motion information of the second obstacle is i It is expressed by the following formula:

[0068] J i =W X (X i -X ref ) 2 +W k k i 2 +W u u i 2 +W a a i 2 +W s slack i 2 +W f f hard 2

[0069] Among them, X iThe second motion information of the second obstacle i can be represented by the position, direction, curvature and speed of the m trajectory points. m is the number of trajectory points, and [x, y, phi, k, v] are the horizontal coordinate, vertical coordinate, orientation angle, curvature and velocity of the trajectory points respectively.

[0070] W X ,W k ,W u ,W a ,W s ,W f Used to represent coefficients. W X (X i -X ref ) 2 For authenticity cost, X ref is the initial motion information of the second obstacle i, which may also include the position, orientation, curvature and speed of multiple trajectory points, which will not be described here one by one. k k i 2 +W u u i 2 +W a a i 2 is the smoothness cost, k i represents the curvature of the second obstacle i, a i is the acceleration of the second obstacle i, u i is the derivative of the curvature of the second obstacle i (or the rate of change of curvature), W s slack i 2 is the obstacle avoidance cost, W f f hard 2 It is the cost of scenario design. Here, the difficult example cost is the design cost under the difficult example scenario.

[0071] In combination with the first aspect, in another possible implementation of the first aspect, the second motion information satisfies one or more of the following constraints: a kinematic constraint, a collision constraint, or a driving regulation constraint, wherein:

[0072] The kinematic constraint is used to indicate the value range of the following behavior parameters in the second motion information: one or more of the velocity, acceleration, curvature of the motion trajectory, and the rate of change of the curvature of the motion trajectory;

[0073] The collision constraint is used to constrain the second obstacle to have no collision with the vehicle and the obstacles in the first driving scene.

[0074] The driving specification constraint is used to constrain the behavior of the first obstacle to meet predefined driving specifications.

[0075] In conjunction with the first aspect, in another possible implementation of the first aspect, the scene data of the first driving scene further includes initial motion information of a fourth obstacle, where the fourth obstacle has no interaction with the ego vehicle and the first obstacle. Generating a target driving scene based on the first motion information of the first obstacle and the second motion information of the second obstacle includes:

[0076] A target driving scene is generated according to the first motion information of the first obstacle, the second motion information of the second obstacle, and the initial motion information of the fourth obstacle.

[0077] In a second aspect, an embodiment of the present application provides a scene generation device, which includes a data acquisition unit and a processing unit. The scene generation device is used to implement any method described in the first aspect.

[0078] In a third aspect, an embodiment of the present application provides a computing device, which includes a memory and a processor, wherein the memory is used to store program instructions; when the processor executes the program instructions in the memory, the computing device implements any method described in the first aspect.

[0079] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on at least one processor, the method described in any one of the first aspects above is implemented.

[0080] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the instructions are executed by at least one processor, the method described in any one of the first aspects is implemented.

[0081] Optionally, the computer program product may be a software installation package. A device or apparatus may download the computer program product and execute the computer program product on the device or apparatus, thereby implementing the aforementioned method.

[0082] The beneficial effects of the technical solutions provided in the second to fifth aspects of this application can refer to the beneficial effects of the technical solution in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The following is a brief introduction to the drawings required for describing the embodiments.

[0084] FIG1 is a schematic diagram of a scenario of a drive test data collection system;

[0085] FIG2 is a schematic diagram of the architecture of a scene generation system provided in an embodiment of the present application;

[0086] FIG3 is a flow chart of a scene generation method provided in an embodiment of the present application;

[0087] FIG4 is a schematic diagram of a driving scenario provided in an embodiment of the present application;

[0088] FIG5 is a schematic diagram of an obstacle category provided in an embodiment of the present application;

[0089] FIG6 is a schematic diagram of two trajectory interaction points provided in an embodiment of the present application;

[0090] FIG7 is a schematic diagram of two motion processes of a vehicle and a first obstacle provided in an embodiment of the present application;

[0091] FIG8 is a schematic diagram of an interactive phase of a cut-in scenario provided by an embodiment of the present application;

[0092] FIG9 is a schematic diagram of a trajectory change provided by an embodiment of the present application;

[0093] FIG10 is a schematic diagram of a scenario for updating motion information of an obstacle provided by an embodiment of the present application;

[0094] FIG11A is a schematic diagram of another scenario of updating the motion information of an obstacle provided by an embodiment of the present application;

[0095] FIG11B is a schematic diagram of another scenario of updating the motion information of an obstacle provided by an embodiment of the present application;

[0096] FIG12 is a schematic diagram of a target driving scenario provided in an embodiment of the present application;

[0097] FIG13 is a schematic structural diagram of a scene generation device provided in an embodiment of the present application;

[0098] FIG14 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] Before introducing the embodiments of the present application, the terms that may be used in the embodiments of the present application are first introduced.

[0100] 1. Obstacles are entities, such as objects, terrain, or facilities, that may delay or impede the movement of a terminal. Objects can include both living and inanimate objects. Furthermore, obstacles can be fixed or mobile.

[0101] 2. Intelligent driving, or autonomous driving, refers to a computer system that uses data collected by a perception system (such as multiple sensors) to identify or detect objects, generates results through calculation and analysis, and controls the terminal or assists a human driver or other decision-making body in controlling the terminal. An intelligent driving algorithm refers to an algorithm that can process data to generate output results, which can be used directly to control the vehicle and / or to assist other decision-making structures (such as human drivers) in controlling the vehicle. Exemplarily, the output results include but are not limited to one or more of obstacle recognition results, path planning, behavioral decisions (such as following, braking, steering, etc.), and motion planning (such as acceleration, deceleration, etc.). In some solutions, the intelligent driving algorithm is integrated into the intelligent driving system, and the meanings of the two can be interchangeable.

[0102] 3. The ego vehicle refers to the vehicle being observed during testing, evaluation, or demonstration. Optionally, the ego vehicle herein may be a test vehicle (or test vehicle), or any vehicle with a perception system. In some cases, intelligent driving algorithms or perception algorithms can be tested through vehicle testing. For example, when testing an intelligent driving algorithm, the actions and behaviors of the ego vehicle can be considered the actions and behaviors of the intelligent driving algorithm. In other words, the ego vehicle herein can be considered the vehicle itself, or it can be considered the intelligent driving algorithm.

[0103] The above terms may be optionally used in the following embodiments.

[0104] The following describes the scenario for collecting scene data used in this application. Figure 1 is a schematic diagram of a drive test data collection system, including a vehicle 10 and a computing device 20. Vehicle 10 may include a perception system 101, which may include, but is not limited to, one or more of a camera, millimeter-wave radar, lidar, ultrasonic radar, and position sensors, as well as posture sensors. Vehicle 10 may also be equipped with an intelligent driving algorithm.

[0105] While the vehicle 10 is driving, the perception system 101 can collect scene data. This scene information can include detection data of obstacles in the scene. As shown in FIG1 , obstacles here refer to objects in the driving scene. For example, obstacles can include inanimate objects, such as other vehicles or roadblocks. More specifically, obstacles can include animate objects, such as pedestrians or animals.

[0106] The scene data collected during driving can be provided to the computing device 20. The computing device 20 is a device with data processing capabilities that can generate scenarios from the scene data. The generated scenarios can be considered as simulations of the vehicle's driving environment and can be used to test algorithms or devices, or as model training data. For example, the generated scenarios can be used to test intelligent driving algorithms. In another example, the generated scenarios can be used to test detection devices such as lidar and millimeter-wave radar. In another example, the generated scenarios can be used to train models such as obstacle recognition models.

[0107] Optionally, the computing device may be a physical device, such as one or more of a host, a laptop, a handheld terminal, a computer room device, a server, etc. Alternatively, the computing device may be a virtual device, for example, the computing device may be a virtual machine, a container, a cloud, etc. The aforementioned servers include but are not limited to general-purpose computers, cloud servers, or blade servers, etc. When the functions of the computing device are implemented by a server, the number of servers it contains may be one or more (such as a server cluster). A virtual machine is a virtualized computing module. The cloud is a software platform that uses application virtualization technology, which enables one or more software and applications to be developed and run in an independent virtualized environment. Optionally, the cloud can be deployed on a public cloud, a private cloud, or a hybrid cloud, etc.

[0108] In some embodiments, the computing device may include a processor, where the processor includes but is not limited to a central processing unit, a graphics processor, an artificial intelligence processor, a microprocessor, or a programmable logic gate array.

[0109] FIG2 is a schematic diagram of the architecture of a scene generation system provided in an embodiment of the present application, wherein the scene generation system includes a perception system 101 and a scene generation device 201 .

[0110] The perception system 101 is used to perceive the environmental information around the autonomous driving vehicle during real-time road testing (such as the position and speed information of surrounding vehicles, pedestrians, bicycles, etc., traffic light information, roadside fences, etc.). Please also refer to the relevant introduction in Figure 1.

[0111] The scene generation device 201 is a device with data processing capabilities, capable of performing one or more operations including editing, updating (e.g., modification, generalization, etc.), and playing a scene. In some solutions, the aforementioned operations can be integrated into a simulator. Therefore, the scene generation device 201 in this solution can be a simulator, such as an autonomous driving simulator.

[0112] As a possible implementation, the scene generation device 201 may include one or more of a scene editing module, a scene generalization module, and a scene playback module. The term "module" here is used to describe the functionality of the scene generation device and does not limit the structure of the scene generation module. The scene editing module is used to update obstacles in the scene, for example, editing the motion information (such as motion trajectory, position, speed, acceleration, orientation, and movement) of obstacles in the scene. The scene generalization module is used to batch-generate obstacle trajectories or behavior parameters (for example, generalizing a single trajectory point to obtain multiple derived trajectory points). The scene generalization module can be used to generalize and modify a single scene data to generate a batch of new scenes. Behavior parameters refer to parameters that can describe the behavior or state of an obstacle, such as one or more parameters including velocity, acceleration, curvature, displacement, and motion mode (e.g., uniform motion, accelerated motion until accelerating to a target speed, decelerated motion until decelerating to a target speed, etc.). It may further include parameters derived from one or more of the aforementioned parameters, such as the rate of change of curvature. The scene playback module is mainly used to simulate and reproduce existing driving scenes. For example, according to the movement timeline of obstacles in the scene, it plays the images of objects in the scene changing over time.

[0113] Optionally, the scene generation system may further include a memory 202, which may store data from the perception system 101, such as structured storage of information sensed by onboard sensors. Furthermore, the memory 202 may be provided within the scene generation device 201. In some implementations, the scene generation device 201 and the memory 202 may be implemented within the computing device 20.

[0114] The following describes the method of an embodiment of the present application. Please refer to Figure 3, which is a flowchart illustrating a scene generation method provided by an embodiment of the present application. Optionally, this method is applied to the computing device 20 shown in Figure 1, or to the scene generation apparatus 201 shown in Figure 2. For ease of description, the following description uses the scene generation apparatus as an example.

[0115] The scene generation method shown in Figure 3 may include one or more steps from step S301 to step S304. It should be understood that for the convenience of description, the description is given in the order of S301 to S304, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. S301 to S304 are as follows:

[0116] Step S301: Acquire scene data of a first driving scene.

[0117] A driving scenario, or simply a scenario, generally refers to a sequence of scenarios that can be used to describe the interactions of a vehicle with other objects while performing a driving task. A scenario is a snapshot of entities, which can include representations of the entities themselves and connections between them. Entities include, but are not limited to, the vehicle, a static scene, a dynamic environment, an actor, and an observer. For example, a driving scenario may include one or more scenarios such as the vehicle traveling along a road, the vehicle passing an intersection and interacting with other vehicles and pedestrians, overtaking, turning, or changing lanes. It should be understood that the aforementioned explanations and distances are provided to facilitate understanding of scenarios and scenes and do not constitute limitations on this solution. Furthermore, the first driving scenario is a distinguishing description provided to facilitate the description of the driving scenario. Optionally, the first driving scenario can be a real scenario experienced by the vehicle during a road test, or a simulated scenario generated by processing a real scenario. In this case, the scene data for the first driving scenario can be real road test data, the collection process of which is shown in Figure 1. Alternatively, the first driving scenario can be a simulated scenario generated through manual or automatic editing.

[0118] The first driving scene includes obstacles. Obstacles are entities in the first driving scene that may affect the vehicle's progress. It should be noted that obstacles may not actually affect the vehicle's progress, but they have the potential or ability to affect the vehicle's progress. In this application, the number of obstacles in the first driving scene can be multiple. For example, the driving scene shown in Figure 1 includes obstacles such as a preceding vehicle, a roadblock, and pedestrians.

[0119] The scene data of the first driving scene may indicate motion information of entities in the first driving scene. The entities here include one or more of the aforementioned obstacles, the vehicle, etc. For ease of distinction, the motion information indicated in the scene data is referred to as initial motion information. Motion information, also known as action information or activity information, includes information describing the behavior and / or actions of the entities. The term "motion information" here takes into account that behaviors or actions may have temporal properties. For example, behavior can be broadly defined, encompassing actions at a specific point in time (e.g., acceleration, lane changing, etc.) or states (e.g., current vehicle speed, current vehicle location). Alternatively, a behavior (or action) may encompass the process of the behavior over a period of time (e.g., accelerating until the vehicle reaches a target speed). In some scenarios, behavior and action may be interchangeable. In short, motion information may describe the entity's motion process in the scene and / or its state at any moment during the motion process.

[0120] Exemplarily, the initial motion information may include one or more of the following parameters: one or more of trajectory, trajectory point, position, orientation, action, behavioral parameters, etc. Among them, behavioral parameters refer to parameters that describe the behavior, action or state of an object, for example, they may include one or more of speed, acceleration, curvature, displacement, mode of motion (for example, uniform motion, accelerated motion until accelerating to the target speed, decelerated motion until decelerating to the target speed, etc.). In some schemes, one or more of the aforementioned trajectory, trajectory point, position, orientation, or action may also be included in the behavioral parameters. It should be noted that the motion information may also include changes in the above parameters over time, for example, the trajectory may be represented by trajectory points (or position points) at multiple moments, or the trajectory may be represented by trajectory points, speed, speed direction, etc. at multiple moments.

[0121] Figure 4 is a schematic diagram of a driving scenario provided by an embodiment of the present application. The driving scenario includes multiple obstacles, denoted as obstacles o1-o9. The scene data of the driving scenario can indicate the trajectory (or position at a certain moment), direction, and speed of the obstacle. Alternatively, the scene data of the driving scenario can be processed (e.g., using logsim) to obtain the trajectory (or position at a certain moment), direction, and speed of the obstacle.

[0122] In this solution, the processing methods for multiple obstacles in the first driving scenario can differ. For example, if the first driving scenario includes a first obstacle and a second obstacle, different processing methods can be used to construct the target driving scenario. It should be understood that the first and second obstacles described herein are obstacles located in the same driving scenario.

[0123] As a possible implementation, there are differences in properties or features between the first obstacle and the second obstacle. The first obstacle and the second obstacle are described below as examples:

[0124] Case 1: The first obstacle and the second obstacle have different interaction levels with the ego vehicle. For example, the interaction level of the first obstacle with the ego vehicle is not lower than the interaction level of the second obstacle with the ego vehicle.

[0125] The degree of interaction can indicate the extent to which an obstacle affects the ego vehicle's driving over a period of time. In conjunction with Figure 4 , the movement of obstacle o8 can significantly affect the ego vehicle's driving. For example, if obstacle o8 is moving quickly, the ego vehicle needs to slow down in time to avoid a collision. On the other hand, if obstacle o8 stops at a distance from the ego vehicle, the ego vehicle can maintain a certain speed through the intersection.

[0126] In some possible scenarios, obstacles in the first driving scene can be divided into different categories based on the degree of interaction. Exemplarily, the category of obstacles can be determined based on the degree of interaction between the obstacles and the ego vehicle and the degree of interaction between obstacles in the first driving scene. For example, the categories of obstacles include first-category obstacles and second-category obstacles, and the degree of interaction between first-category obstacles and the ego vehicle is higher than the degree of interaction between second-category obstacles and the ego vehicle. Furthermore, based on the scene data of the first driving scene, the degree of interaction between entities in the first driving scene can be determined, where the degree of interaction between entities can include between obstacles and the ego vehicle, between obstacles, and so on. Optionally, the degree of interaction can be automatically identified by an interaction degree calculation algorithm, for example, the degree of interaction between entities can be determined based on a rule-based method (such as determining the order of arrival at trajectory interaction points) or a learning-based method.

[0127] The following examples illustrate several rule-based methods for determining the degree of interaction. In some approaches, the degree of interaction between an obstacle and the ego vehicle can be indicated by the degree of temporal and spatial overlap of the obstacle's trajectory with the ego vehicle's trajectory. For example, the closer the obstacle's position is to the ego vehicle's position at a certain moment or time period during the ego vehicle's travel, the higher the degree of interaction between the obstacle and the ego vehicle. For another example, if there is a trajectory intersection between the ego vehicle and the obstacle, the closer the time at which they pass through the trajectory intersection, the higher the degree of interaction between the obstacle and the ego vehicle. In Figure 4 , the ego vehicle is traveling straight ahead and its trajectory with obstacle o6 has a spatial intersection, meaning both vehicles pass through an intersection. However, obstacle o6 has already passed the intersection before the ego vehicle reaches it. While their trajectories overlap, the time at which they pass through the overlapping point is significantly different, so obstacle o6 can be considered to have no interaction with the ego vehicle. Similarly, obstacle o2 has a stronger degree of interaction with the ego vehicle than obstacle o6. Since obstacle o1 and the ego vehicle's trajectory have an intersection point, and the two pass through the intersection point very close in time, the degree of interaction between obstacle o1 and the ego vehicle is high. Learning-based methods can determine the degree of interaction by training models using training data, such as reinforcement learning, deep learning, or neural networks.

[0128] In some possible situations, the first obstacle belongs to a first category of obstacles, the second obstacle is an obstacle that interacts with the first obstacle, and / or the second obstacle is an obstacle that interacts with the ego vehicle in addition to the first obstacle.

[0129] In some possible scenarios, the first obstacle belongs to the first category of obstacles, and the second obstacle includes obstacles in the second category of obstacles and / or includes obstacles in the first category of obstacles other than the first obstacle. Furthermore, the first category of obstacles can be referred to as strongly interactive obstacles, and the second category of obstacles can be referred to as weakly interactive obstacles. FIG5 is a schematic diagram of an obstacle category provided in an embodiment of the present application. The first driving scenario may include strongly interactive obstacles o1, o5, and o8, weakly interactive obstacles o2, o3, o4, and o7, and non-interactive obstacles o6 and o9. It should be understood that the strength here can be absolute. For example, some solutions support using quantitative standards to evaluate the strength of the interaction between obstacles and the vehicle. Alternatively, the strength can be relative. For example, in some solutions, the present application supports sorting the interaction between the obstacles in the first driving scenario and the vehicle, taking the top N obstacles with the highest degree of interaction as strongly interactive obstacles, and taking the top N+1 to M obstacles as weakly interactive obstacles, where N and M are both positive integers and M>N.

[0130] In some possible scenarios, obstacles in the first driving scenario include a first category of obstacles and a second category of obstacles, where: the first category of obstacles are obstacles whose degree of interaction with the ego vehicle satisfies a first condition; the second category of obstacles include obstacles whose degree of interaction with the ego vehicle satisfies a second condition and / or obstacles that interact with a third obstacle, where the third obstacle is at least one obstacle in the first category, and the first condition is different from the second condition.

[0131] The following is a detailed introduction to the first and second conditions, as well as the conditions under which interaction between two obstacles is indicated:

[0132] Design 1: The first condition includes the time interval between the obstacle and the ego vehicle passing through the trajectory interaction point falling within a first duration range. A trajectory interaction point here can be a point where the trajectories overlap, or a location where the distance between the two trajectories is less than or equal to a distance threshold. See Figure 6 for a schematic diagram of two trajectory interaction points provided in an embodiment of the present application. As shown in part (a) of Figure 6, the location where the trajectory of obstacle o1 overlaps with the ego vehicle's is interaction point 601. As shown in part (b) of Figure 6, the trajectory of obstacle o5 does not overlap with the ego vehicle's trajectory. The closest distance between the two trajectories is d0. If d0 is less than a preset distance threshold, the points on the two trajectories corresponding to this distance are considered trajectory interaction points. In other words, if positions 602 and 603 are located on the trajectories of the ego vehicle and obstacle o5, respectively, and the distance between them is less than a preset distance threshold, then positions 602 and 603 serve as trajectory interaction points for the ego vehicle and obstacle o5, respectively. The distance threshold can be pre-set or custom, such as 5 meters (m), 3m, or 1m.

[0133] Exemplarily, the first duration range can be a range that is less than or equal to the first duration, such as less than 5 seconds (s), less than or equal to 7s, etc. In conjunction with part (a) of Figure 6, taking the first duration range of less than or equal to 5s as an example, if the time when the vehicle passes through the trajectory interaction point 601 is t1, and the time when the obstacle o1 passes through the trajectory interaction point 601 is t2, if the time interval between t2 and t1 is less than or equal to 5s, then the obstacle o1 belongs to the first type of obstacle. Furthermore, the first duration range may also have a lower bound. For example, the first duration range may be a range that is greater than or greater than or equal to the second duration, and less than or less than or equal to the first duration. For example, the first duration range may be (0.5s, 5s]. In some cases, a duration range of less than or equal to 0.5s can be used to determine other types of obstacles.

[0134] The second condition includes that the time interval between the obstacle and the vehicle passing through the trajectory interaction point falls within a second duration range, and the lower bound of the second duration range is greater than the upper bound of the first duration range. For example, the second duration range can be a duration range that is greater than or equal to or greater than a third duration, where the third duration is greater than or equal to the first duration. Exemplarily, the first duration range is less than 5s, and the second duration range is greater than or equal to 5s. Furthermore, the second condition may also include an upper bound, for example, the second duration range may be (5s, 12s].

[0135] An interaction exists between the obstacle and the third obstacle when the following conditions are met: the obstacle and the third obstacle have a trajectory interaction point, and the time interval between the two passing through the trajectory interaction point falls within a third duration range. For example, the third duration range is less than or equal to the fourth duration, such as less than 3 seconds or 5 seconds. For another example, the third duration range may include a lower bound, i.e., greater than or equal to the fifth duration, such as the fourth duration range of [0.5, 3 seconds].

[0136] Design 2, the first condition includes that the obstacle's trajectory and the vehicle's trajectory have a trajectory interaction point, and the first time interval between the two passing the trajectory interaction point is the first N time intervals in the time interval sequence, where N is an integer greater than 0.

[0137] The time interval sequence includes M time intervals arranged in order of duration, each of the M time intervals being the interval between the moment when a self-vehicle interactive obstacle passes through a first trajectory interaction point with the self-vehicle and the moment when the self-vehicle passes through the first trajectory interaction point. The self-vehicle interactive obstacle is an obstacle that has a trajectory interaction point with the self-vehicle in the first driving scene. Please refer to Table 1, which shows the interaction relationship between obstacles and the self-vehicle in a first driving scene provided by an embodiment of the present application. The first driving scene is the driving scene described in Figure 4, in which obstacles o3 and o9 do not have a trajectory interaction point with the self-vehicle and can be regarded as having no interaction with the self-vehicle. The remaining obstacles, namely obstacles o1, o2, o4, o5, o6, o7, and o8, all have trajectory interaction points with the self-vehicle, and therefore the above obstacles are self-vehicle interactive obstacles. The time intervals between the time when the aforementioned ego-vehicle interaction obstacles pass through their respective trajectory interaction points and the time when the ego-vehicle passes through the corresponding trajectory interaction points are shown in Table 1. Sorting the time intervals from smallest to largest, they form the following sequence (for ease of description, the corresponding obstacle is indicated in parentheses): 1.50s (o1), 2.80s (o5), 3s (o8), 5.01s (o2), 5.12s (o4), 5.15s (o7), 12.50s (o6), where M = 7. The obstacles ranked in the top three (i.e., N = 3) by time interval are considered Category I obstacles, namely, obstacles o1, o5, and o8. It is not difficult to see that the first N time intervals in the time interval sequence are less than (or less than or equal to) all other time intervals in the sequence. It should be understood that the above order is merely an example; if sorting were performed from largest to smallest, the first obstacle, which would be the last N obstacles in the sequence, would be Category I.

[0138] Table 1

[0139] Furthermore, the second condition includes that the obstacle's trajectory and the ego-vehicle's trajectory have an intersection point, and the first time interval between the two passing through the intersection point falls within the first N+1 to K intervals in the sequence, where K is an integer and K>N, and M≥K or M>K. Continuing with Table 1, the second category of obstacles can include the first 4 to 6 obstacles, that is, N=3 and K=6, meaning that the second category of obstacles can include obstacles o2, o4, and o7.

[0140] Furthermore, the second category of obstacles may also include obstacles that interact with the first category of obstacles, wherein the third obstacle is at least one of the first category of obstacles. The interactive obstacles herein may illustratively include obstacles that meet at least one of the following two conditions: an obstacle that has a trajectory interaction point with the third obstacle and the time interval between the trajectory interaction point falls within a third duration range, for example, the third duration range is 0-3 seconds; or multiple obstacles may be ranked by the degree of interaction between them and the third obstacle, thereby selecting one or more obstacles as obstacles that interact with the third obstacle in descending order of interaction degree.

[0141] For example, in conjunction with Figure 4, taking Case 1 as an example, the third obstacle may be obstacle o1 (o1 is a first-category obstacle), and there is a trajectory interaction point between obstacle o1 and obstacle o3, and the time interval between the two passing through the trajectory interaction point is less than the fourth time length (e.g., 5s). Therefore, there is an interaction between obstacle o1 and obstacle o3, and therefore obstacle o3 belongs to the second-category obstacle.

[0142] It should be understood that the above two designs are merely for the purpose of facilitating the description of obstacle classification. These two designs are described as examples for distinguishing obstacle types. These two designs can also be combined. One example combination is as follows: Obstacles corresponding to the first N time intervals within a first time range are selected from M time intervals as first-category obstacles. If fewer than N time intervals fall within the first time range, only obstacles whose corresponding time intervals fall within the first time range are selected as first-category obstacles. Of course, other combined designs are possible during implementation, and these are not exemplified here.

[0143] In some possible scenarios, the obstacles in the first driving scene may also include a third category of obstacles, also known as background obstacles. These third category obstacles have no interaction with either the ego vehicle or the first category of obstacles. Exemplarily, these third category obstacles have no track interaction points with the ego vehicle and no track interaction points with the first category of obstacles. In conjunction with Design 1, if obstacle o6 has a track interaction point with the ego vehicle, but the time interval between these track interactions exceeds the upper bound of the second duration range (i.e., 12 seconds), then obstacle o6 is considered a third category obstacle. Alternatively, in conjunction with Design 2, if obstacle o6 has a track interaction point with the ego vehicle, but the time interval between these track interactions does not fall within the top K positions, then obstacle o6 is considered a third category obstacle. Obstacle o9, however, has no track interaction points with either the ego vehicle or the first category of obstacles and is therefore a third category obstacle. In some possible scenarios, the third category obstacle may also be any obstacle in the first driving scene other than the ego vehicle, the first category of obstacles, or the second category of obstacles.

[0144] Optionally, the motion trajectory used in the aforementioned determination of the interaction type may be a motion trajectory within a predefined time period. For example, the vehicle may be continuously traveling, and the first driving scene may be a scene within a local time period within the duration of the driving.

[0145] Case 2: The first and second obstacles have different levels of interest. The level of interest is determined by the user or an algorithm. For example, the first obstacle is user-selected. Because it was selected (e.g., selected or edited) by the user, it has a higher level of interest than the other obstacles. For another example, the first and second obstacles are both user-selected, but the first obstacle's trajectory can be freely edited, while the second obstacle's trajectory does not require user editing. In this case, the first obstacle has a higher level of interest than the second obstacle.

[0146] As a possible implementation, the first obstacle can be selected by a user or an algorithm (e.g., a rule-based obstacle selection algorithm or a machine learning-based obstacle selection algorithm). For example, a user may select a rabbit obstacle, also known as a "cast car," from among multiple obstacles in a first driving scenario. The user can control the rabbit obstacle to cause it to perform a pre-defined interaction with the vehicle. In this case, the scenario generation device may receive the user's selection or editing operation for a particular obstacle or obstacles, and the selected obstacle becomes the first obstacle.

[0147] Optionally, in this case, the second obstacle can be user-specified or automatically determined. For example, the user can select some obstacles from a set of obstacles as the second obstacle. In another example, the second obstacle can be determined by: when the trajectory of the designated rabbit obstacle changes, obstacles that may conflict with it are used as the second obstacle, or obstacles other than the first obstacle that may conflict with the ego vehicle's trajectory are used as the second obstacle. In other words, the second obstacle is an obstacle that interacts with the first obstacle, or the second obstacle is an obstacle other than the first obstacle that interacts with the ego vehicle.

[0148] Furthermore, in scenario 2, the first driving scene may include background obstacles in addition to the first and second obstacles. Background obstacles are obstacles that do not interact with the first and second obstacles. In some implementations, background obstacles may be other obstacles in the first driving scene besides the first and second obstacles.

[0149] In case 3, the first obstacle can be any obstacle in the first driving scene, and the second obstacle can be an obstacle that interacts with the first obstacle, or the second obstacle can be an obstacle other than the first obstacle that interacts with the ego vehicle. Optionally, the first driving scene can also include background obstacles in addition to the ego vehicle, the first obstacle, and the second obstacle.

[0150] It should be understood that the above three scenarios are provided as examples to facilitate understanding of obstacles and are not intended to be strict limitations on the solution. The above two scenarios can also be combined. For example, the first obstacle can be a user-specified strong interaction obstacle, and the second obstacle can be a strong interaction obstacle and / or a weak interaction obstacle that conflicts with the trajectory of the first obstacle.

[0151] It should be understood that the first obstacle may be one or more obstacles, and the second obstacle may also be one or more obstacles. In addition, in conjunction with the above description of driving scene data, when the first driving scene includes the first obstacle and the second obstacle, the scene data of the first driving scene may include initial motion information of the first obstacle and initial motion information of the second obstacle.

[0152] As a possible implementation, the first driving scenario can be obtained based on data collected while the vehicle is driving. The scenario data of the first driving scenario can also include motion information of the vehicle. Optionally, the initial motion information of the vehicle can be collected by sensors of the vehicle, such as one or more of a wheel speedometer, an inertial measurement unit, a positioning system, a radar sensor, an image sensor, or the like.

[0153] Optionally, the scene data of the driving scene acquired in step S301 may be converted based on sensor data (e.g., the perception system 101 shown in Figures 1 and 2). For example, the sensor data may be subjected to logsim processing to obtain the motion information of the obstacle and the ego vehicle in the first driving scene.

[0154] Step S302: adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle.

[0155] Specifically, similar to the initial motion information, the first motion information may include one or more of the following parameters: one or more of the motion trajectory, trajectory points, position, orientation, action, and behavior parameters of the first obstacle.

[0156] The following is an exemplary introduction to the implementation method of adjusting the initial motion information of the first obstacle to obtain the first motion information:

[0157] Implementation Method 1: Adjust the values ​​or thresholds of the multiple behavior parameters of the first obstacle to generate first motion information of the first obstacle. For the description of the behavior parameters, refer to the above. The first motion information includes the updated values ​​of the multiple behavior parameters and / or the updated thresholds of the behavior parameters.

[0158] Taking the update of parameter values ​​as an example, referring to Table 2, the initial motion information may include the values ​​of parameters such as absolute speed (such as target speed), relative speed (such as relative to the vehicle speed), and relative distance. The update indication may include new values ​​of some or all parameters in the initial motion information (i.e., updated values). The scene generation device may obtain the first motion information according to the update indication, and the first motion information includes the new values ​​of some or all parameters in the aforementioned initial motion information. Of course, when the value of a certain parameter is not updated, the value of the parameter in the first motion information is the same as the value in the initial motion information. For example, the updated value of the obstacle model in Table 2 is the same as the value in the initial motion information.

[0159] Table 2

[0160] Please refer to Figure 7, which is a schematic diagram of two motion processes of a self-vehicle and a first obstacle provided in an embodiment of the present application. Referring to part (a) of Figure 7, the motion information of the first obstacle may include the behavior parameters of the first obstacle at time t4, such as a speed v = 40 kilometers per hour (km / h). The behavior parameters can be adjusted, referring to part (b) of Figure 7. After the adjustment, the speed of the first obstacle at time t4 is 35 km / h. For another example, the motion information of the first obstacle may include a threshold value of the behavior parameters for the first obstacle to change lanes. Referring to part (a) of Figure 7, in the first driving scenario, when the relative distance between the first obstacle and the self-vehicle reaches d1, the first obstacle performs a lane change operation. Referring to part (b) of Figure 7, the threshold value of the relative distance can be updated to d2, and then when the relative distance between the first obstacle and the self-vehicle is d2, the first obstacle performs a lane change operation. Furthermore, the obstacle model can also be replaced, for example, the lane change model after time t4 can be updated to an S-turn model.

[0161] As a possible implementation, the initial motion information of the first obstacle includes a first action set and a first trigger set, and the adjustment of the initial motion information of the first obstacle includes adjusting the behavior parameters in the first action set and / or adjusting the values ​​of the behavior parameters in the first trigger set. The first action set is used to describe the first action of the first obstacle and includes the values ​​of the first behavior parameters of the first action. The first trigger set is used to describe the conditions for executing the first action. Exemplarily, the first action may include one or more of changing lanes, overtaking, following a vehicle, turning, braking, accelerating, decelerating, reversing, entering the next action stage, etc. It should be understood that the parameters of the first action set may be different from the parameters in the first trigger set. Please refer to Table 3, which is information on behavior parameters that may be included in an action set provided in an embodiment of the present application, wherein a behavior label can be used to represent a behavior parameter, and its name and meaning can be found in the description of the first column of Table 3.

[0162] Table 3 Action set

[0163] Table 4 is a diagram of possible behavior parameter information included in a trigger set provided in an embodiment of the present application. Condition labels are used to represent behavior parameters in the trigger set. Their names and meanings can be found in the description of the first column of Table 4.

[0164] Table 4 Trigger Set

[0165] Accordingly, the scene generation device adjusts the first action set and / or the first trigger set of the first obstacle to generate the first motion information of the first obstacle. For example, in conjunction with Figure 7, the first action set can be the action set of the first obstacle at time t4, and the first action set can include absolute speed (i.e., objTargetSpeed ​​in Table 3), relative speed (i.e., objectRelSpeed ​​in Table 3), and running to the target speed with acceleration (objTargetSpeed ​​and objectAcc in Table 3). It is further possible to select the lane changing method of the first obstacle, set the interaction model satisfied by the first obstacle, etc. After updating the value of the first action set, as shown in part (b) of Figure 7, the motion process of the first obstacle changes accordingly. The same is true for the trigger set, and examples are not given here one by one.

[0166] In one scenario, the first action set and the first trigger set can be the action set and trigger set corresponding to any moment or stage of the first obstacle. In this case, this solution can support editing the state and behavior of the obstacle at any moment or stage of the first obstacle's movement, thereby increasing the flexibility of scene generation.

[0167] As another possible scenario, there is an interaction between the first obstacle and the ego vehicle, and the interaction between the first obstacle and the ego vehicle can be decomposed into at least one interaction stage. The action of the first obstacle in each interaction stage can be described by an action set, and the triggering method of the interaction stage can be described by a trigger set. The aforementioned first action set and first trigger set can be an action set and a trigger set in a certain interaction stage or in several interaction nodes. The division of the interaction stage can be related to the type of interaction, wherein the interaction type can include one or more of the following interaction types: cut-in, cross, merge, lane-change, etc. Please refer to Figure 8, which is a schematic diagram of the interactive stages of a cut-in scenario provided by an embodiment of the present application. The cut-in scenario can include a car chasing stage, a speed adjustment stage, and a cut-in execution stage. In the car chasing stage, the social car (regarded as the first obstacle) can be designed to accelerate so that the relative distance between it and the self-car reaches the chase distance, triggering entry into the next stage; in the speed adjustment stage, the social car can be designed to adjust its speed (for example, accelerate or decelerate) so that the relative distance between it and the self-car reaches the overtake distance, triggering entry into the next stage; in the cut-in stage, the social car can change lanes and insert itself in front of the self-car. It is not difficult to see that each stage can be represented by a corresponding action set (for example, speed, acceleration, motion model, etc.) and a trigger set (for example, relative distance, relative speed, etc.). That is, the first action set and the first trigger set can be the corresponding action set and trigger set in any one or more of the aforementioned stages.

[0168] In this case, the scenario generation device can identify the interaction category between the first obstacle and the ego vehicle based on the initial motion information of the first obstacle and the initial motion information of the ego vehicle, and determine at least one interaction stage based on the interaction category corresponding to the first obstacle. Furthermore, the scenario generation device adjusts the action set and trigger set parameters for the first interaction stage to obtain the first motion information of the first obstacle. The first interaction stage can be part or all of at least one interaction stage. Furthermore, when identifying the interaction category, the scenario generation device can identify the interaction category between the first obstacle and the ego vehicle based on the scenario data of the first driving scenario.

[0169] As a possible implementation, interaction type identification can be based on a rule-based method or a learning-based method. For example, Table 5 lists the interaction types and their characteristics. The scene generation device can use the motion information between the first obstacle and the ego vehicle. When the motion information between the first obstacle and the ego vehicle indicates that the motion relationship between the first obstacle and the ego vehicle satisfies a certain characteristic, the interaction type is the interaction type corresponding to the characteristic. For example, if the initial motion information of the obstacle and the initial motion information of the ego vehicle determine that the obstacle cuts in from the lane parallel to the ego vehicle to the front of the ego vehicle, then the interaction type between the obstacle and the ego vehicle is a cut-in type.

[0170] Table 5

[0171] Furthermore, each interaction type may include one or more interaction stages. For example, merge may include two stages: speed regulation and import, and lane-change may include one stage. It should be noted that Table 5 is an exemplary interaction type and its characteristics made to facilitate understanding of the identification of interaction types. In the specific implementation process, the identification of interaction types can also be achieved based on other rules. In addition, the number and names of interaction stages corresponding to the interaction type can also have other designs, for example, more or fewer names can be included. This application does not strictly limit the interaction type, its characteristics, and the included interaction stages.

[0172] As a possible implementation, the above adjustment can be based on user input. For example, the user manually inputs the updated behavior parameter value or threshold. Accordingly, the scene generation device receives the user input of the aforementioned behavior parameter value or threshold.

[0173] As another possible implementation, the above adjustment can be obtained through the output of an algorithm or AI model. For example, the scene generation device can receive the values ​​of multiple sets of behavioral parameters (such as action sets, trigger sets, etc.) output by the algorithm or AI model. At this time, the first motion information of the first obstacle can be obtained based on the values ​​of each set of behavioral parameters. Among them, the values ​​of the multiple sets of behavioral parameters output by the model are different from the values ​​of the original behavioral parameters, thereby changing the motion process of the first obstacle. For example, in the original driving scene (i.e., the first driving scene), the first obstacle turns left at a speed of 30 yards, and the AI ​​model can adjust the speed when turning left to 25, 35, 40, 45, and 50 yards, thereby obtaining at least 4 updated motion information, so that in the at least 4 new scenes obtained, the first obstacle has more possible turning radius when turning left. In short, the algorithm or AI can realize the generalization modification of the behavioral parameters of the first obstacle, so that the motion information of multiple obstacles can be quickly generalized from one driving scene, so that multiple derivative scenes can be quickly obtained subsequently. Exemplarily, in conjunction with FIG. 2 , the generalization process may be implemented by a scene generalization module in the scene generation device 201 .

[0174] Implementation method 2: The initial motion trajectory of the first obstacle is adjusted to obtain a first motion trajectory, and the first motion information includes the first motion trajectory.

[0175] Please refer to Figure 9, which is a schematic diagram of a trajectory change provided in an embodiment of the present application, wherein the solid line with an arrow is the initial motion trajectory of the obstacle, and the dotted line with an arrow is the second motion trajectory (i.e., the updated motion trajectory). Taking obstacle o5 as an example, its updated motion trajectory has a larger turning radius.

[0176] In some scenarios, a motion trajectory includes multiple trajectory points, and the position of at least one of the multiple trajectory points can be updated to form a new trajectory.

[0177] As a possible implementation, the update of trajectory points can be indicated by an operation input by the user. For example, the scene construction device can receive a user input update operation for some or all of the multiple trajectory points, and obtain a first motion trajectory based on the update operation. Please refer to Figure 10, which is a schematic diagram of a scene for updating the motion information of an obstacle provided in an embodiment of the present application. The solid arrow represents the initial motion trajectory of obstacle o5, which may include multiple trajectory points, such as trajectory point 1001. The scene generation device can output an operation interface, which may include the initial motion trajectory of obstacle o6 and the trajectory points on that trajectory. The user can perform operations on the operation interface, such as one or more of information input, click, drag, and long press operations. Information input operations include inputting the horizontal coordinate, vertical coordinate, or distance from the starting point of trajectory point 1001, and dragging operations include dragging trajectory point 1001 to another location. Accordingly, the scene generation device receives the user's operation instruction, which includes the target position to which the track point 1001 is dragged, and can thereby obtain the updated trajectory of the obstacle o5 based on the target position to which the track point 1001 is dragged. Of course, the scene generation device can record the user's position adjustment of one or more track points and generate the updated trajectory of the obstacle o5 when the user confirms the new trajectory (for example, when the user instructs to save or submits the modification).

[0178] As another possible implementation, the update of trajectory points can be indicated by the output of an algorithm or AI model. For example, the scene generation device can calculate key trajectory points among the trajectory points of the initial trajectory of the first obstacle, and generate one or more derived interaction points based on the key trajectory points (this process can also be called a generalization process), thereby obtaining one or more motion trajectories of the first obstacle based on the one or more derived interaction points. Key trajectory points, for example, are interaction points between the first obstacle and the ego vehicle's motion trajectory, or turning points, deceleration points, etc. of the first obstacle. The determination of key points is not strictly limited here. Taking the key trajectory points as trajectory interaction points as an example, the scene construction device can determine the trajectory interaction points between the initial motion trajectory of the first obstacle and the initial motion trajectory of the ego vehicle, and obtain derived trajectory points based on the trajectory interaction points. Derivative trajectory points are trajectory points located on the initial motion trajectory of the ego vehicle and different from the trajectory interaction points. The first motion trajectory is obtained based on the derived trajectory points. For example, after obtaining the derived trajectory points, the first obstacle's trajectory interaction points and some or all of the trajectory points other than the trajectory interaction points are smoothly connected to obtain the first motion trajectory.

[0179] Furthermore, before generating a motion trajectory based on the derived trajectory points, the derived trajectory points may also undergo conditional screening. In other words, only derived trajectory points that meet the preset conditions will be used to generate the motion trajectory of the first obstacle. Otherwise, the derived trajectory points may be discarded or updated. For example, the scene generation device can determine whether the derived trajectory points exceed the road boundary, whether the motion trajectory obtained based on the derived trajectory points collides with other obstacles (or the vehicle), etc. In one possible implementation, if the derived trajectory points do not exceed the road boundary and the motion trajectory obtained based on the derived trajectory points does not collide with other obstacles (or the vehicle), the first motion trajectory of the first obstacle is obtained based on the derived trajectory points.

[0180] In some solutions, a certain distribution law is satisfied between the derived interaction points derived from the same key trajectory point, or between the derived interaction points and the initial key trajectory points. Taking the key trajectory point as the trajectory interaction point between the first obstacle and the vehicle as an example, a distribution law can be as follows: along the motion trajectory of the vehicle, the distance between any two interaction points is equal, and the interaction points here include derived interaction points and basic interaction points. Please refer to Figures 11A and 11B, which are schematic diagrams of another scenario for updating the motion information of an obstacle provided in an embodiment of the present application. Among them, the key trajectory points are shown as hollow squares in Figures 11A and 11B, and the derived trajectory points derived based on the key trajectory points are shown as solid circles in Figures 11A and 11B. It can be seen that there is a certain distance between the derived trajectory points and the trajectory key points, and between the derived trajectory points.

[0181] Optionally, there may be multiple first obstacles. For example, the trajectories or behavior parameters of the multiple obstacles may be updated. For another example, the scene generation device may automatically derive the motion trajectory of each of the multiple obstacles, or update the behavior parameters of each of the multiple obstacles.

[0182] Step S303: obtaining second motion information of the second obstacle according to the initial motion information of the second obstacle and the first motion information of the first obstacle.

[0183] Specifically, after the motion information of the first obstacle is updated, the motion information of the second obstacle can be updated based on the updated motion information of the first obstacle (i.e., the second motion information). When updating the motion information of the second obstacle, the update can be based on the initial motion information and combined with the first motion information of the first obstacle. This ensures that the motion trajectory of the second obstacle is as close to the original trajectory as possible without colliding with the second obstacle, thereby improving authenticity.

[0184] As a possible implementation, the trajectory update of the second obstacle can be achieved through an AI model. For example, the scene generation device inputs the initial motion information of the second obstacle and the first motion information of the first obstacle into the AI ​​model to obtain the second motion information of the second obstacle. Specifically, the AI ​​model is trained through the motion information of obstacles in multiple driving scenes, and can achieve joint optimization of multiple obstacles. Furthermore, the motion information of the obstacle output by the AI ​​model can meet one or more of the following constraints: kinematic constraints, collision constraints, or driving specification constraints, etc. (introduced below). Furthermore, the AI ​​model can also update the motion information of the second obstacle at the lowest cost. For a description of the cost, please refer to the description below.

[0185] As a possible implementation, for the trajectory update of the second obstacle, the optimal motion information may be obtained by calculating the updated trajectory cost.

[0186] Specifically, the scene generation device may determine the second motion information of the second obstacle based on the initial motion information of the second obstacle, the cost of the second motion information of the second obstacle, and the first motion information of the first obstacle. When calculating the cost, the cost of the second motion information can be expressed in an algebraic form, so that the scene generation device can obtain a set of algebraic solutions through a numerical solution method to obtain the motion trajectory of the second obstacle.

[0187] The cost of the second motion information includes at least one of a realism cost, a smoothness cost, an obstacle avoidance cost, and a scenario design cost. Specifically, the second motion information includes a second motion trajectory of the second obstacle, and the second motion trajectory includes multiple trajectory points. The realism cost is used to indicate the difference between the second motion information of the second obstacle and the initial motion information of the second obstacle. The realism cost is related to the coordinates, orientation, curvature, and velocity of the multiple trajectory points. The smoothness cost is used to indicate the smoothness of the position changes of the multiple trajectory points. The smoothness cost is related to the curvature and acceleration of the second motion trajectory. The obstacle avoidance cost is used to indicate the obstacle avoidance range of the second obstacle. The scenario design cost is a predefined scenario cost design. Exemplarily, the scenario design cost is used to flexibly adjust the cost of making decisions in different test scenarios for the vehicle. For example, corresponding scenario design costs can be predefined for difficult scenarios, special weather scenarios (such as rainy scenarios, blizzard scenarios, heavy fog scenarios, etc.), special terrain scenarios (such as winding mountain road scenarios, desert area scenarios, high temperature scenarios, frozen soil scenarios, etc.).

[0188] Optionally, when the second obstacle includes multiple obstacles, the cost of the second motion information of the second obstacle is the sum of the costs of the second motion information of the multiple obstacles.

[0189] For example, when the second obstacle includes one obstacle (hereinbelow, obstacle i is taken as an example), the cost J of the second motion information of the second obstacle is i It can be calculated by the following formula:

[0190] J i =W X (X i -X ref ) 2 +W k k i 2 +W u u i 2 +W a a i 2 +W s slack i 2 +W f f hard 2

[0191] Among them, X i The second motion information of the second obstacle i can be represented by the position, direction, curvature and speed of the m trajectory points. m is the number of trajectory points, and [x, y, phi, k, v] are the horizontal coordinate, vertical coordinate, orientation angle, curvature and velocity of the trajectory points respectively.

[0192] W X ,W k ,W u ,W a ,W s ,W f Used to represent coefficients. W X (X i -X ref ) 2 For authenticity cost, X ref is the initial motion information of the second obstacle i, which may also include the position, orientation, curvature and speed of multiple trajectory points, which will not be described here one by one. k k i 2 +W u u i 2 +W a a i 2 is the smoothness cost, ki represents the curvature of the second obstacle i, a i is the acceleration of the second obstacle i, u i is the derivative of the curvature of the second obstacle i (or the rate of change of curvature), W s slack i 2 is the obstacle avoidance cost, W f f hard 2 The hard case cost is the design cost under the hard case scenario, that is, W f f hard 2 It is the cost of scene design.

[0193] As another example, when the second obstacle includes n (n is an integer greater than 1) obstacles, the cost of the second motion information of the second obstacle may satisfy the following formula:

[0194] Among them, J i The cost of the second motion information of the i-th obstacle among n (n is an integer and n>0) obstacles included in the second obstacle, i=1, 2, 3, ... and i≤n.

[0195] In some possible situations, the second motion information also satisfies one or more of the following constraints: kinematic constraints, collision constraints, or driving specification constraints. The kinematic constraints are used to indicate the value ranges of the following behavior parameters in the second motion information: curvature, velocity, acceleration, rate of change of curvature, etc. The collision constraint is used to constrain the second obstacle to collision with other objects. The other objects here may include the vehicle, one or more obstacles in the first driving scene (e.g., the first obstacle), etc. The driving specification constraint is used to constrain the behavior of the first obstacle to meet predefined driving specifications.

[0196] The following is an example introduction to each of the above constraints:

[0197] (1) Kinematic constraints are used to constrain the range of curvature, velocity, acceleration, curvature derivative, etc., so as to ensure that the driving trajectory is more reasonable and complies with the kinematic constraints. For example, the curvature k, velocity v, acceleration a, and curvature derivative u of the second obstacle fall within the following ranges:

[0198] k≥kappaLowBound,k≤kappaUpperBound

[0199] v≥vLowBound,v≤vUpperBound

[0200] u≥uLowBound,u≤uUpperBound

[0201] a≥aLowBound,a≤aUpperBound

[0202] kappaLowBound and kappaUpperBound are used to represent the curvature threshold, while vLowBound and vUpperBound are used to represent the velocity threshold. The remaining parameters are similar and will not be explained here one by one.

[0203] (2) The motion information of the second obstacle must conform to the kinematic model. For example, the displacement x, displacement y, time t, curvature k, velocity v, acceleration a, curvature change rate u, θ, etc. of the second obstacle must satisfy the following formula:

[0204] x · (t) = v(t)cos(θ(t))

[0205] y · (t) = v(t) sin(θ(t))

[0206] θ · (t)=v(t)κ(t)

[0207] k · (t) = u(t)

[0208] v · (t) = a(t)

[0209] (3) Collision constraint, ensuring that the second obstacle does not collide with other obstacles.

[0210] The following describes the constraints for preventing collisions between the second obstacle, between the second obstacle and the first obstacle, and between the second obstacle and the ego vehicle. For example, if the second obstacle consists of two obstacles, their positions are represented by (x1, y1) and (x2, y2), respectively. The ego vehicle's position is represented by (egox, egoy), and the first obstacle's position is represented by (keyObjx, keyObjy).

[0211] For example, the non-collision between the second obstacles can be expressed by the following formula:

[0212] (x1-x2)*(x1-x2)+(y1-y2)*(y1-y2)≥slack

[0213] slack≥slackLower,slack≤slackUpper

[0214] Wherein, slackLower represents the lower threshold value for collision identification, and slackUpper represents the upper threshold value for collision identification. The aforementioned threshold values ​​may be predefined or predesigned.

[0215] For example, the non-collision between the second obstacle and the ego vehicle can be expressed by the following formula:

[0216] (x1-egox)*(x1-egox)+(y1-egoy)*(y1-egoy)≥ slackEgo

[0217] slackEgo≥slackEgoLower,slackEgo≤slackEgoUpper

[0218] For example, the non-collision between the second obstacle and the first obstacle can be expressed by the following formula:

[0219] (x1-keyObjx)*(x1-keyObjx)+(y1-keyObjy)*(y1-keyObjy)≥ slackKeyObj

[0220] slackKeyObj≥slackKeyObjLower,slackKeyObj≤slackKeyObjUpper

[0221] (4) Driving compliance constraint, which is used to constrain the driving trajectory of the second obstacle to comply with road constraints, such as not compacting the road edge. For example, a driving compliance constraint of the second obstacle (x1, y1) can be expressed by the following formula:

[0222] -slackNormal≤(x1-xNormal)*sin(thetaNormal)-(y1-xNormal)*cos(thetaNormal)≤slackNoraml.

[0223] Among them, slackNormal, xNormal, yNormal, etc. are used to represent parameters related to road specifications.

[0224] As a possible implementation manner, the scene generation device may use a solver to solve the aforementioned cost design, or the aforementioned one or more constraints, to obtain the second motion information of the second obstacle.

[0225] Step S304: Generate a target driving scene according to the first motion information of the first obstacle and the second motion information of the second obstacle.

[0226] Optionally, the target driving scenario can be used to test an intelligent driving algorithm, or the target driving scenario can be used as model training data, for example, for training an obstacle recognition model, or for training an autonomous driving decision model.

[0227] In one possible scenario, the target driving scenario can be played in a scenario generation device. During playback of the target driving scenario, a first obstacle in the target driving scenario moves according to a motion process indicated by first motion information. This motion process includes the trajectory of the motion and the behavioral parameters during the motion process. Correspondingly, a second obstacle moves according to a motion process indicated by second motion information. With reference to Figure 2 , this scenario playback function can be implemented by a scenario playback module in the scenario generation device 201.

[0228] For example, see Figure 12, which is a schematic diagram of a target driving scenario provided by an embodiment of the present application, wherein the arrow represents the initial motion trajectory, and the dotted arrow represents the motion trajectory obtained after adjustment.

[0229] Optionally, the scene data of the first driving scene further includes initial motion information of a fourth obstacle. The scene generation device may generate a target driving scene based on the first motion information of the first obstacle, the second motion information of the second obstacle, and the initial motion information of the fourth obstacle.

[0230] Further optionally, the fourth obstacle has no interaction with the ego vehicle and the first obstacle. For example, the fourth obstacle may belong to the third category of obstacles (i.e., background obstacles).

[0231] The method of the embodiment of the present application is described above, and the device of the embodiment of the present application is provided below.

[0232] It is understandable that the multiple devices provided in the embodiments of the present application, such as scene generation devices, computing devices, etc., in order to implement the functions in the above-mentioned method embodiments, include hardware structures, software units, or combinations of hardware structures and software structures for executing each function. It should be easily appreciated by those skilled in the art that, in combination with the various functions described in the embodiments disclosed herein, the devices and modules in the devices can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different device implementations to implement the aforementioned method embodiments in different usage scenarios, and the different implementations of the devices should not be considered to exceed the scope of the embodiments of the present application.

[0233] Several possible arrangements are listed below.

[0234] Please refer to Figure 13, which is a schematic diagram of the structure of a scene generation device provided in an embodiment of the present application. The scene generation device 130 may include an acquisition unit 1301 and a processing unit 1302. The scene generation device 130 is used to implement the aforementioned scene generation method, such as the scene generation method in the embodiment shown in Figure 3.

[0235] In one possible implementation, the acquiring unit 1301 is configured to acquire scene data of a first driving scene, where the scene data is used to indicate initial motion information of an obstacle in the first driving scene, where the obstacles in the first driving scene include a first obstacle and a second obstacle.

[0236] The processing unit 1302 is configured to:

[0237] Adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle;

[0238] Obtaining second motion information of the second obstacle based on the initial motion information of the second obstacle and the first motion information of the first obstacle;

[0239] A target driving scene is generated according to the first motion information of the first obstacle and the second motion information of the second obstacle.

[0240] In a possible implementation, the scene data further includes initial motion information of the vehicle.

[0241] In a possible implementation, the processing unit 1302 is further configured to obtain second motion information of the second obstacle based on the initial motion information of the ego vehicle, the initial motion information of the second obstacle, and the first motion information of the first obstacle.

[0242] In a possible implementation, the degree of interaction between the first obstacle and the ego vehicle is higher than or not lower than the degree of interaction between the second obstacle and the ego vehicle.

[0243] In a possible implementation, the processing unit 1302 is further configured to: adjust the initial motion information of the first obstacle according to user input to generate first motion information of the first obstacle.

[0244] In a possible implementation, the processing unit 1302 is further configured to: adjust the initial motion information of the first obstacle through an artificial intelligence model to generate first motion information of the first obstacle.

[0245] In a possible implementation manner, the second obstacle is an obstacle that interacts with the first obstacle.

[0246] In a possible implementation manner, the second obstacle is an obstacle other than the first obstacle that interacts with the vehicle.

[0247] In one possible implementation, the processing unit 1302 is further configured to: determine a category of an obstacle based on a degree of interaction between the obstacle and the ego-vehicle and a degree of interaction between obstacles in the first driving scene, where the category of the obstacle includes a first category of obstacles and a second category of obstacles;

[0248] The interaction degree between the first type of obstacles and the ego vehicle is higher than that between the second type of obstacles and the ego vehicle.

[0249] Optionally, the first obstacle belongs to the first category of obstacles, and the second obstacle includes at least one obstacle from the second category and / or includes at least one obstacle from the first category other than the first obstacle. In some embodiments, the first category of obstacles may be referred to as strongly interactive obstacles, and the second category of obstacles may be referred to as weakly interactive obstacles.

[0250] In a possible implementation, the obstacles in the first driving scene include first-category obstacles and second-category obstacles. For explanations of the first-category obstacles and the second-category obstacles, refer to the above.

[0251] In one possible implementation, the initial motion information of the first obstacle includes multiple behavior parameters of the first obstacle, where the multiple behavior parameters are used to describe the behavior of the first obstacle. Processing unit 1302 is further configured to adjust the values ​​or thresholds of the multiple behavior parameters of the first obstacle to generate the first motion information of the first obstacle.

[0252] In one possible implementation, the initial motion information of the first obstacle includes a first action set and a first trigger set of the first obstacle. The processing unit 1302 is further configured to adjust the first action set and / or the first trigger set of the first obstacle to generate the first motion information of the first obstacle.

[0253] In a possible implementation, the initial motion information of the first obstacle includes an initial motion trajectory of the first obstacle.

[0254] The processing unit 1302 is further configured to adjust the initial motion trajectory of the first obstacle to obtain a first motion trajectory, where the first motion information includes the first motion trajectory.

[0255] In one possible implementation, the initial motion trajectory of the first obstacle includes multiple trajectory points. The acquisition unit 1301 is further configured to receive a user input for updating some or all of the multiple trajectory points. The processing unit 1302 is further configured to obtain the first motion trajectory based on the update operation.

[0256] In a possible implementation, the scene data further includes the initial motion trajectory of the vehicle. The processing unit 1302 is further configured to:

[0257] Determining a trajectory intersection point between the initial motion trajectory of the first obstacle and the initial motion trajectory of the ego vehicle;

[0258] According to the trajectory interaction points, derived trajectory points are derived. The positions of the derived trajectory points are located on the initial motion trajectory of the vehicle and are different from the positions of the trajectory interaction points.

[0259] According to the derived trajectory points, a first motion trajectory is obtained.

[0260] In a possible implementation, the processing unit 1302 is further configured to:

[0261] Second motion information of the second obstacle is determined based on the initial motion information of the second obstacle, the cost of the second motion information of the second obstacle, and the first motion information of the first obstacle.

[0262] Optionally, the cost of the second motion information includes one or more of authenticity cost, smoothness cost, obstacle avoidance cost, and scene design cost.

[0263] In a possible implementation, the cost J of the second motion information of the second obstacle i It is expressed by the following formula:

[0264] J i =W X (X i -X ref ) 2 +W k k i 2 +W u u i 2 +W a a i 2 +W s slack i 2 +W f f hard 2

[0265] The parameters are described above.

[0266] In a possible implementation manner, the second motion information satisfies one or more of the following constraints: kinematic constraints, collision constraints, or driving regulation constraints.

[0267] In a possible implementation, the scene data of the first driving scene further includes initial motion information of a fourth obstacle, and the fourth obstacle has no interaction with the ego vehicle and the first obstacle.

[0268] The processing unit 1302 is further configured to generate a target driving scene according to the first motion information of the first obstacle, the second motion information of the second obstacle, and the initial motion information of the fourth obstacle.

[0269] Figure 14 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Computing device 140 is a device with computing capabilities. The device here can be a physical device, such as a controller, processor, server (such as a rack server), host, etc., or a virtual device, such as a virtual machine, container, etc.

[0270] As shown in Figure 14, computing device 140 includes a processor 1402 and a memory 1401, and optionally includes a bus 1404 and a communication interface 1403. Processor 1402 and memory 1401 communicate with each other via bus 1404. It should be understood that this application does not limit the number of processors and memories in computing device 140.

[0271] Memory 1401 is used to provide storage space, which can optionally store application data, user data, operating systems, and computer programs. Memory 1401 may include volatile memory, such as random access memory (RAM). Memory 1401 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0272] Processor 1402 is a module for performing calculations and may include any one or more of a controller (e.g., a storage controller), a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), a coprocessor (assisting the central processor in completing corresponding processing and applications), an application-specific integrated circuit (ASIC), a microcontroller unit (MCU), a virtual machine, a container, and the like.

[0273] The communication interface 1403 is used to provide information input or output for at least one processor. And / or, the communication interface 1403 can be used to receive data sent externally and / or send data to the outside. The communication interface 1403 can be a wired link interface such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission and other wireless communication technologies, etc.). Optionally, the communication interface 1403 can also include a transmitter (such as a radio frequency transmitter, antenna, etc.) coupled to the interface, or a receiver, etc.

[0274] Bus 1404 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG14 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 1404 may include a pathway for transmitting information between various components of computing device 140 (e.g., memory 1401, processor 1402, and communication interface 1403).

[0275] In the embodiment of the present application, the memory 1401 stores executable instructions, and the processor 1402 executes the executable instructions to implement the aforementioned scene generation method, such as the scene generation method in the embodiment shown in Figure 3. That is, the memory 1401 stores instructions for executing the scene generation method.

[0276] The embodiment of the present application further provides a computing device cluster, the computing device cluster comprising at least one computing device 140 , each computing device 140 including a processor 1402 and a memory 1401 ;

[0277] The processor 1402 of at least one computing device 140 is configured to execute instructions stored in the memory 1401 of at least one computing device 140, so that the computing device cluster implements the aforementioned scene generation method, such as the scene generation method in the embodiment shown in FIG3 . Optionally, the memory stores instructions for executing the scene generation method.

[0278] An embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed by at least one processor, the aforementioned scene generation method is implemented, such as the scene generation method in the embodiment shown in Figure 3.

[0279] The computer-readable storage medium can be any available medium that can be stored by a computing device, or a data storage device such as a data center that contains one or more available media. The computer-readable storage medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0280] The present application provides a computer program product, which includes computer instructions. When the instructions are executed on at least one processor, the aforementioned scene generation method is implemented, such as the scene generation method in the embodiment shown in FIG3 .

[0281] Optionally, the computer program product may be a software installation package or an image package. When the aforementioned method is required, the computer program product may be downloaded and executed on a computing device.

[0282] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0283] The “at least one” mentioned in the embodiments of this application refers to one or more, and “plurality” refers to two or more. “At least one of the following items” or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. “And / or” describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character “ / ” generally indicates that the previous and next associated objects are in an “or” relationship.

[0284] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, timing, priority, or importance of multiple objects. For example, the first storage device and the second storage device are only for ease of description and do not indicate differences in the device structure, deployment order, or importance of the first and second storage devices.

[0285] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0286] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A scene generation method, characterized in that: Methods include: Acquire scene data of a first driving scene, where the scene data of the first driving scene is used to indicate initial motion information of the ego vehicle and obstacles in the first driving scene, where the obstacles in the first driving scene include a first obstacle and a second obstacle; adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle; Obtaining second motion information of the second obstacle according to the initial motion information of the second obstacle and the first motion information of the first obstacle; A target driving scene is generated according to the first motion information of the first obstacle and the second motion information of the second obstacle.

2. The method according to claim 1, characterized in that The obtaining, according to the initial motion information of the second obstacle and the first motion information of the first obstacle, second motion information of the second obstacle includes: The second motion information of the second obstacle is obtained according to the initial motion information of the vehicle, the initial motion information of the second obstacle and the first motion information of the first obstacle.

3. The method according to claim 1 or 2, characterized in that: The degree of interaction between the first obstacle and the vehicle is not lower than the degree of interaction between the second obstacle and the vehicle.

4. The method according to any one of claims 1 to 3, characterized in that: The second obstacle is an obstacle that interacts with the first obstacle and / or an obstacle that interacts with the vehicle other than the first obstacle.

5. The method according to any one of claims 1 to 4, characterized in that: The adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes: The initial motion information of the first obstacle is adjusted according to user input or through an artificial intelligence model to generate first motion information of the first obstacle.

6. The method according to any one of claims 1 to 5, characterized in that: The obstacles in the first driving scene include first-category obstacles and second-category obstacles, wherein: The first type of obstacle is an obstacle whose degree of interaction with the vehicle satisfies a first condition. The second type of obstacles include obstacles whose interaction degree with the ego vehicle satisfies a second condition and / or obstacles that interact with a third obstacle, the third obstacle is at least one obstacle in the first type of obstacles, and the first condition is different from the second condition; The first obstacle belongs to the first category of obstacles, and the second obstacle includes at least one obstacle in the second category of obstacles and / or includes at least one obstacle in the first category of obstacles other than the first obstacle.

7. The method according to claim 6, characterized in that The first condition includes: the time interval between the obstacle and the ego vehicle passing through the trajectory interaction point falls within a first time range; The second condition includes: the time interval between the obstacle and the vehicle passing through the trajectory interaction point falls within a second time range, and the lower limit of the second time range is greater than the upper limit of the first time range; There is interaction between the obstacle and the third obstacle when the following conditions are met: There is a trajectory interaction point between the obstacle and the third obstacle, and the time interval between the two passing the trajectory interaction point falls within the third time range.

8. The method according to claim 6, characterized in that The first condition includes: there is a trajectory interaction point between the movement trajectory of the obstacle and the movement trajectory of the vehicle, and the first time interval between the two passing the trajectory interaction point is the first N time intervals of the time interval sequence, N is an integer greater than 0, and the first N time intervals of the time interval sequence are less than or equal to the time intervals other than the first N time intervals in the time interval sequence; The second condition includes: there is a trajectory interaction point between the movement trajectory of the obstacle and the movement trajectory of the vehicle, and the first time interval between the two passing the trajectory interaction point falls within the first N+1 to K of the time interval sequence, where K is an integer and K>N; The time interval sequence includes M time intervals arranged in order of duration, each of the M time intervals is the interval between the moment when an ego-vehicle interaction obstacle passes through a first trajectory interaction point with the ego-vehicle and the moment when the ego-vehicle passes through the first trajectory interaction point, and the ego-vehicle interaction obstacle is an obstacle that has a trajectory interaction point with the ego-vehicle in the first driving scene, M is an integer and M≥K.

9. The method according to any one of claims 1 to 8, characterized in that: The initial motion information of the first obstacle includes a plurality of behavior parameters of the first obstacle, and the plurality of behavior parameters are used to describe the behavior of the first obstacle; and the adjusting the initial motion information of the first obstacle to generate the first motion information of the first obstacle includes: The values ​​or thresholds of the multiple behavior parameters of the first obstacle are adjusted to generate first motion information of the first obstacle.

10. The method according to any one of claims 1 to 8, characterized in that: The initial motion information of the first obstacle includes a first action set and a first trigger set of the first obstacle, the first action set is used to describe a first action of the first obstacle, the first action set includes a value of a first behavior parameter of the first action, and the first trigger set is used to describe a condition for executing the first action; The adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes: A first action set and / or a first trigger set of the first obstacle is adjusted to generate first motion information of the first obstacle.

11. The method according to any one of claims 1 to 8, characterized in that: The initial motion information of the first obstacle includes an initial motion trajectory of the first obstacle; The adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle includes: The initial motion trajectory of the first obstacle is adjusted to obtain a first motion trajectory, and the first motion information includes the first motion trajectory.

12. The method according to claim 11, characterized in that The initial motion trajectory of the first obstacle includes a plurality of trajectory points, and the adjusting the initial motion trajectory of the first obstacle to obtain the first motion trajectory includes: receiving an update operation of a part or all of the plurality of track points input by a user; According to the updating operation, the first motion trajectory is obtained.

13. The method according to claim 11, characterized in that The adjusting the initial motion trajectory of the first obstacle to obtain the first motion trajectory includes: Determining a trajectory interaction point between an initial motion trajectory of the first obstacle and an initial motion trajectory of the ego vehicle; According to the trajectory interaction point, a derived trajectory point is obtained, wherein the derived trajectory point is a trajectory point located on the initial motion trajectory of the vehicle and different from the trajectory interaction point; A first motion trajectory is obtained according to the derived trajectory points.

14. The method according to any one of claims 1 to 13, characterized in that: Obtaining second motion information of the second obstacle according to the initial motion information of the second obstacle and the first motion information of the first obstacle includes: The second motion information of the second obstacle is determined based on the initial motion information of the second obstacle, the cost of the second motion information of the second obstacle, and the first motion information of the first obstacle.

15. The method according to claim 14, characterized in that The cost of the second motion information includes at least one of a authenticity cost, a smoothness cost, an obstacle avoidance cost, and a scene design cost, the second motion information includes a second motion trajectory of the second obstacle, and the second motion trajectory includes a plurality of trajectory points. The authenticity cost is used to indicate the difference between the second motion information of the second obstacle and the initial motion information of the second obstacle, and the authenticity cost is related to the coordinates, orientation, curvature and speed of the plurality of trajectory points in the second motion trajectory. The smoothness cost is used to indicate the smoothness of position changes of the plurality of trajectory points in the second motion trajectory, and the smoothness cost is related to the curvature and acceleration of the second motion trajectory; The obstacle avoidance cost is used to indicate the obstacle avoidance range of the second obstacle. The scenario design cost is a predefined scenario cost design.

16. The method according to any one of claims 1 to 15, characterized in that: The second motion information satisfies one or more of the following constraints: kinematic constraints, collision constraints, or driving regulation constraints, wherein: The kinematic constraint is used to indicate the value range of the following behavior parameters in the second motion information: one or more of speed, acceleration, curvature of the motion trajectory, and rate of change of curvature of the motion trajectory; The collision constraint is used to constrain the second obstacle to have no collision with the vehicle and obstacles in the first driving scene. The driving specification constraint is used to constrain the behavior of the first obstacle to meet a predefined driving specification.

17. The method according to any one of claims 1 to 16, characterized in that: The scene data of the first driving scene further includes initial motion information of a fourth obstacle, and the fourth obstacle has no interaction with the vehicle and the first obstacle. The generating a target driving scene according to the first motion information of the first obstacle and the second motion information of the second obstacle includes: A target driving scene is generated according to the first motion information of the first obstacle, the second motion information of the second obstacle, and the initial motion information of the fourth obstacle.

18. A scene generation device, characterized in that: The device comprises an acquisition unit and a processing unit, wherein: The acquisition unit is used to acquire scene data of a first driving scene, where the scene data of the first driving scene is used to indicate initial motion information of the ego vehicle and obstacles in the first driving scene, where the obstacles in the first driving scene include a first obstacle and a second obstacle; The processing unit is used for: adjusting the initial motion information of the first obstacle to generate first motion information of the first obstacle; Obtaining second motion information of the second obstacle according to the initial motion information of the second obstacle and the first motion information of the first obstacle; A target driving scene is generated according to the first motion information of the first obstacle and the second motion information of the second obstacle.

19. The scene generation device according to claim 18, characterized in that: The processing unit is also used for: The second motion information of the second obstacle is obtained according to the initial motion information of the vehicle, the initial motion information of the second obstacle and the first motion information of the first obstacle.

20. The scene generation device according to claim 18 or 19, characterized in that: The degree of interaction between the first obstacle and the vehicle is not lower than the degree of interaction between the second obstacle and the vehicle.

21. The scene generation device according to any one of claims 18 to 20, characterized in that: The second obstacle is an obstacle that interacts with the first obstacle and / or an obstacle that interacts with the vehicle other than the first obstacle.

22. The scene generation device according to any one of claims 18 to 21, characterized in that: The processing unit is also used for: The initial motion information of the first obstacle is adjusted according to user input or through an artificial intelligence model to generate first motion information of the first obstacle.

23. The scene generation device according to any one of claims 18 to 22, characterized in that: The obstacles in the first driving scene include first-category obstacles and second-category obstacles, wherein: The first type of obstacle is an obstacle whose degree of interaction with the vehicle satisfies a first condition. The second type of obstacles include obstacles whose interaction degree with the ego vehicle satisfies a second condition and / or obstacles that interact with a third obstacle, the third obstacle is at least one obstacle in the first type of obstacles, and the first condition is different from the second condition; The first obstacle belongs to the first category of obstacles, and the second obstacle includes obstacles in the second category of obstacles and / or includes obstacles in the first category of obstacles except the first obstacle.

24. The scene generation device according to any one of claims 18 to 23, characterized in that: The initial motion information of the first obstacle includes a plurality of behavior parameters of the first obstacle, where the plurality of behavior parameters are used to describe the behavior of the first obstacle; and the processing unit is further used to: The values ​​or thresholds of the multiple behavior parameters of the first obstacle are adjusted to generate first motion information of the first obstacle.

25. The scene generation device according to any one of claims 18 to 23, characterized in that: The initial motion information of the first obstacle includes a first action set and a first trigger set of the first obstacle, the first action set is used to describe a first action of the first obstacle, the first action set includes a value of a first behavior parameter of the first action, and the first trigger set is used to describe a condition for executing the first action; The processing unit is also used for: A first action set and / or a first trigger set of the first obstacle is adjusted to generate first motion information of the first obstacle.

26. The scene generation device according to any one of claims 18 to 23, characterized in that: The initial motion information of the first obstacle includes an initial motion trajectory of the first obstacle; The processing unit is also used for: The initial motion trajectory of the first obstacle is adjusted to obtain a first motion trajectory, and the first motion information includes the first motion trajectory.

27. The scene generation device according to claim 26, characterized in that: The initial motion trajectory of the first obstacle includes a plurality of trajectory points; The acquisition unit is further used to receive an update operation of a part or all of the plurality of track points input by a user; The processing unit is further configured to obtain the first motion trajectory according to the updating operation.

28. The scene generation device according to claim 27, characterized in that: The processing unit is also used for: Determining a trajectory interaction point between an initial motion trajectory of the first obstacle and an initial motion trajectory of the ego vehicle; According to the trajectory interaction point, a derived trajectory point is obtained, wherein the derived trajectory point is a trajectory point located on the initial motion trajectory of the vehicle and different from the trajectory interaction point; A first motion trajectory is obtained according to the derived trajectory points.

29. The scene generation device according to any one of claims 18 to 28, characterized in that: The processing unit is further used for: The second motion information of the second obstacle is determined based on the initial motion information of the second obstacle, the cost of the second motion information of the second obstacle, and the first motion information of the first obstacle.

30. The scene generation device according to claim 29, characterized in that: The cost of the second motion information includes at least one of a authenticity cost, a smoothness cost, an obstacle avoidance cost, and a scene design cost, the second motion information includes a second motion trajectory of the second obstacle, and the second motion trajectory includes a plurality of trajectory points. The authenticity cost is used to indicate the difference between the second motion information of the second obstacle and the initial motion information of the second obstacle, and the authenticity cost is related to the coordinates, orientation, curvature and speed of the plurality of trajectory points in the second motion trajectory. The smoothness cost is used to indicate the smoothness of position changes of the plurality of trajectory points in the second motion trajectory, and the smoothness cost is related to the curvature and acceleration of the second motion trajectory; The obstacle avoidance cost is used to indicate the obstacle avoidance range of the second obstacle. The scenario design cost is a predefined scenario cost design.

31. The scene generation device according to any one of claims 18 to 30, characterized in that: The second motion information satisfies one or more of the following constraints: kinematic constraints, collision constraints, or driving regulation constraints, wherein: The kinematic constraint is used to indicate the value range of the following behavior parameters in the second motion information: one or more of curvature, velocity, acceleration, and curvature derivative; The collision constraint is used to constrain the second obstacle to have no collision with the vehicle and obstacles in the first driving scene. The driving specification constraint is used to constrain the behavior of the first obstacle to meet a predefined driving specification.

32. The scene generation device according to any one of claims 18 to 31, characterized in that: The scene data of the first driving scene further includes initial motion information of a fourth obstacle, and the fourth obstacle has no interaction with the vehicle and the first obstacle. The processing unit is further used for: According to the first motion information of the first obstacle, the second motion information of the second obstacle and the initial motion information of the fourth obstacle information and generate target driving scenarios.

33. A computing device, characterized in that The computing device includes a memory and a processor, wherein the memory is used to store program instructions; When the processor executes the program instructions in the memory, the computing device is caused to perform the method according to any one of claims 1 to 17.

34. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images