Method for editing dynamic driving scene and related equipment

Abstract

The invention discloses a method for editing a dynamic driving scene and related equipment, and relates to the technical field of video processing, and the method comprises the steps: determining a target vehicle from an original video in response to a vehicle behavior instruction for the original video, the scene corresponding to the original video is divided into a dynamic moving vehicle scene and a static background scene; and performing 4D track editing on the target vehicle based on an editing instruction carried in the vehicle behavior instruction, the dynamic moving vehicle scene and the static background scene, and generating an edited 4D dynamic driving video. According to the method, the requirement of the current technology on algorithm training data is met.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method and related equipment for editing dynamic driving scenes. Background Technology

[0002] Currently, to support the training data requirements of autonomous driving (AD) systems, a large amount of diverse and high-quality scene data is typically needed. Current technologies usually rely on synthetic scene generation techniques to generate the scene data required for training. These synthetic scene generation techniques construct driving scenes based on technologies such as 3D Gaussian splatting and Neural Radiation Field (NeRF). However, although synthetic scene generation techniques have the flexibility and scalability of scene design and can simulate diverse traffic layouts and vehicle behaviors, they lack the physical and semantic information of real-world environments, resulting in deviations from real driving scenarios and making it difficult to meet the current technology's requirements for algorithm training data. Summary of the Invention

[0003] The main purpose of this application is to provide a method and related equipment for editing dynamic driving scenarios, aiming to solve the technical problem that the synthesized data deviates from the real driving scenario in the current technology, making it difficult to meet the current technology's requirements for algorithm training data.

[0004] To achieve the above objectives, this application proposes a method for editing dynamic driving scenarios, the method comprising: In response to a vehicle behavior instruction for the original video, a target vehicle is determined from the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene. Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, the target vehicle's 4D trajectory is edited to generate an edited 4D dynamic driving video.

[0005] In one embodiment, the step of performing 4D trajectory editing on the target vehicle based on the editing instructions carried in the vehicle behavior instructions, the dynamically moving vehicle scene, and the static background scene to generate an edited 4D dynamic driving video includes: Determine the target illumination direction corresponding to the target vehicle in each frame, and determine the projection position and region corresponding to the target vehicle; Based on the target illumination direction, the corresponding projection position and region, determine the Gaussian sphere set corresponding to the target vehicle; Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene, the target vehicle in each frame is 4D trajectory edited, and the shadow Gaussian sphere color of each edited frame is adjusted based on the Gaussian sphere set to generate an edited 4D dynamic driving video.

[0006] In one embodiment, the step of adjusting the shadow Gaussian sphere color of each edited frame based on a Gaussian sphere set to generate an edited 4D dynamic driving video includes: Calculate the size of the Gaussian sphere set to determine the distribution range of the Gaussian sphere set along each axis, wherein the distribution range of the Gaussian sphere set along each axis is determined based on the difference between the maximum and minimum values ​​of the center coordinates on the three coordinate axes x, y, and z; Based on the coordinates of the Gaussian sphere, the target weight W is obtained by taking the maximum coordinate difference between the first and second coordinates of the Gaussian sphere set and the preset MLP network. Based on the target weight W, the spherical harmonic coefficients of the Gaussian sphere of the target vehicle in the corresponding shadow area are adjusted, and the shadow Gaussian sphere color is adjusted for each edited frame based on the spherical harmonic coefficients of the Gaussian sphere to generate the edited 4D dynamic driving video.

[0007] In one embodiment, the step of determining the target vehicle from the original video in response to a vehicle behavior instruction for the original video includes: In response to a vehicle behavior instruction for the original video, wherein the vehicle behavior instruction carries one or more of the following: vehicle appearance description information, lane position description information, relative position description information, and relative order description information; Based on one or more of the following: the cosine similarity between the appearance description information and the visual features of the corresponding vehicle in the original video; the lane position description information and the lane position constraints of the corresponding vehicle in the original video; the relative position description information and the relative position constraints of the original video; and the relative order description information and the relative order constraints of the original video, a comprehensive vehicle ranking score is calculated, and the target vehicle is determined based on the comprehensive vehicle ranking score.

[0008] In one embodiment, the step of performing 4D trajectory editing on the target vehicle based on the editing instructions carried in the vehicle behavior instructions, the dynamically moving vehicle scene, and the static background scene to generate an edited 4D dynamic driving video includes: Based on the editing instructions carried in the vehicle behavior instructions, the type of the editing instructions is determined, wherein the type of the editing instructions includes the type of repositioning of the target vehicle, the type of adding a new vehicle, and the type of motion planning for the target vehicle; Based on the type of editing instruction, the dynamic moving vehicle scene and the static background scene, the target vehicle is subjected to 4D trajectory editing, and the target vehicle after 4D trajectory editing is subjected to collision correction to generate an edited 4D dynamic driving video.

[0009] In one embodiment, the step of performing 4D trajectory editing on the target vehicle based on the type of the editing instruction, the dynamic moving vehicle scene, and the static background scene, and then performing collision correction on the target vehicle after 4D trajectory editing, includes: When the type of the editing instruction is vehicle position adjustment, the editing instruction is converted into a quantized spatial offset; the spatial offset is applied to all trajectory points of the target vehicle, and based on all trajectory points with applied spatial offset, the dynamic moving vehicle scene and static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle, and collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video; When the type of the editing instruction is motion planning, the target vehicle is trajectory planned based on the action information, velocity curve information, preset vehicle kinematic constraint information and preset scene context in the editing instruction. Based on the planned trajectory, collision detection and correction are performed on the dynamic moving vehicle scene and static background scene in each frame to generate the edited 4D dynamic driving video. When the type of the editing instruction is to add a new vehicle type, the corresponding vehicle is used as the reference vehicle. Based on the coordinates of the reference vehicle and the relative offset between the reference vehicle and the new vehicle in the editing instruction, the target coordinates of the new vehicle are calculated. Based on the target coordinates of the new vehicle in each frame, the dynamic moving vehicle scene and the static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle. Collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video.

[0010] In one embodiment, after the step of performing 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video, the following steps are included: Determine the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame from the 4D dynamic driving scene. Based on the scale and transparency of the Gaussian spheres corresponding to all the moving vehicles, an unreliable region mask is determined. The masked video formed by the unreliable region mask and the edited 4D dynamic driving video are used as inputs to the preset optimization model, and the optimized target video is output. The optimized target video is then used to fill the gaps in unreliable regions using the generation capabilities of the diffusion model, thus generating the target video.

[0011] Furthermore, to achieve the above objectives, this application also proposes an apparatus for editing dynamic driving scenes, the apparatus comprising: A response module is used to determine the target vehicle from the original video in response to a vehicle behavior command for the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene. The editing module is used to perform 4D trajectory editing on the target vehicle based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, and generate an edited 4D dynamic driving video.

[0012] Furthermore, to achieve the above objectives, this application also proposes a device for editing dynamic driving scenarios, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of editing dynamic driving scenarios as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of editing dynamic driving scenarios as described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of editing dynamic driving scenarios as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: Compared to related technologies, which typically rely on synthetic scene generation techniques to generate scene data for training (which deviates from real-world driving scenarios) and thus fail to meet the current demands for algorithm training data, this application addresses this issue. It identifies a target vehicle from an original video in response to vehicle behavior commands, wherein the original video scene is divided into a dynamic moving vehicle scene and a static background scene. Based on editing instructions carried in the vehicle behavior commands, the dynamic moving vehicle scene, and the static background scene, the application performs 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video. It is understood that in this application, data that deviates from the real scene is not purely synthesized. Instead, based on vehicle behavior commands, the target vehicle is determined from the original video (real scene) which has been divided into dynamic moving vehicle scene and static background scene. Then, based on editing commands, the target vehicle's trajectory is edited in 4D according to the dynamic moving vehicle scene and static background scene. It is understood that the video in this application is obtained by editing the 4D trajectory of the corresponding vehicle in the real scene (derived from the real scene, therefore, the edited scene conforms to the logic of real driving). Moreover, this application performs 4D trajectory editing (compared to 2D and 2.5D editing, it is a comprehensive and multi-dimensional editing), which can more comprehensively simulate the dynamic changes of the real driving scene, provide higher quality scene resources for autonomous driving data enhancement and simulation, and thus meet the current technology's requirements for algorithm training data. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for editing dynamic driving scenarios according to Embodiment 1 of this application; Figure 2 This is a schematic diagram illustrating the entire process of editing dynamic driving scenarios according to this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the method for editing dynamic driving scenarios according to this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the method for editing dynamic driving scenarios in this application; Figure 5This is a schematic diagram of the module structure of the device for editing dynamic driving scenarios according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in editing dynamic driving scenarios in this embodiment of the application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is: in response to a vehicle behavior instruction for the original video, a target vehicle is determined from the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene; Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, the target vehicle's 4D trajectory is edited to generate an edited 4D dynamic driving video.

[0023] In related technologies, scene generation techniques are typically used to generate scene data for training (which deviates from real driving scenarios), making it difficult to meet the current technology's requirements for algorithm training data. This application addresses this by determining a target vehicle from an original video in response to vehicle behavior commands, wherein the original video scene has been divided into a dynamic moving vehicle scene and a static background scene; based on editing instructions carried in the vehicle behavior commands, the dynamic moving vehicle scene, and the static background scene, the target vehicle's 4D trajectory is edited to generate an edited 4D dynamic driving video. It is understood that in this application, data that deviates from the real scene is not purely synthesized. Instead, based on vehicle behavior commands, the target vehicle is determined from the original video (real scene) which has been divided into dynamic moving vehicle scene and static background scene. Then, based on editing commands, the target vehicle's trajectory is edited in 4D according to the dynamic moving vehicle scene and static background scene. It is understood that the video in this application is obtained by editing the 4D trajectory of the corresponding vehicle in the real scene (derived from the real scene, therefore, the edited scene conforms to the logic of real driving). Moreover, this application performs 4D trajectory editing (compared to 2D and 2.5D editing, it is a comprehensive and multi-dimensional editing), which can more comprehensively simulate the dynamic changes of the real driving scene, provide higher quality scene resources for autonomous driving data enhancement and simulation, and thus meet the current technology's requirements for algorithm training data.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, or a device for editing dynamic driving scenarios. The following description uses a device for editing dynamic driving scenarios as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide a method for editing dynamic driving scenarios, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for editing dynamic driving scenarios according to this application.

[0026] In this embodiment, the method for editing dynamic driving scenarios includes steps S10 to S20: Step S10: In response to a vehicle behavior instruction for the original video, determine the target vehicle from the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene. It should be noted that the execution subject of this embodiment is a device for editing dynamic driving scenes, and this device for editing dynamic driving scenes belongs to a system for editing dynamic driving scenes.

[0027] Specifically, the device for editing dynamic driving scenes responds to vehicle behavior commands for the original video, then identifies the target vehicle from the original video. It should be noted that the vehicle behavior commands can be given by the user via natural language; that is, the user inputs a natural language command, the device receives the command, and obtains the input natural language command through a built-in Large Language Model (LLM). The natural language command is then decomposed to obtain vehicle behavior commands for the original video (this process allows users to accurately edit dynamic driving scenes without needing complex professional tools or tedious manual operations, lowering the barrier to entry). The device then responds to the vehicle behavior commands for the original video. It should be noted that the original video is either already stored in the device for editing dynamic driving scenes, or it is a video received from a camera in real-time and processed accordingly.

[0028] In this embodiment, the original video scene has been divided into a dynamic moving vehicle scene and a static background scene. Prioritizing this division allows for faster generation of the edited video after trajectory editing of the target vehicles (multiple vehicles), thereby improving the efficiency of training data generation. Furthermore, dividing the original video into dynamic moving vehicle scenes and static background scenes enables the use of a dynamic-static separation strategy in subsequent video reconstruction, ensuring that each vehicle is rendered individually, thus improving visual representation accuracy. In other words, dividing the original video scene into dynamic moving vehicle scenes and static background scenes in this embodiment serves multiple purposes.

[0029] In this embodiment, in response to a vehicle behavior instruction for the original video, the target vehicle first needs to be determined from the original video.

[0030] The method for determining the target vehicle from the original video in response to a vehicle behavior instruction for the original video can be as follows: Method 1: Directly identify the target vehicle from the original video based on the unique vehicle identifier in the vehicle behavior command.

[0031] Method 2: The step of determining the target vehicle from the original video in response to a vehicle behavior instruction for the original video includes: Step S11, in response to a vehicle behavior instruction for the original video, wherein the vehicle behavior instruction carries one or more of the following: vehicle appearance description information, lane position description information, relative position description information, and relative order description information. Step S12: Based on one or more of the following: the cosine similarity between the appearance description information and the visual features of the corresponding vehicle in the original video; the lane position description information and the lane position constraint of the corresponding vehicle in the original video; the relative position description information and the relative position constraint of the original video; and the relative order description information and the relative order constraint of the original video, calculate the vehicle comprehensive ranking score, and determine the target vehicle based on the vehicle comprehensive ranking score.

[0032] In this embodiment, the vehicle behavior instructions are first decomposed: specifically, the large language model (LLM) is used to decompose the vehicle behavior instructions into appearance description information (such as "white vehicle"), lane position description (such as "middle lane"), relative position description information (such as "in front of my vehicle"), relative order description information (such as "third vehicle in line"), etc. (the above are just examples, and the actual situation may be different).

[0033] While decomposing the vehicle behavior instructions, this embodiment also extracts the visual features of different frames in the original video. Specifically, this embodiment uses the average value of CLIP features corresponding to a certain vehicle in different frames of the original video as the visual feature corresponding to that vehicle (which can minimize errors or mistakes).

[0034] The specific process of visual feature extraction can be as follows: combining the CLIP features of vehicle a in multiple frames, calculate the visual (dynamic) feature V_a of vehicle a (V_a is the average value of the CLIP features of vehicle a in all frames in which it appears), and the calculation formula is as follows:

[0035] Where F_a is the set of frames containing vehicle a, and I_i is the rendered image of frame i.

[0036] In this embodiment, the specific method for determining the target vehicle may be: Method 1: First, determine the cosine similarity between the appearance description information T_a and the visual feature V_a to measure the degree of matching and perform a preliminary ranking of candidate vehicles. After the preliminary ranking, perform a multi-constraint joint ranking based on lane position description information, relative position description information, and relative order description information: Specifically, based on lane position description information combined with lane position constraint P_lane (determining whether the vehicle is in the lane specified by the instruction), based on relative position description information combined with relative position constraint P_rel (determining whether the positional relationship between the vehicle and the reference object conforms to the instruction), and based on relative order description information combined with relative order constraint P_order (determining whether the position of the vehicle in the sequence conforms to the instruction), calculate the comprehensive vehicle ranking score S(a), where S(a) consists of the scaling result of the appearance matching similarity and the sum of the three constraint terms. Based on S(a), select the target vehicle that best matches the vehicle behavior instruction.

[0037] Method 2: Without performing two sorting operations, directly calculate the vehicle comprehensive ranking score based on the cosine similarity between the appearance description information and the visual features of the corresponding vehicle in the original video, the lane position description information and the lane position constraints of the corresponding vehicle in the original video, the relative position description information and the relative position constraints of the original video, and the relative order description information and the relative order constraints of the original video. The target vehicle is then determined based on the comprehensive ranking score.

[0038] Step S20: Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, perform 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video.

[0039] In this embodiment, based on the editing instructions carried in the vehicle behavior instructions, such as adding or adjusting vehicle instructions, the dynamic moving vehicle scene and the static background scene, the target vehicle is subjected to 4D trajectory editing (based on 4D scene representation, breaking through the limitations of traditional 2D and 2.5D editing, it can perform all-round and multi-dimensional editing of dynamic driving scenes, and can more comprehensively simulate the dynamic changes of real driving scenes, providing higher quality scene resources for autonomous driving data enhancement and simulation), and an edited 4D dynamic driving video is generated.

[0040] The step of editing the target vehicle's trajectory based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene to generate an edited 4D dynamic driving video includes: Step S21: Determine the target illumination direction corresponding to the target vehicle in each frame, and determine the projection position and area corresponding to the target vehicle; Step S22: Based on the target illumination direction, the corresponding projection position and region, determine the set of Gaussian spheres corresponding to the target vehicle; In this embodiment, the dynamic lighting perception shadow modeling module in the device for editing dynamic driving scenes is used to achieve this. Specifically, the target lighting direction corresponding to the target vehicle in each frame is determined. Specifically, the target vehicle's forward direction is taken as the x-axis, the left side as the y-axis, and the top as the z-axis. Then, the global lighting direction (d_y, d_z) of each frame is calculated from the real video, and d_x=0 is set.

[0041] After determining the target illumination direction, the projected position and area corresponding to the target vehicle are determined. Specifically, based on the original coordinates (x, y, z) of the corresponding point of the target vehicle and the ground z-axis height g_z, the projected coordinates (x', y', z') of the corresponding point are calculated. The calculation formula is as follows:

[0042] Subsequently, for each movable vehicle a (target vehicle a), the range of its 2D projection area is determined, covering the minimum to maximum x-coordinate of all projected points of the vehicle in the x direction and the minimum to maximum y-coordinate of all projected points of the vehicle in the y direction. The set of Gaussian spheres within this range is the set of Gaussian spheres G_a corresponding to the vehicle.

[0043] Step S23: Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene, perform 4D trajectory editing on the target vehicle in each frame, and adjust the shadow Gaussian sphere color of each edited frame based on the Gaussian sphere set to generate the edited 4D dynamic driving video.

[0044] In this embodiment, after obtaining the Gaussian sphere set, based on the editing instructions carried in the vehicle behavior instructions, the dynamically moving vehicle scene, and the static background scene, the target vehicle in each frame undergoes 4D trajectory editing. Furthermore, based on the Gaussian sphere set, the shadow Gaussian sphere color of each edited frame is adjusted to generate an edited 4D dynamic driving video. In this embodiment, when editing based on the original video, the dynamic lighting-aware shadow modeling module adaptively adjusts the light and shadow, thus ensuring the realism and consistency of the shadows during the editing process, guaranteeing a more realistic and high-quality scene after editing.

[0045] The step of adjusting the shadow Gaussian sphere color of each edited frame based on the Gaussian sphere set to generate the edited 4D dynamic driving video includes: Step A1: Calculate the size of the Gaussian sphere set to determine the distribution range of the Gaussian sphere set along each axis, wherein the distribution range of the Gaussian sphere set along each axis is determined based on the difference between the maximum and minimum values ​​of the center coordinates on the x, y, and z coordinate axes; Step A2: Based on the coordinates of the Gaussian sphere, the maximum coordinate difference between the first and second coordinates of the Gaussian sphere set, and the preset MLP network, the target weight W is obtained; Step A3: Adjust the spherical harmonic coefficients of the Gaussian sphere of the target vehicle in the corresponding shadow area based on the target weight W, and adjust the shadow Gaussian sphere color of each edited frame based on the spherical harmonic coefficients of the Gaussian sphere to generate the edited 4D dynamic driving video.

[0046] In this embodiment, the size S_a of the Gaussian sphere set for the target vehicle a is first calculated. S_a is represented by the difference between the maximum and minimum values ​​of the center coordinates of this set on the x, y, and z axes, reflecting the distribution range of the Gaussian sphere set along each axis. Then, using coordinates based on the Gaussian spheres, the maximum coordinate difference between the first and second coordinates of the Gaussian sphere set is divided by S_a and normalized before being used as the input to a preset MLP network to obtain the weight W. The spherical harmonic coefficients SH (used to represent the color of the Gaussian spheres) of the shaded region are then adjusted using the weight W, calculated as follows:

[0047] This allows for adaptive adjustment of shadow colors, ensuring the realism and consistency of shadows during the editing process.

[0048] In this embodiment, refer to Figure 2 This is a complete workflow diagram. The inputs are the original video and vehicle behavior commands, and the output is the edited 4D dynamic driving video or optimized video. The intermediate processing steps include: vehicle selection, trajectory editing, shadow modeling, and shadow Gaussian sphere color adjustment, etc. Figure 2In this context, Vehicle BehaviorPrompt refers to vehicle behavior instructions or prompts; The gray car in the middle lane moves to the right lane; Vehicle Behavior Assign refers to vehicle behavior assignment; Vehicle Selection refers to vehicle selection; Trajectory Editing refers to the technique of modifying the motion path or development trend of an object; Appearance refers to appearance; Relocate refers to relocation; Relative position refers to relative position; Motion planning refers to motion planning; Motion Planning is a technique for designing collision-free motion sequences for robots from initial state to target state, which must meet conditions such as obstacle avoidance, shortest path, and energy consumption constraints, and is widely used in autonomous driving, robotic arm control, etc.; Lane refers to lane; Order refers to sequence; Collision Revision refers to collision correction; Global Light refers to global illumination; Agent xyz refers to xyz detection; Norm refers to standard; Shadow refers to shadow; Elements Composition refers to element composition; New Agent-Trajectory Pairs refers to new trajectories; Agents GaussianSpheres refers to Gaussian spheres; Background Gaussian Spheres are background Gaussian spheres; Video Refinement is video refining; Unreliable Mask is an unreliable mask; Coarse Video is a coarse video; Refine Video is an optimized video.

[0049] Compared to related technologies, which typically rely on synthetic scene generation techniques to generate scene data for training (which deviates from real-world driving scenarios) and thus fail to meet the current demands for algorithm training data, this application addresses this issue. It identifies a target vehicle from an original video in response to vehicle behavior commands, wherein the original video scene is divided into a dynamic moving vehicle scene and a static background scene. Based on editing instructions carried in the vehicle behavior commands, the dynamic moving vehicle scene, and the static background scene, the application performs 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video. It is understood that in this application, data that deviates from the real scene is not purely synthesized. Instead, based on vehicle behavior commands, the target vehicle is determined from the original video (real scene) which has been divided into dynamic moving vehicle scene and static background scene. Then, based on editing commands, the target vehicle's trajectory is edited in 4D according to the dynamic moving vehicle scene and static background scene. It is understood that the video in this application is obtained by editing the 4D trajectory of the corresponding vehicle in the real scene (derived from the real scene, therefore, the edited scene conforms to the logic of real driving). Moreover, this application performs 4D trajectory editing (compared to 2D and 2.5D editing, it is a comprehensive and multi-dimensional editing), which can more comprehensively simulate the dynamic changes of the real driving scene, provide higher quality scene resources for autonomous driving data enhancement and simulation, and thus meet the current technology's requirements for algorithm training data.

[0050] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The step of determining the target vehicle from the original video in response to a vehicle behavior instruction for the original video includes: Step B1: Based on the editing instructions carried in the vehicle behavior instructions, determine the type of the editing instructions, wherein the type of the editing instructions includes the type of repositioning the target vehicle, the type of adding a new vehicle, and the type of motion planning for the target vehicle; Step B2: Based on the type of editing instruction, the dynamic moving vehicle scene and the static background scene, perform 4D trajectory editing on the target vehicle, and perform collision correction on the target vehicle after 4D trajectory editing to generate an edited 4D dynamic driving video.

[0051] In this embodiment, the type of the editing instruction is determined (including one of the types of target vehicle relocation, adding new vehicles, and motion planning for the target vehicle). Then, based on the type of the editing instruction, the dynamic moving vehicle scene, and the static background scene, the target vehicle's 4D trajectory is edited, and collision correction is performed on the target vehicle after 4D trajectory editing to generate an edited 4D dynamic driving video. It should be noted that in this embodiment, collision correction is performed on the target vehicle after 4D trajectory editing before rendering, rather than rendering first, to avoid the inefficiency caused by multiple renderings, thereby improving the generation efficiency of the 4D dynamic driving video.

[0052] In addition, in this embodiment, the device for editing dynamic driving scenarios includes a Manager Agent. This Manager Agent acts as a top-level coordinator, segmenting intents from editing instructions (e.g., determining whether the instruction is a "vehicle position adjustment instruction" or a "add new vehicle instruction"), and identifying the types of other agents to be invoked and their corresponding scheduling timings. Then, based on the corresponding types and timings, and based on the dynamic moving vehicle scenario and the static background scenario, the target vehicle's 4D trajectory is edited, and collision correction is performed on the target vehicle after the 4D trajectory editing.

[0053] The steps of performing 4D trajectory editing on the target vehicle based on the type of the editing instruction, the dynamic moving vehicle scene, and the static background scene, and then performing collision correction on the target vehicle after 4D trajectory editing, include: Step B11: When the type of the editing instruction is vehicle position adjustment, the editing instruction is converted into a quantized spatial offset; the spatial offset is applied to all trajectory points of the target vehicle, and based on all trajectory points with applied spatial offset, the dynamic moving vehicle scene and static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle, and collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video. In this embodiment, when the type of the editing instruction is vehicle position adjustment, the manager agent calls the relocation agent: then based on the relocation agent, the natural language relocation instruction (such as "move the vehicle 5 meters to the left") is converted into a quantifiable spatial offset (along the x and y axes of the vehicle coordinate system). During rendering, this offset transformation is uniformly applied to all trajectory points of the target vehicle to achieve overall offset editing of the vehicle position.

[0054] Step B12: When the type of the editing instruction is motion planning, the target vehicle is trajectory planned based on the action information, velocity curve information, preset vehicle kinematic constraint information and preset scene context in the editing instruction. Based on the planned trajectory, collision detection and correction are performed on the dynamic moving vehicle scene and static background scene in each frame to generate the edited 4D dynamic driving video. In this embodiment, when the type of the editing instruction is motion planning, the manager agent calls the motion planning agent: The motion planning agent performs trajectory planning based on the action information (straight, stop, lane change, turn) and speed curve information (default, slow, fast) in the editing instruction; in the BEV (bird's-eye view) map, it combines vehicle kinematic constraints (such as maximum steering angle, acceleration limit) and scene context (such as road structure and traffic rule data in Waymo Open Dataset) to ensure that the trajectory conforms to physical laws, while efficiently completing collision detection and trajectory optimization.

[0055] Step B13: When the type of the editing instruction is to add a new vehicle type, the corresponding vehicle is used as the reference vehicle. Based on the coordinates of the reference vehicle and the relative offset between the reference vehicle and the new vehicle in the editing instruction, the target coordinates of the new vehicle are calculated. Based on the target coordinates of the new vehicle in each frame, the dynamic moving vehicle scene and the static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle. Collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video.

[0056] In this embodiment, when the type of the editing instruction is to add a new vehicle type, the manager agent calls the adding agent: the adding agent parses the message passed by the manager agent, extracts the reference vehicle (the base vehicle (corresponding self-vehicle) with the relative position of the new vehicle) and the relative offset (the position difference along the x and y axes), and calculates the precise placement coordinates of the new vehicle; then it calls the preset motion planning module (with the corresponding model built-in) to generate a complete trajectory for the newly added vehicle that conforms to road rules and traffic flow logic, so as to realize the fusion of the new vehicle with the original scene.

[0057] In this embodiment, it should be noted that regardless of whether the editing instruction is for repositioning the target vehicle, adding a new vehicle, or performing motion planning on the target vehicle, collision detection and correction are required for the generated trajectory. That is, the Collision Revision Agent needs to be invoked: after motion planning is completed, the Collision Revision Agent performs collision detection and correction on the generated trajectory.

[0058] Specifically, for frontal collision scenarios, the collision correction agent inserts a "safety interpolation frame" before the collision occurs to avoid collision by gradually reducing the vehicle speed; for rear-end collision scenarios, the collision correction agent takes the scene state before the collision as input, increases the vehicle speed, and then calls the planning module again to generate a new collision-free trajectory.

[0059] In this embodiment, the type of editing instruction is determined based on the editing instructions carried in the vehicle behavior instructions. The types of editing instructions include repositioning the target vehicle, adding a new vehicle, and performing motion planning on the target vehicle. Based on the type of editing instruction, the target vehicle's 4D trajectory is edited using the dynamic moving vehicle scene and the static background scene. Collision correction is then performed on the target vehicle after 4D trajectory editing to generate an edited 4D dynamic driving video. In this embodiment, multiple target vehicles can be edited differently to meet the needs of diverse training data.

[0060] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After the step of performing 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video, the following steps are included: Step S30: Determine the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame from the 4D dynamic driving scene. Step S40: Determine the unreliable region mask based on the scale and transparency of the Gaussian spheres corresponding to all the moving vehicles; Step S50: Using the masked video formed by the unreliable region mask and the edited 4D dynamic driving video as input to the preset optimization model, the optimized target video is output. Step S60: Fill the gaps in unreliable regions in the optimized target video using the generation capability of the diffusion model to generate the target video.

[0061] In this embodiment, the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame are first determined from the 4D dynamic driving scene in order to determine the unreliable region mask.

[0062] Specifically, based on the characteristic that "the smaller the Gaussian sphere, the higher the visual reliability of the reconstructed region," Gaussian spheres associated with all moving vehicles in the frame are selected to calculate the confidence map. The confidence map is jointly determined by the scale and transparency of the Gaussian spheres corresponding to all moving vehicles, as well as the cumulative product of the transparency of each Gaussian sphere. The calculation formula is as follows:

[0063] Where M represents the set of Gaussian spheres corresponding to the moving vehicle, s_i is the Gaussian sphere scale, and α_i is the Gaussian sphere transparency. The mean of the confidence map is defined as the pixel reliability P_conf, and a retention rate r_s is set. When the pixel reliability is less than the product of the maximum pixel reliability and the retention rate, the region is determined to be an unreliable region, and an unreliable mask U_mask is generated. Finally, the coarsely edited image is multiplied by the unreliable mask to obtain the mask image I_m, so as to accurately locate the region that needs to be optimized.

[0064] After determining the mask image I_m, the mask video V_m composed of the mask images and the coarse video V_c (the video corresponding to the 4D dynamic driving scene) obtained by coarse editing are used as inputs to the preset optimization model, and the optimized video V_r is output. The optimization target video is then filled with the gaps in unreliable areas by the generation capability of the diffusion model (optimizing texture and material performance) to generate the target video, thereby improving visual fidelity.

[0065] In this embodiment, the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame are determined from the 4D dynamic driving scene. Based on the scale and transparency of the Gaussian spheres corresponding to all moving vehicles, an unreliable region mask is determined. The masked video formed by the unreliable region mask and the edited 4D dynamic driving video are used as input to a preset optimization model, and an optimized target video is output. The optimized target video is then used to fill the gaps in the unreliable regions using the generation capability of a diffusion model to generate the target video. In this embodiment, the generated video is also post-processed, such as optimizing texture and material representation, thereby improving visual fidelity, enhancing the realism of the video, and ultimately improving the quality of the training data.

[0066] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the editing of dynamic driving scenarios in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0067] This application also provides a device for editing dynamic driving scenarios, please refer to... Figure 5 The device for editing dynamic driving scenarios includes: The response module 10 is used to determine the target vehicle from the original video in response to a vehicle behavior command for the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene. The editing module 20 is used to perform 4D trajectory editing on the target vehicle based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, and generate an edited 4D dynamic driving video.

[0068] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: Determine the target illumination direction corresponding to the target vehicle in each frame, and determine the projection position and region corresponding to the target vehicle; Based on the target illumination direction, the corresponding projection position and region, determine the Gaussian sphere set corresponding to the target vehicle; Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene, the target vehicle in each frame is 4D trajectory edited, and the shadow Gaussian sphere color of each edited frame is adjusted based on the Gaussian sphere set to generate an edited 4D dynamic driving video.

[0069] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: Calculate the size of the Gaussian sphere set to determine the distribution range of the Gaussian sphere set along each axis, wherein the distribution range of the Gaussian sphere set along each axis is determined based on the difference between the maximum and minimum values ​​of the center coordinates on the three coordinate axes x, y, and z; Based on the coordinates of the Gaussian sphere, the target weight W is obtained by taking the maximum coordinate difference between the first and second coordinates of the Gaussian sphere set and the preset MLP network. Based on the target weight W, the spherical harmonic coefficients of the Gaussian sphere of the target vehicle in the corresponding shadow area are adjusted, and the shadow Gaussian sphere color is adjusted for each edited frame based on the spherical harmonic coefficients of the Gaussian sphere to generate the edited 4D dynamic driving video.

[0070] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: In response to a vehicle behavior instruction for the original video, wherein the vehicle behavior instruction carries one or more of the following: vehicle appearance description information, lane position description information, relative position description information, and relative order description information; Based on one or more of the following: the cosine similarity between the appearance description information and the visual features of the corresponding vehicle in the original video; the lane position description information and the lane position constraints of the corresponding vehicle in the original video; the relative position description information and the relative position constraints of the original video; and the relative order description information and the relative order constraints of the original video, a comprehensive vehicle ranking score is calculated, and the target vehicle is determined based on the comprehensive vehicle ranking score.

[0071] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: Based on the editing instructions carried in the vehicle behavior instructions, the type of the editing instructions is determined, wherein the type of the editing instructions includes the type of repositioning of the target vehicle, the type of adding a new vehicle, and the type of motion planning for the target vehicle; Based on the type of editing instruction, the dynamic moving vehicle scene and the static background scene, the target vehicle is subjected to 4D trajectory editing, and the target vehicle after 4D trajectory editing is subjected to collision correction to generate an edited 4D dynamic driving video.

[0072] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: When the type of the editing instruction is vehicle position adjustment, the editing instruction is converted into a quantized spatial offset; the spatial offset is applied to all trajectory points of the target vehicle, and based on all trajectory points with applied spatial offset, the dynamic moving vehicle scene and static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle, and collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video; When the type of the editing instruction is motion planning, the target vehicle is trajectory planned based on the action information, velocity curve information, preset vehicle kinematic constraint information and preset scene context in the editing instruction. Based on the planned trajectory, collision detection and correction are performed on the dynamic moving vehicle scene and static background scene in each frame to generate the edited 4D dynamic driving video. When the type of the editing instruction is to add a new vehicle type, the corresponding vehicle is used as the reference vehicle. Based on the coordinates of the reference vehicle and the relative offset between the reference vehicle and the new vehicle in the editing instruction, the target coordinates of the new vehicle are calculated. Based on the target coordinates of the new vehicle in each frame, the dynamic moving vehicle scene and the static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle. Collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video.

[0073] In one embodiment, the apparatus for editing dynamic driving scenes is further configured to: Determine the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame from the 4D dynamic driving scene. Based on the scale and transparency of the Gaussian spheres corresponding to all the moving vehicles, an unreliable region mask is determined. The masked video formed by the unreliable region mask and the edited 4D dynamic driving video are used as inputs to the preset optimization model, and the optimized target video is output. The optimized target video is then used to fill the gaps in unreliable regions using the generation capabilities of the diffusion model, thus generating the target video.

[0074] The apparatus for editing dynamic driving scenes provided in this application, employing the method for editing dynamic driving scenes in the above embodiments, can solve the technical problem of editing dynamic driving scenes. Compared with the prior art, the beneficial effects of the apparatus for editing dynamic driving scenes provided in this application are the same as those of the method for editing dynamic driving scenes provided in the above embodiments, and other technical features in the apparatus for editing dynamic driving scenes are the same as those disclosed in the method of the above embodiments, and will not be repeated here.

[0075] This application provides a device for editing dynamic driving scenes. The device for editing dynamic driving scenes includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the editing of dynamic driving scenes as described in Embodiment 1 above.

[0076] The following is for reference. Figure 6 This document illustrates a schematic diagram of a device suitable for editing dynamic driving scenes in implementing embodiments of this application. The device for editing dynamic driving scenes in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, tablets, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The device shown for editing dynamic driving scenarios is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0077] like Figure 6As shown, the device for editing dynamic driving scenes may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the device for editing dynamic driving scenes. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the device editing dynamic driving scenes to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a device editing dynamic driving scenes with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0078] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0079] The device for editing dynamic driving scenarios provided in this application, employing the method for editing dynamic driving scenarios in the above embodiments, can solve the technical problem of editing dynamic driving scenarios. Compared with the prior art, the beneficial effects of the device for editing dynamic driving scenarios provided in this application are the same as those of the device for editing dynamic driving scenarios provided in the above embodiments, and other technical features in this device for editing dynamic driving scenarios are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0080] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0081] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0082] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the editing of dynamic driving scenarios in the above embodiments.

[0083] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0084] The aforementioned computer-readable storage medium may be included in a device for editing dynamic driving scenarios; or it may exist independently and not assembled into a device for editing dynamic driving scenarios.

[0085] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a device for editing a dynamic driving scene, the device for editing the dynamic driving scene: collects access information from different application sources through an access collection program and stores the access information in a message queue, wherein the access collection program is deployed in the runtime environment of the application; performs service call dependency deduction on the access information in the message queue to obtain the call relationship between the different applications; and generates an application topology architecture based on the call relationship.

[0086] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0088] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0089] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described editing of dynamic driving scenes, thereby solving the technical problem of editing dynamic driving scenes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the editing of dynamic driving scenes provided in the above embodiments, and will not be repeated here.

[0090] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described above for editing dynamic driving scenarios.

[0091] The computer program product provided in this application can solve the technical problem of editing dynamic driving scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of editing dynamic driving scenarios provided in the above embodiments, and will not be repeated here.

[0092] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for editing dynamic driving scenarios, characterized in that, The method for editing dynamic driving scenarios includes: In response to a vehicle behavior instruction for the original video, a target vehicle is determined from the original video, wherein the scene corresponding to the original video has been divided into a dynamic moving vehicle scene and a static background scene. Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene and the static background scene, the target vehicle's 4D trajectory is edited to generate an edited 4D dynamic driving video.

2. The method for editing dynamic driving scenes as described in claim 1, characterized in that, The step of editing the target vehicle's trajectory based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene to generate an edited 4D dynamic driving video includes: Determine the target illumination direction corresponding to the target vehicle in each frame, and determine the projection position and region corresponding to the target vehicle; Based on the target illumination direction, the corresponding projection position and region, determine the Gaussian sphere set corresponding to the target vehicle; Based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene, the target vehicle in each frame is 4D trajectory edited, and the shadow Gaussian sphere color of each edited frame is adjusted based on the Gaussian sphere set to generate an edited 4D dynamic driving video.

3. The method for editing dynamic driving scenarios as described in claim 1, characterized in that, The step of adjusting the shadow Gaussian sphere color of each edited frame based on the Gaussian sphere set to generate the edited 4D dynamic driving video includes: Calculate the size of the Gaussian sphere set to determine the distribution range of the Gaussian sphere set along each axis, wherein the distribution range of the Gaussian sphere set along each axis is determined based on the difference between the maximum and minimum values ​​of the center coordinates on the three coordinate axes x, y, and z; Based on the coordinates of the Gaussian sphere, the target weight W is obtained by taking the maximum coordinate difference between the first and second coordinates of the Gaussian sphere set and the preset MLP network. Based on the target weight W, the spherical harmonic coefficients of the Gaussian sphere of the target vehicle in the corresponding shadow area are adjusted, and the shadow Gaussian sphere color is adjusted for each edited frame based on the spherical harmonic coefficients of the Gaussian sphere to generate the edited 4D dynamic driving video.

4. The method for editing dynamic driving scenes as described in claim 1, characterized in that, The step of determining the target vehicle from the original video in response to a vehicle behavior instruction for the original video includes: In response to a vehicle behavior instruction for the original video, wherein the vehicle behavior instruction carries one or more of the following: vehicle appearance description information, lane position description information, relative position description information, and relative order description information; Based on one or more of the following: the cosine similarity between the appearance description information and the visual features of the corresponding vehicle in the original video; the lane position description information and the lane position constraints of the corresponding vehicle in the original video; the relative position description information and the relative position constraints of the original video; and the relative order description information and the relative order constraints of the original video, a comprehensive vehicle ranking score is calculated, and the target vehicle is determined based on the comprehensive vehicle ranking score.

5. The method for editing dynamic driving scenarios as described in claim 1, characterized in that, The step of performing 4D trajectory editing on the target vehicle based on the editing instructions carried in the vehicle behavior instructions, the dynamic moving vehicle scene, and the static background scene to generate an edited 4D dynamic driving video includes: Based on the editing instructions carried in the vehicle behavior instructions, the type of the editing instructions is determined, wherein the type of the editing instructions includes the type of repositioning of the target vehicle, the type of adding a new vehicle, and the type of motion planning for the target vehicle; Based on the type of editing instruction, the dynamic moving vehicle scene and the static background scene, the target vehicle is subjected to 4D trajectory editing, and the target vehicle after 4D trajectory editing is subjected to collision correction to generate an edited 4D dynamic driving video.

6. The method for editing dynamic driving scenes as described in claim 1, characterized in that, The steps of performing 4D trajectory editing on the target vehicle based on the type of the editing instruction, the dynamic moving vehicle scene, and the static background scene, and then performing collision correction on the target vehicle after 4D trajectory editing, include: When the type of the editing instruction is vehicle position adjustment, the editing instruction is converted into a quantized spatial offset; the spatial offset is applied to all trajectory points of the target vehicle, and based on all trajectory points with applied spatial offset, the dynamic moving vehicle scene and static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle, and collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video; When the type of the editing instruction is motion planning, the target vehicle is trajectory planned based on the action information, velocity curve information, preset vehicle kinematic constraint information and preset scene context in the editing instruction. Based on the planned trajectory, collision detection and correction are performed on the dynamic moving vehicle scene and static background scene in each frame to generate the edited 4D dynamic driving video. When the type of the editing instruction is to add a new vehicle type, the corresponding vehicle is used as the reference vehicle. Based on the coordinates of the reference vehicle and the relative offset between the reference vehicle and the new vehicle in the editing instruction, the target coordinates of the new vehicle are calculated. Based on the target coordinates of the new vehicle in each frame, the dynamic moving vehicle scene and the static background scene in each frame, a complete trajectory conforming to preset rules is generated for the new vehicle. Collision detection and correction are performed on the complete trajectory to generate the edited 4D dynamic driving video.

7. The method for editing dynamic driving scenes as described in claim 1, characterized in that, After the step of performing 4D trajectory editing on the target vehicle to generate an edited 4D dynamic driving video, the following steps are included: Determine the scale and transparency of the Gaussian spheres corresponding to all moving vehicles in the current frame from the 4D dynamic driving scene. Based on the scale and transparency of the Gaussian spheres corresponding to all the moving vehicles, an unreliable region mask is determined. The masked video formed by the unreliable region mask and the edited 4D dynamic driving video are used as inputs to the preset optimization model, and the optimized target video is output. The optimized target video is then used to fill the gaps in unreliable regions using the generation capabilities of the diffusion model, thus generating the target video.

8. A device for editing dynamic driving scenes, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the step of editing a dynamic driving scene as claimed in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the step of editing the dynamic driving scene as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the step of editing a dynamic driving scene as described in any one of claims 1 to 7.

Images