Video editing method and system for vehicle door state based on diffusion model and physical prior

Abstract

The present application belongs to the technical field of automatic driving perception, and particularly relates to a vehicle door state video editing method and system based on a diffusion model and physical priori. The method comprises: inputting a real driving video sequence and three-dimensional target detection and tracking label information thereof; screening a target vehicle based on the information, analyzing the visibility of the target vehicle in each frame and each perspective to determine an editing perspective; for each frame of each editing perspective, determining an editing region based on vehicle three-dimensional bounding box projection and constructing a text instruction; inputting an original video segment, frame-by-frame region information and the text instruction into a video diffusion model supporting mask editing to edit the vehicle door state in time sequence; based on a physical mapping relationship between a vehicle door size and an opening angle, automatically calculating an expansion distance, expanding a bounding box in a three-dimensional space and projecting to generate two-dimensional labels of each frame and each perspective; and finally constructing a synthetic data set. The present application solves the problems of vehicle door opening scene data scarcity and automatic generation of fine labeling.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving perception technology, specifically relating to a method and system for editing vehicle door status videos based on diffusion models and physical priors. Background Technology

[0002] Current autonomous driving perception algorithms rely on large-scale 3D labeled data for model training. However, in mainstream datasets, vehicle annotations typically only provide 3D bounding boxes at the whole vehicle level, lacking annotations on component states. In real-world scenarios, door opening occurs infrequently, representing long-tail data, making it difficult for models to learn effectively. To address the data scarcity problem, current solutions mainly fall into two categories: one is methods based on 3D simulators (such as CARLA) rendering, which generate images with significant domain differences from real-world scenes, and the production cost of 3D assets is high; the other is driving scene synthesis methods based on generative models (such as MagicDrive), as well as more advanced multi-view editing techniques (such as SceneCrafter) and general video diffusion models. These methods can perform scene-level or target-level editing, but the editing granularity is relatively coarse.

[0003] However, the above solution has a major problem: it cannot automatically generate extended 3D annotations that precisely match the door state while performing time-consistent fine-grained editing based on real driving videos, and it cannot modify the vehicle door state in a controllable and high-quality manner while preserving the real scene context, and simultaneously produce high-precision annotation data that conforms to physical laws and can be used for model training. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the current inability to perform time-consistent and detailed editing of the door status of a target vehicle based on real driving videos, and to simultaneously utilize physical priors to automatically generate extended 3D annotations.

[0005] To address the aforementioned technical problems, a first aspect of this invention discloses a method for editing vehicle door state videos based on a diffusion model and physical priors, the method comprising:

[0006] Input a driving video sequence from at least one perspective and the corresponding 3D target detection and tracking annotation information;

[0007] Based on the three-dimensional target detection and tracking annotation information, target vehicles are selected from the video sequence, and the visibility of the target vehicles in each frame and from each viewpoint is analyzed to determine the editing viewpoint for editing the door status and its effective frame range.

[0008] For each valid frame of each editing viewpoint, the editing area is determined based on the 3D bounding box projection of the target vehicle, and text editing instructions are constructed based on the target vehicle information;

[0009] The original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions are input into a video diffusion model that supports mask editing to perform time-consistent door state editing and generate the edited video.

[0010] Based on the physical mapping relationship between the door size and the opening angle, the lateral expansion distance after the door is opened is calculated, the three-dimensional bounding box of the target vehicle is expanded in three-dimensional space, and the expanded three-dimensional bounding box is projected onto each frame image of each editing view to generate corresponding two-dimensional annotations.

[0011] The quality of the edited videos and annotations is assessed and screened.

[0012] By aggregating the quality-filtered samples and their annotations, a synthetic dataset is constructed for training autonomous driving perception models.

[0013] As an optional implementation, in the first aspect of the invention, analyzing the visibility of the target vehicle in each frame and at each viewpoint includes:

[0014] The 3D bounding box of the target vehicle is projected onto the image plane using camera parameters from each frame and viewpoint.

[0015] Views whose projection area is greater than a preset threshold and are completely within the image range are included in the set of visible viewpoints of that frame.

[0016] Based on the angle between the door opening direction vector and the optical axis direction of each view camera, the view that can observe the car body on the side with the door open is determined from the set of visible viewpoints as the editing viewpoint;

[0017] When the input contains multiple perspectives, the remaining visible perspectives are marked as consistency constraint perspectives.

[0018] As an optional implementation, in the first aspect of the present invention, determining the editing region based on the three-dimensional bounding box projection of the target vehicle specifically involves:

[0019] For each valid frame of each editing viewpoint, the three-dimensional bounding box is expanded along the door opening direction by a preset expansion ratio in the two-dimensional projection box of that viewpoint to obtain the editing region mask of that frame and that viewpoint.

[0020] The preset expansion ratio is determined according to the vehicle category;

[0021] Due to the movement of the vehicle, the edited area mask is updated frame by frame as the target vehicle's projection position changes.

[0022] As an optional implementation, in the first aspect of the present invention, the video diffusion model supporting mask editing is a model capable of receiving the original video, spatial temporal mask and text instructions as conditional inputs, and performing conditional denoising generation within the mask area;

[0023] When multiple editing perspectives exist, the consistency of editing results across multiple perspectives is ensured by assigning the same random noise seed and the same text editing instructions to each perspective.

[0024] As an optional implementation, in the first aspect of the present invention, the calculation of the lateral extension distance based on the physical mapping relationship between the door size and the opening angle specifically includes:

[0025] Based on the target vehicle category, obtain the typical width dimensions of the corresponding door. ;

[0026] Determine the corresponding opening and closing angle based on the target opening and closing state. ;

[0027] According to the formula Calculate the lateral expansion distance.

[0028] As an optional implementation, in the first aspect of the invention, the three-dimensional bounding box of the target vehicle is extended in three-dimensional space, specifically as follows:

[0029] Along the direction the car door opens, increase the lateral expansion distance in the width dimension of the three-dimensional bounding box. And offset the center coordinates along the direction the car door opens. ;

[0030] Since the target vehicle is stationary, the expansion amount remains consistent across all frames.

[0031] As an optional implementation, in the first aspect of the present invention, the quality assessment and screening of the edited video and annotations includes at least one of the following assessment dimensions:

[0032] Single-frame image quality assessment evaluates the quality score of the image within the edited area.

[0033] Temporal consistency assessment evaluates the smoothness of changes in edited regions between consecutive frames.

[0034] Consistency verification is performed to verify whether the edited vehicle can be continuously and correctly detected by the pre-trained vehicle detection model.

[0035] Cross-view consistency check: When there are multiple editing views, evaluate the similarity of appearance features of the door area in different views at the same moment.

[0036] Boundary transition naturalness assessment evaluates the degree of naturalness of the transition between the edited area and the original video boundary.

[0037] As an optional implementation, in the first aspect of the present invention, the video diffusion model supporting mask editing includes the VACE model from Alibaba Tongyi Labs or a video editing model based on the Wan series.

[0038] A second aspect of this invention discloses a vehicle door state video editing system based on a diffusion model and physical priors, used to implement the vehicle door state video editing method based on a diffusion model and physical priors described in any of the above embodiments. The system includes:

[0039] The information input module is used to input a driving video sequence from at least one perspective and the corresponding 3D target detection and tracking annotation information;

[0040] The target analysis module is used to filter target vehicles from the video sequence based on the three-dimensional target detection and tracking annotation information, and analyze the visibility of the target vehicles in each frame and from each viewpoint to determine the editing viewpoint for editing the door status and its effective frame range.

[0041] The region instruction module is used to determine the editing region based on the three-dimensional bounding box projection of the target vehicle for each valid frame of each editing view, and to construct text editing instructions based on the target vehicle information.

[0042] The video editing module is used to input the original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions into a video diffusion model that supports mask editing, to perform time-consistent door state editing, and generate the edited video;

[0043] The annotation generation module is used to calculate the lateral expansion distance after the door is opened based on the physical mapping relationship between the door size and the opening angle, expand the three-dimensional bounding box of the target vehicle in three-dimensional space, and project the expanded three-dimensional bounding box onto each frame image of each editing view to generate corresponding two-dimensional annotations.

[0044] The module is used to evaluate and filter the quality of the edited videos and annotations, and to summarize the samples that pass the quality screening and their annotations to build a synthetic dataset for training autonomous driving perception models.

[0045] A third aspect of this invention discloses another video editing system for vehicle door status based on a diffusion model and physical priors, the system comprising:

[0046] Memory containing executable program code;

[0047] A processor coupled to the memory;

[0048] The processor calls the executable program code stored in the memory to execute the door state video editing method based on diffusion model and physical prior disclosed in the first aspect of the present invention.

[0049] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which, when invoked by a processor, are used to execute the video editing method for car door status based on diffusion model and physical prior disclosed in the first aspect of the present invention.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] By inputting real driving video sequences and their 3D annotation information, the system automatically selects suitable parked vehicles for editing and analyzes their multi-view visibility. Then, using a video diffusion model, it performs time-consistent fine-tuning of the door status within a precise region updated frame by frame, based on text instructions, to generate high-quality synthetic videos. Simultaneously, based on the deterministic physical relationship between door size and opening angle, it automatically calculates the expansion distance, expands the 3D bounding box in one go, and projects it to generate full-sequence 2D annotations. Thus, while fully preserving the real scene background, it simultaneously solves the dual problems of scarce visual data and lack of accurate annotations in door opening scenes through an automated closed-loop process, providing a low-cost, high-realism training data source for autonomous driving perception models. Attached Figure Description

[0052] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart illustrating the video editing method for vehicle door status based on diffusion model and physical prior disclosed in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of the vehicle door state video editing system based on diffusion model and physical prior disclosed in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of another vehicle door status video editing system based on diffusion model and physical prior disclosed in an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] This invention discloses a video editing method and system for car door states based on a diffusion model and physical priors. By inputting real driving videos and 3D annotations, the system automatically filters target vehicles and determines the editing perspective. It uses a video diffusion model to perform time-consistent fine-grained editing of the car door states in a specified area. Simultaneously, it automatically calculates and generates extended 3D and 2D annotations based on the physical relationship between the car door size and opening angle. Finally, after quality screening, a synthetic dataset is constructed. Thus, while preserving the characteristics of real scenes, it solves the core problems of scarce training data and missing annotations for car door opening scenes in a low-cost and automated manner.

[0060] Example 1

[0061] Please see Figure 1 , Figure 1 This is a flowchart illustrating the video editing method for vehicle door states based on a diffusion model and physical priors disclosed in an embodiment of the present invention. Figure 1 The described video editing method for vehicle door status based on diffusion models and physical priors is applied to data processing chips, processing terminals, or processing servers. The processing server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 1As shown, the video editing method for car door states based on the diffusion model and physical priors can include the following operations:

[0062] 101. Input a driving video sequence from at least one perspective and the corresponding 3D target detection and tracking annotation information.

[0063] Specifically, it provides a realistic visual scene foundation and accurate geometric reference information, enabling subsequent door status editing to be carried out in a realistic road context and ensuring the correspondence between the editing position and the actual three-dimensional position of the vehicle.

[0064] As can be seen, by directly using real driving videos and existing annotation systems as input, this invention can ensure that the generated data retains the original real features in non-editing areas such as background and lighting, while providing the necessary initialization data for spatial positioning of editing areas and annotation calculations based on physical priors.

[0065] 102. Based on the three-dimensional target detection and tracking annotation information, target vehicles are selected from the video sequence, and the visibility of the target vehicles in each frame and from each viewpoint is analyzed to determine the editing viewpoint for editing the door status and its effective frame range.

[0066] Specifically, through preset filtering conditions and geometric analysis, the system automatically identifies vehicle targets suitable for editing the door status from the input raw data, and accurately locks the camera angle and corresponding time segment that can observe the side of the door.

[0067] As can be seen, this process automates and refines the selection of editing targets and editing areas, providing a clear input range for subsequent targeted video editing and annotation generation operations. It avoids blind calculations across all perspectives in the entire sequence, improving the overall process's targeting and processing efficiency.

[0068] 103. For each valid frame of each editing viewpoint, the editing area is determined based on the three-dimensional bounding box projection of the target vehicle, and text editing instructions are constructed based on the target vehicle information.

[0069] Specifically, by projecting and expanding the 3D bounding box, a precise editing region mask that is updated frame by frame was obtained; at the same time, text instructions constructed based on information such as vehicle type and orientation clarified the specific semantic requirements for editing.

[0070] As can be seen, this process transforms abstract editing intentions into a spatial mask sequence and text prompts that the model can execute. This ensures that the video diffusion model can generate controllable content at the correct image location based on explicit semantic descriptions, ultimately achieving high-quality, meaningful content. Figure 1 This provides a foundation for editing the status of the car doors.

[0071] 104. Input the original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions into a video diffusion model that supports mask editing, perform time-consistent door state editing, and generate the edited video.

[0072] Specifically, by leveraging the core capabilities of the video diffusion model, new video content that matches the expected door status was generated within a specified editing area based on text instructions, and the temporal attention mechanism built into the model ensured the continuity of the generated content in the time dimension.

[0073] As can be seen, this process enables refined and controllable modification of the vehicle component state on a real video base, ultimately outputting a visually reasonable and frame-stable edited video, providing crucial synthetic visual data for subsequent annotation generation and model training.

[0074] 105. Based on the physical mapping relationship between the door size and the opening angle, calculate the lateral expansion distance after the door is opened, expand the three-dimensional bounding box of the target vehicle in three-dimensional space, and project the expanded three-dimensional bounding box onto each frame image of each editing viewpoint to generate corresponding two-dimensional annotations.

[0075] Specifically, by utilizing explicit physical laws, the state of the car door is transformed into specific geometric extensions, which in turn drive the automatic updating of 3D annotations and 2D projection, thus realizing the transformation from semantic intent to precise spatial annotation.

[0076] As can be seen, this invention generates labeled data for the edited vehicle that strictly matches the opening and closing state of its doors and maintains geometric consistency in three-dimensional space and two-dimensional images from all perspectives without human intervention, providing directly usable supervision information for the training of the perception model.

[0077] 106. Evaluate and screen the quality of the edited videos and annotations, and compile the samples that pass the quality screening and their annotations to construct a synthetic dataset for training autonomous driving perception models.

[0078] Specifically, multi-dimensional quality assessment and screening were performed on the videos and annotations generated during the synthesis process, removing samples with defects in image quality, temporal consistency, or detection consistency, thereby ensuring the reliability of the data used for subsequent training.

[0079] As can be seen, by integrating quality gating and screening mechanisms, this method can automatically construct a dataset composed of high-quality, compliant synthetic samples, providing a directly usable and effective source of supervised data for training autonomous driving perception models.

[0080] As an optional embodiment, the step described above, analyzing the visibility of the target vehicle in each frame and from each viewpoint, includes:

[0081] The 3D bounding box of the target vehicle is projected onto the image plane using camera parameters from each frame and viewpoint.

[0082] Views whose projection area is greater than a preset threshold and are completely within the image range are included in the set of visible viewpoints of that frame.

[0083] Based on the angle between the door opening direction vector and the optical axis direction of each view camera, the view that can observe the car body on the side with the door open is determined from the set of visible viewpoints as the editing viewpoint;

[0084] When the input contains multiple perspectives, the remaining visible perspectives are marked as consistency constraint perspectives.

[0085] In this embodiment of the invention, through a series of automated geometric calculations and logical judgments, effective viewpoints and frame ranges suitable for editing the vehicle door status are precisely selected from the original multi-view video data, providing a precise spatial basis for subsequent editing operations. This mainly includes:

[0086] 1. Project the 3D bounding box of the target vehicle onto the image plane using the camera's intrinsic and extrinsic parameters for each frame and viewpoint. This step associates the vehicle's position in 3D space with the pixel coordinates of the 2D image. Its core is to use the camera's perspective projection model for calculation, ensuring the accuracy of the spatial correspondence.

[0087] 2. Two objective selection criteria are set. First, the pixel area of ​​the projection region must be greater than a preset threshold. This ensures that the target vehicle has sufficient size and resolution in the image, providing a visually feasible basis for subsequent editing. Second, the projection region must be completely within the image boundary. This avoids incomplete editing caused by the target being truncated by the image edges. Only viewpoints that meet both conditions will be included in the set of visible viewpoints for the current frame.

[0088] 3. To ensure the door side being edited is visible, this step introduces a geometric relationship assessment between the door opening direction vector and the camera's optical axis. By calculating the angle between these two direction vectors, we can further filter from the set of visible viewpoints those where the camera's optical axis is approximately opposite to the normal direction of the door opening side. Only from these viewpoints is the door opening action visually observable; therefore, they are determined as the primary viewpoints for performing the editing operation, i.e., the editing viewpoints.

[0089] 4. When the system input includes multiple camera views, the remaining views that meet the basic visibility conditions but are not the optimal editing views are marked as consistency constraint views. These views will not be used as the main view for content generation in subsequent multi-view joint editing processes, but can be used as constraints to ensure that the generated car door state presents visual consistency under different views.

[0090] As can be seen, through layers of geometric screening and logical judgment, intelligent and automated selection of editing perspective is achieved. This not only ensures the effectiveness and integrity of the edited target in the image, but more importantly, it ensures that the selected perspective can realistically and clearly reflect the visual changes in the opening and closing state of the car door, and provides a clear input range for achieving high-quality multi-view collaborative editing, thereby improving the overall usability of the synthetic data.

[0091] As an optional embodiment, the step of determining the editing area based on the 3D bounding box projection of the target vehicle in the above steps specifically involves:

[0092] For each valid frame of each editing viewpoint, the three-dimensional bounding box is expanded along the door opening direction by a preset expansion ratio in the two-dimensional projection box of that viewpoint to obtain the editing region mask of that frame and that viewpoint.

[0093] The preset expansion ratio is determined according to the vehicle category;

[0094] Due to the movement of the vehicle, the edited area mask is updated frame by frame as the target vehicle's projection position changes.

[0095] In this embodiment of the invention, an accurate and dynamically changing editing region is generated in an automated manner that is closely related to the target geometry and motion state, providing reliable spatial operation guidance for subsequent video editing models.

[0096] Specifically, this method first utilizes camera projection relationships to transform the bounding box of the target vehicle in three-dimensional space into an initial rectangular region on each frame of the two-dimensional image. This projection box accurately defines the vehicle's basic position in the image.

[0097] Furthermore, to ensure that the editing operation fully covers the space that the car door might occupy when opened, this method does not simply use a projection frame, but rather expands it along the direction the door opens. The expansion ratio is preset according to the type of target vehicle; for example, the door sizes of sedans, SUVs, or trucks are different, and their preset expansion ratios are also different. This differentiated processing based on vehicle type makes the estimation of the editing area more closely match physical reality, avoiding the problem of the area being too large or too small caused by a universal fixed ratio.

[0098] More importantly, due to the potential relative motion between the vehicle and the target vehicle in a real-world scene, the target's position in the image changes frame by frame. This method achieves real-time updates to the edit region mask by reprojecting the 3D bounding box frame by frame and calculating its expanded region. This means that the edit region can closely follow the moving target vehicle, forming a dynamic mask that moves with the time sequence, thus ensuring the continuity and accuracy of the edit position throughout the video clip and providing a crucial spatial foundation for generating time-stable edit results.

[0099] As an optional embodiment, in the above steps, the video diffusion model that supports mask editing is a model that can receive the original video, spatial temporal mask and text instructions as conditional inputs, and perform conditional denoising generation within the mask area;

[0100] When multiple editing perspectives exist, the consistency of editing results across multiple perspectives is ensured by assigning the same random noise seed and the same text editing instructions to each perspective.

[0101] In this embodiment of the invention, a practical solution is provided that utilizes the core capabilities of existing video generation models and applies lightweight control to achieve consistency in multi-view editing, taking into account both effectiveness and feasibility of implementation.

[0102] Specifically, the scheme first defines the functional requirements for the video diffusion model. The model must be able to receive multimodal conditional inputs, including the original video to be edited, predefined edit region masks in spatial and temporal dimensions, and text instructions describing the editing intent. During inference, the model uses the original video as a base, and within the region specified by the mask, guided by the text instructions, performs a series of denoising steps to ultimately generate new content that meets the requirements, while non-masked areas remain unchanged. This ensures the accuracy and controllability of the editing process.

[0103] To address the challenge of multi-view editing, this invention proposes an efficient guarantee strategy that requires no additional model training. Its core lies in controlling two key variables in the generation process: first, assigning the same random noise seed to the generation process for all editing perspectives; and second, using identical text editing instructions.

[0104] In the generative framework of the diffusion model, the random noise seed determines the initial state of the denoising process. Setting the same seed for all perspectives means that their generation processes start from the exact same random noise starting point. The same text instructions provide a unified semantic optimization direction for the generation processes of all perspectives. This is equivalent to allowing multiple independent generation tasks to start from a common starting point and move towards the same clear goal, thereby greatly increasing the probability that their outputs will maintain consistency in content and style.

[0105] As can be seen, this invention does not require modifying complex model structures or performing time-consuming additional training as with the introduction of cross-view attention mechanisms. Instead, it effectively promotes consistency among multiple perspectives in a low-cost manner by cleverly controlling the initial and input conditions of the generation process, providing reliable technical support for constructing high-quality multi-view synthetic data.

[0106] As an optional embodiment, the step above, calculating the lateral extension distance based on the physical mapping relationship between the door size and the opening angle, specifically involves:

[0107] Based on the target vehicle category, obtain the typical width dimensions of the corresponding door. ;

[0108] Determine the corresponding opening and closing angle based on the target opening and closing state. ;

[0109] According to the formula Calculate the lateral expansion distance.

[0110] In this embodiment of the invention, a deterministic and quantifiable physical model is introduced to automatically calculate the space occupied after the car door is opened, thereby transforming the ambiguous semantic state "car door open" into a precise geometric expansion quantity, providing a core basis for generating dimensionally accurate annotation data.

[0111] Specifically, this method establishes a clear mapping relationship from door status to spatial expansion. First, based on the target vehicle category, such as sedan, SUV, or truck, it queries or retrieves the preset typical width dimensions of doors for that category, denoted as […]. This is a priori physical dimension parameter.

[0112] Next, based on the desired door opening / closing state, such as 30 degrees, 60 degrees, or 90 degrees, a specific opening / closing angle value is determined. This angle It defines the angle through which the door rotates relative to the closed position.

[0113] The most crucial step is applying geometric principles to perform calculations. This method is based on formulas. To calculate the lateral extension distance. The physical meaning here is: approximate the car door as a rigid rectangle rotating around the hinge. When the car door at an angle... When opened, the maximum lateral displacement of its outermost edge in the direction perpendicular to the side of the vehicle body is precisely the width of the door. Multiply by the sine of the door opening angle θ sine function It accurately reflects the nonlinear relationship of lateral displacement as the angle changes.

[0114] Therefore, the core contribution of this invention is to provide a physical, deterministic calculation method. It avoids the errors in annotation dimensions caused by relying on manual estimation or fixed empirical values, ensuring that regardless of the vehicle model or opening angle, the expansion of the generated annotation frame in the horizontal width conforms to the actual external contour dimensions of the vehicle in that state, thereby improving the geometric authenticity and physical reliability of the synthesized annotation data.

[0115] As an optional embodiment, the step of expanding the three-dimensional bounding box of the target vehicle in three-dimensional space in the above steps specifically involves:

[0116] Along the direction the car door opens, increase the lateral expansion distance in the width dimension of the three-dimensional bounding box. And offset the center coordinates along the direction the car door opens. ;

[0117] Since the target vehicle is stationary, the expansion amount remains consistent across all frames.

[0118] In this embodiment of the invention, a clear, operable and efficient three-dimensional spatial geometric transformation method is provided, which accurately transforms the abstract lateral expansion distance calculated in the previous step into a new three-dimensional bounding box that can truly reflect the space occupied by the vehicle and its open door, and provides an accurate geometric basis for the subsequent generation of two-dimensional annotations from various perspectives.

[0119] Specifically, this step involves two core geometric operations. First, it determines the direction in which the expansion operation is applied, i.e., along the direction the door opens. Next, it performs a specific transformation: increasing the width dimension of the original 3D bounding box by the calculated lateral expansion distance. Geometrically, this means that the original bounding box only encloses the car body when the doors are closed, while the expanded bounding box includes the additional lateral space occupied by the doors when they are opened at a specific angle.

[0120] Simply increasing the width would result in the new bounding box center remaining at the original vehicle body center, which does not reflect the actual situation where the vehicle's overall center of gravity or spatial center shifts after the door is opened. Therefore, the second key operation in this step is to offset the center coordinates of the bounding box along the same door opening direction. The distance. This offset is set based on geometric symmetry considerations: when the width increases... Then, move the center point in the opening direction by half the width of the new width to ensure that the new bounding box can evenly wrap around the "original car body" and the "open door" as a whole.

[0121] This invention clarifies an important prerequisite for implementation: since the target vehicle is set to be stationary or parked, the calculated 3D bounding box expansion, including the width increment and center offset, remains constant across all frames of the entire video sequence. This setting offers significant efficiency advantages. It means that for the same target vehicle and the same door state, complex 3D spatial geometry calculations only need to be performed once. Subsequently, this unified, expanded 3D bounding box can be projected onto the image plane of each editing viewpoint in each frame using the varying camera extrinsic and intrinsic parameter matrices of each frame, thereby automatically generating 2D annotations that naturally change over time and with different viewpoints.

[0122] As can be seen, this invention successfully transforms a physical law (the extended distance of a car door opening) into a precise 3D geometric processing flow that can be automatically executed. It not only ensures the accuracy of the generated 3D annotations in terms of spatial size and position, reflecting the actual physical occupancy, but also significantly improves the computational efficiency of the entire annotation generation process through a "one-time calculation, full-sequence projection" mechanism, ensuring geometric consistency of annotations across multiple views and frames.

[0123] As an optional embodiment, the quality assessment and screening of the edited video and annotations in the above steps includes at least one of the following assessment dimensions:

[0124] Single-frame image quality assessment evaluates the quality score of the image within the edited area.

[0125] Temporal consistency assessment evaluates the smoothness of changes in edited regions between consecutive frames.

[0126] Consistency verification is performed to verify whether the edited vehicle can be continuously and correctly detected by the pre-trained vehicle detection model.

[0127] Cross-view consistency check: When there are multiple editing views, evaluate the similarity of appearance features of the door area in different views at the same moment.

[0128] Boundary transition naturalness assessment evaluates the degree of naturalness of the transition between the edited area and the original video boundary.

[0129] In this embodiment of the invention, a multi-dimensional, automated quality gating mechanism is established. This mechanism can systematically identify and filter out potentially defective samples in the synthesized results from different technical levels, thereby ensuring the overall reliability and usability of the final output dataset and providing quality assurance for the training of downstream models.

[0130] Specifically, this invention defines several key and complementary evaluation dimensions. Single-frame image quality evaluation primarily focuses on the visual realism of the edited region itself in static scenes, such as sharpness and texture reasonableness, avoiding low-level image defects such as blurriness or structural distortion. Temporal consistency evaluation, from the perspective of dynamic video, measures the smoothness and coherence of changes in the edited region between consecutive frames, which is crucial to ensuring that the generated content appears natural and flicker-free over time. Detection consistency verification introduces an external, pre-trained vehicle detection model as an objective criterion, verifying whether the edited vehicle can be continuously and stably detected by mature perception algorithms. This verifies the "detectability" of the edited target and indirectly ensures that the vehicle's appearance and position do not undergo distortions that violate common sense due to editing. Cross-view consistency checking is specifically designed for multi-view input scenarios. By comparing the appearance features of the door area in different camera views at the same time, it ensures that the generated door state is consistent and uniform under different observation angles, which is key to constructing high-fidelity multi-view data. The boundary transition naturalness assessment focuses on the junction between the edited area and the original unmodified area of ​​the video, evaluating whether the blending of color, lighting and texture is natural, and avoiding stiff and unrealistic splicing marks.

[0131] By comprehensively applying one or more of the above-mentioned dimensions of evaluation, this method can effectively identify synthetic samples that have problems in any aspect of visual quality, temporal stability, algorithm compatibility, multi-view coordination, or boundary fusion. Only samples that pass this set of composite screening criteria will be retained and incorporated into the final dataset.

[0132] As can be seen, this invention is not a single quality check, but a systematic quality filtering pipeline. By introducing multi-level, multi-angle automated evaluation criteria, it significantly improves the overall quality and credibility of synthetic datasets, ensuring that the training data provided for autonomous driving perception models is not only sufficient in quantity, but also meets high standards in terms of visual realism, spatiotemporal consistency, and algorithm usability, thereby directly improving the performance of models trained using this data.

[0133] As an optional embodiment, the video diffusion model supporting mask editing in the above steps includes the VACE model from Alibaba Tongyi Labs or a video editing model based on the Wan series.

[0134] In this embodiment of the invention, specific model names are explicitly specified, such as the VACE model developed by Alibaba Tongyi Labs, or video editing models based on the Wan architecture known in the industry, which has significant practical implications. These models are not new creations of this invention, but are referenced as components of existing technologies. Their mention is intended to illustrate the existence of commercially available video diffusion models that meet the aforementioned technical requirements and support mask editing, and that their functionality is publicly available.

[0135] Taking the VACE model as an example, it is a video generation and editing framework publicly released by Alibaba's Tongyi Lab. It has the ability to accept text, images, or masks as conditional inputs and perform content editing in specified areas of the video. Video editing models based on the Wan series also refer to a class of diffusion models known in public research or open source communities that adopt a similar architecture to the Wan model and focus on video editing tasks.

[0136] As can be seen, this invention links an abstract technical requirement, "a video diffusion model supporting mask editing," with a specific, publicly available implementation. This clarifies that the solution is not built on a fictitious or future technology, but rather can be immediately constructed and implemented based on existing, mature open-source models or publicly available research results, lowering the barrier to understanding and the difficulty of reproduction. This invention focuses on how to systematically utilize such models to solve the specific problem of door state editing and annotation generation, rather than protecting the model itself.

[0137] Example 2

[0138] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of the vehicle door state video editing system based on diffusion model and physical prior, as disclosed in an embodiment of the present invention. Figure 2 The described video editing system for vehicle door status based on diffusion models and physical priors can be applied to data processing chips, processing terminals, or processing servers. The processing server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 2 As shown, the door status video editing system based on the diffusion model and physical priors can include the following operations:

[0139] The information input module 201 is used to input a driving video sequence from at least one perspective and the corresponding three-dimensional target detection and tracking annotation information.

[0140] Specifically, it provides a realistic visual scene foundation and accurate geometric reference information, enabling subsequent door status editing to be carried out in a realistic road context and ensuring the correspondence between the editing position and the actual three-dimensional position of the vehicle.

[0141] As can be seen, by directly using real driving videos and existing annotation systems as input, this invention can ensure that the generated data retains the original real features in non-editing areas such as background and lighting, while providing the necessary initialization data for spatial positioning of editing areas and annotation calculations based on physical priors.

[0142] The target analysis module 202 is used to filter target vehicles from the video sequence based on the three-dimensional target detection and tracking annotation information, and analyze the visibility of the target vehicles in each frame and from each viewpoint to determine the editing viewpoint and its effective frame range for editing the door status.

[0143] Specifically, through preset filtering conditions and geometric analysis, the system automatically identifies vehicle targets suitable for editing the door status from the input raw data, and accurately locks the camera angle and corresponding time segment that can observe the side of the door.

[0144] As can be seen, this process automates and refines the selection of editing targets and editing areas, providing a clear input range for subsequent targeted video editing and annotation generation operations. It avoids blind calculations across all perspectives in the entire sequence, improving the overall process's targeting and processing efficiency.

[0145] The region instruction module 203 is used to determine the editing region based on the three-dimensional bounding box projection of the target vehicle for each valid frame of each editing view, and to construct text editing instructions based on the target vehicle information.

[0146] Specifically, by projecting and expanding the 3D bounding box, a precise editing region mask that is updated frame by frame was obtained; at the same time, text instructions constructed based on information such as vehicle type and orientation clarified the specific semantic requirements for editing.

[0147] As can be seen, this process transforms abstract editing intentions into a spatial mask sequence and text prompts that the model can execute. This ensures that the video diffusion model can generate controllable content at the correct image location based on explicit semantic descriptions, ultimately achieving high-quality, meaningful content. Figure 1 This provides a foundation for editing the status of the car doors.

[0148] The video editing module 204 is used to input the original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions into a video diffusion model that supports mask editing, to perform time-consistent door state editing, and generate the edited video.

[0149] Specifically, by leveraging the core capabilities of the video diffusion model, new video content that matches the expected door status was generated within a specified editing area based on text instructions, and the temporal attention mechanism built into the model ensured the continuity of the generated content in the time dimension.

[0150] As can be seen, this process enables refined and controllable modification of the vehicle component state on a real video base, ultimately outputting a visually reasonable and frame-stable edited video, providing crucial synthetic visual data for subsequent annotation generation and model training.

[0151] The annotation generation module 205 is used to calculate the lateral expansion distance after the door is opened based on the physical mapping relationship between the door size and the opening angle, expand the three-dimensional bounding box of the target vehicle in three-dimensional space, and project the expanded three-dimensional bounding box onto each frame image of each editing view to generate corresponding two-dimensional annotations.

[0152] Specifically, by utilizing explicit physical laws, the state of the car door is transformed into specific geometric extensions, which in turn drive the automatic updating of 3D annotations and 2D projection, thus realizing the transformation from semantic intent to precise spatial annotation.

[0153] As can be seen, this invention generates labeled data for the edited vehicle that strictly matches the opening and closing state of its doors and maintains geometric consistency in three-dimensional space and two-dimensional images from all perspectives without human intervention, providing directly usable supervision information for the training of the perception model.

[0154] Module 206 is used to perform quality assessment and screening of the edited videos and annotations, and to summarize the samples and their annotations that pass the quality screening to build a synthetic dataset for training autonomous driving perception models.

[0155] Specifically, multi-dimensional quality assessment and screening were performed on the videos and annotations generated during the synthesis process, removing samples with defects in image quality, temporal consistency, or detection consistency, thereby ensuring the reliability of the data used for subsequent training.

[0156] As can be seen, by integrating quality gating and screening mechanisms, this method can automatically construct a dataset composed of high-quality, compliant synthetic samples, providing a directly usable and effective source of supervised data for training autonomous driving perception models.

[0157] Example 3

[0158] Please see Figure 3 , Figure 3 This is a schematic diagram of another vehicle door state video editing system based on a diffusion model and physical priors disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include:

[0159] Memory 301 storing executable program code;

[0160] Processor 302 coupled to memory 301;

[0161] The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in the video editing method for car door status based on diffusion model and physical prior disclosed in Embodiment 1 of the present invention.

[0162] Example 4

[0163] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in the video editing method for car door status based on diffusion model and physical prior disclosed in Embodiment 1 of this invention.

[0164] Example 5

[0165] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the video editing method for door status based on diffusion model and physical prior described in Embodiment 1.

[0166] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0168] Finally, it should be noted that the video editing method and system for car door status based on diffusion model and physical prior disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video editing method for vehicle door status based on diffusion model and physical priors, characterized in that, The method includes: Input a driving video sequence from at least one perspective and the corresponding 3D target detection and tracking annotation information; Based on the three-dimensional target detection and tracking annotation information, target vehicles are selected from the video sequence, and the visibility of the target vehicles in each frame and from each viewpoint is analyzed to determine the editing viewpoint for editing the door status and its effective frame range. For each valid frame of each editing viewpoint, the editing area is determined based on the 3D bounding box projection of the target vehicle, and text editing instructions are constructed based on the target vehicle information; The original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions are input into a video diffusion model that supports mask editing to perform time-consistent door state editing and generate the edited video. Based on the physical mapping relationship between the door size and the opening angle, the lateral expansion distance after the door is opened is calculated, the three-dimensional bounding box of the target vehicle is expanded in three-dimensional space, and the expanded three-dimensional bounding box is projected onto each frame image of each editing view to generate corresponding two-dimensional annotations. The quality of the edited videos and annotations is assessed and screened. By aggregating the quality-filtered samples and their annotations, a synthetic dataset is constructed for training autonomous driving perception models.

2. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The analysis of the visibility of the target vehicle in each frame and from each viewpoint includes: The 3D bounding box of the target vehicle is projected onto the image plane using camera parameters from each frame and viewpoint. Views whose projection area is greater than a preset threshold and are completely within the image range are included in the set of visible viewpoints of that frame. Based on the angle between the door opening direction vector and the optical axis direction of each view camera, the view that can observe the car body on the side with the door open is determined from the set of visible viewpoints as the editing viewpoint; When the input contains multiple perspectives, the remaining visible perspectives are marked as consistency constraint perspectives.

3. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The determination of the editing area based on the 3D bounding box projection of the target vehicle specifically involves: For each valid frame of each editing viewpoint, the three-dimensional bounding box is expanded along the door opening direction by a preset expansion ratio in the two-dimensional projection box of that viewpoint to obtain the editing region mask of that frame and that viewpoint. The preset expansion ratio is determined according to the vehicle category; Due to the movement of the vehicle, the edited area mask is updated frame by frame as the target vehicle's projection position changes.

4. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The video diffusion model that supports mask editing is a model that can receive the original video, spatial temporal mask and text instructions as conditional inputs, and perform conditional denoising generation within the mask area. When multiple editing perspectives exist, the consistency of editing results across multiple perspectives is ensured by assigning the same random noise seed and the same text editing instructions to each perspective.

5. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The calculation of the lateral extension distance based on the physical mapping relationship between the door size and the opening angle is as follows: Based on the target vehicle category, obtain the typical width dimensions of the corresponding door. ; Determine the corresponding opening and closing angle based on the target opening and closing state. ; According to the formula Calculate the lateral expansion distance.

6. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 5, characterized in that, The three-dimensional bounding box of the target vehicle in three-dimensional space is extended, specifically as follows: Along the direction the car door opens, increase the lateral expansion distance in the width dimension of the three-dimensional bounding box. And offset the center coordinates along the direction the car door opens. ; Since the target vehicle is stationary, the expansion amount remains consistent across all frames.

7. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The quality assessment and screening of the edited and annotated videos includes at least one of the following assessment dimensions: Single-frame image quality assessment evaluates the quality score of the image within the edited area. Temporal consistency assessment evaluates the smoothness of changes in edited regions between consecutive frames. Consistency verification is performed to verify whether the edited vehicle can be continuously and correctly detected by the pre-trained vehicle detection model. Cross-view consistency check: When there are multiple editing views, evaluate the similarity of appearance features of the door area in different views at the same moment. Boundary transition naturalness assessment evaluates the degree of naturalness of the transition between the edited area and the original video boundary.

8. The video editing method for vehicle door status based on diffusion model and physical prior as described in claim 1, characterized in that, The video diffusion model that supports mask editing includes the VACE model from Alibaba Tongyi Labs or a video editing model based on the Wan series.

9. A vehicle door state video editing system based on diffusion model and physical prior, used to implement the vehicle door state video editing method based on diffusion model and physical prior as described in any one of claims 1-8, characterized in that, The system includes: The information input module is used to input a driving video sequence from at least one perspective and the corresponding 3D target detection and tracking annotation information; The target analysis module is used to filter target vehicles from the video sequence based on the three-dimensional target detection and tracking annotation information, and analyze the visibility of the target vehicles in each frame and from each viewpoint to determine the editing viewpoint for editing the door status and its effective frame range. The region instruction module is used to determine the editing region based on the three-dimensional bounding box projection of the target vehicle for each valid frame of each editing view, and to construct text editing instructions based on the target vehicle information. The video editing module is used to input the original video clips from each editing perspective, the corresponding frame-by-frame editing area information, and the text editing instructions into a video diffusion model that supports mask editing, to perform time-consistent door state editing, and generate the edited video; The annotation generation module is used to calculate the lateral expansion distance after the door is opened based on the physical mapping relationship between the door size and the opening angle, expand the three-dimensional bounding box of the target vehicle in three-dimensional space, and project the expanded three-dimensional bounding box onto each frame image of each editing view to generate corresponding two-dimensional annotations. The module is used to evaluate and filter the quality of the edited videos and annotations, and to summarize the samples that pass the quality screening and their annotations to build a synthetic dataset for training autonomous driving perception models.

10. A video editing system for vehicle door status based on a diffusion model and physical priors, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the door state video editing method based on diffusion model and physical prior as described in any one of claims 1-8.

Images