Video editing method and device based on two-dimensional gaussian function, equipment and medium

Abstract

The present application relates to the technical field of video editing, and particularly relates to a video editing method and device based on a two-dimensional Gaussian function, equipment and a medium, the method first acquires a video to be edited, a video frame sequence corresponding to the video to be edited, prompt text and a pre-trained diffusion model, generates a Gaussian distribution set based on the video frame sequence and constructs an optimized three-dimensional scene model, inputs the prompt text into the diffusion model, generates an initial edited video frame, freezes the geometric parameters of the optimized three-dimensional scene model, renders all Gaussian distributions of the frozen three-dimensional scene model, optimizes the appearance parameters of the frozen three-dimensional scene model based on the obtained rendered image and the initial edited video frame, obtains a target three-dimensional scene model, acquires the video to be edited, pre-processes the video to be edited, inputs each obtained pre-processed video segment into the target three-dimensional scene model for frame-by-frame rendering, and obtains a target edited video, thereby realizing high-quality text-guided video editing.

Description

Technical Field

[0001] This invention relates to the field of video editing technology, and in particular to a video editing method, apparatus, device, and medium based on a two-dimensional Gaussian function. Background Technology

[0002] The breakthrough development of diffusion generation models has significantly improved the quality and effect of text-guided image editing. Building related visual applications based on this technical framework has become an important trend in the industry, with applications covering a wide range of fields such as film production, commercial advertising, and immersive media in augmented reality and virtual reality.

[0003] However, directly training a dedicated video diffusion network faces unavoidable technical obstacles in practical implementation. Firstly, large, diverse, and meticulously organized video corpora are scarce, failing to provide sufficient data support for network training. Secondly, directly abandoning mature and efficient image diffusers wastes technical and computational resources, while also degrading the quality of generated video frames. To address these issues, zero-shot video editing workflows that require no training have gradually emerged. These solutions reuse frozen image diffusion models for video editing, balancing processing speed and visual fidelity. However, these methods still employ a purely two-dimensional inference model, making it difficult to maintain temporal continuity when handling scenes with significant motion or depth changes. Issues such as temporal discrepancies and inconsistencies in detail can easily arise, failing to meet the application requirements of high-quality text-guided video editing and limiting the large-scale application of related technologies in real-world scenarios. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a video editing method, apparatus, device and medium based on a two-dimensional Gaussian function, which aims to ensure the continuity of video timing and realize high-quality text-guided video editing.

[0005] The first aspect of this invention provides a video editing method based on a two-dimensional Gaussian function, comprising: acquiring a video to be edited, a video frame sequence corresponding to the video to be edited, a prompt text, and a pre-trained diffusion model; generating a Gaussian distribution set based on the video frame sequence, and constructing an optimized three-dimensional scene model based on the Gaussian distribution set; inputting the prompt text into the diffusion model to generate an initial edited video frame; freezing the geometric parameters of the optimized three-dimensional scene model to obtain a frozen three-dimensional scene model, rendering all Gaussian distributions of the frozen three-dimensional scene model to obtain a rendered image; optimizing the appearance parameters of the frozen three-dimensional scene model based on the rendered image and the initial edited video frame to obtain a target three-dimensional scene model; preprocessing the video to be edited to obtain multiple preprocessed video segments, and inputting each preprocessed video segment into the target three-dimensional scene model for frame-by-frame rendering to obtain the target edited video.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the video frame sequence includes multiple video frames; generating a Gaussian distribution set based on the video frame sequence includes: using an automatic video object segmentation network to distinguish the foreground region and background region of each video frame; using a mask cropping variant to sparsely reconstruct the foreground region and background region of each video frame to obtain a foreground point cloud and a background point cloud; calculating a loose boundary volume based on the foreground point cloud and the background point cloud, and generating multiple sets of supplementary background points based on the loose boundary volume; merging the multiple sets of supplementary background points with the background point cloud to obtain a target background point cloud; generating a foreground Gaussian distribution subset based on the foreground point cloud, and generating a background Gaussian distribution subset based on the target background point cloud; integrating the foreground Gaussian distribution subset and the background Gaussian distribution subset to obtain the Gaussian distribution set.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of constructing an optimized 3D scene model based on the Gaussian distribution set includes: constructing a foreground two-dimensional Gaussian scattering set based on the foreground Gaussian distribution subset, and constructing a background two-dimensional Gaussian scattering set based on the background Gaussian distribution subset; integrating the foreground two-dimensional Gaussian scattering set and the background two-dimensional Gaussian scattering set to obtain an initial 3D scene model; using a differentiable renderer to perform differentiable rendering on the foreground two-dimensional Gaussian scattering set and the background two-dimensional Gaussian scattering set respectively to obtain a foreground rendered image and a background rendered image; obtaining a learnable blending map, and generating an overall rendered image based on the learnable blending map, the foreground rendered image, and the background rendered image; obtaining a real video frame corresponding to the overall rendered image, calculating a reconstruction loss based on the overall rendered image and the real video frame, and optimizing the parameters of the initial 3D scene model based on the reconstruction loss to obtain the optimized 3D scene model.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of optimizing the parameters of the initial 3D scene model based on the reconstruction loss to obtain the optimized 3D scene model includes: introducing depth distortion loss and normal consistency loss into the reconstruction loss to construct a total loss function; iteratively updating the parameters of the initial 3D scene model using an adaptive density enhancement mechanism based on the total loss function; and outputting the optimized 3D scene model when a preset iteration stopping condition is met.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the step of iteratively updating the parameters of the initial 3D scene model using an adaptive density enhancement mechanism based on the total loss function includes: updating the parameters of the initial 3D scene model in one round based on the total loss function to obtain multiple current Gaussian distributions, and determining a detail missing metric for each current Gaussian distribution; obtaining preset filtering rules and preset segmentation rules, and based on the filtering rules, selecting at least one Gaussian distribution to be segmented from all the current Gaussian distributions according to the detail missing metric; segmenting the Gaussian distribution to be segmented based on the segmentation rules to obtain a new Gaussian distribution; introducing the new Gaussian distribution into the initial 3D scene model to replace the corresponding Gaussian distribution to be segmented, thereby obtaining an updated 3D scene model; and iteratively updating the parameters of the updated 3D scene model using an adaptive density enhancement mechanism based on the total loss function.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the appearance parameters include color parameters and opacity parameters. The step of optimizing the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model includes: constructing a reconstruction loss objective function based on the rendered image and the initial edited video frame; and optimizing the color parameters and opacity parameters of all Gaussian distributions in the frozen 3D scene model according to the reconstruction loss objective function to obtain the target 3D scene model.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the step of preprocessing the video to be edited to obtain multiple preprocessed video segments, and inputting each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain the target edited video, includes: obtaining a preset segment division rule; dividing the video to be edited into multiple video segments based on the segment division rule; using an appearance parameter averaging algorithm to perform temporal smoothing processing on the overlapping frame regions of adjacent segments in the video segments to obtain multiple preprocessed video segments; inputting each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain multiple target edited video segments; and splicing the multiple target edited video segments in chronological order to obtain the target edited video.

[0012] A second aspect of the present invention provides a video editing device based on a two-dimensional Gaussian function, comprising: a data acquisition module for acquiring a video to be edited, a video frame sequence corresponding to the video to be edited, prompt text, and a pre-trained diffusion model; a model construction module for generating a Gaussian distribution set based on the video frame sequence, and constructing an optimized three-dimensional scene model based on the Gaussian distribution set; a diffusion module for inputting the prompt text into the diffusion model to generate an initial edited video frame; a freezing module for freezing the geometric parameters of the optimized three-dimensional scene model to obtain a frozen three-dimensional scene model, and rendering all Gaussian distributions of the frozen three-dimensional scene model to obtain a rendered image; an optimization module for optimizing the appearance parameters of the frozen three-dimensional scene model based on the rendered image and the initial edited video frame to obtain a target three-dimensional scene model; and an editing module for preprocessing the video to be edited to obtain multiple preprocessed video segments, and inputting each preprocessed video segment into the target three-dimensional scene model for frame-by-frame rendering to obtain a target edited video.

[0013] A third aspect of the present invention provides a video editing device based on a two-dimensional Gaussian function, the video editing device based on a two-dimensional Gaussian function comprising: a memory and at least one processor, the memory storing instructions; at least one of the processors calling the instructions in the memory to cause the video editing device based on a two-dimensional Gaussian function to perform the various steps of the video editing method based on a two-dimensional Gaussian function described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the video editing method based on a two-dimensional Gaussian function as described above.

[0015] In the technical solution of this invention, the following steps are first taken: First, the video to be edited, the corresponding video frame sequence, prompt text, and a pre-trained diffusion model are obtained. A Gaussian distribution set is generated based on the video frame sequence, and an optimized 3D scene model is constructed based on the Gaussian distribution set. The prompt text is input into the diffusion model to generate initial edited video frames. The geometric parameters of the optimized 3D scene model are frozen to obtain a frozen 3D scene model. All Gaussian distributions of the frozen 3D scene model are rendered to obtain a rendered image. Then, the appearance parameters of the frozen 3D scene model are optimized based on the rendered image and the initial edited video frames to obtain a target 3D scene model. Finally, the video to be edited is obtained, preprocessed to obtain multiple preprocessed video segments, and each preprocessed video segment is input into the target 3D scene model for frame-by-frame rendering to obtain the target edited video. This approach aims to ensure the temporal continuity of the video and achieve high-quality text-guided video editing. Attached Figure Description

[0016] Figure 1 A logic flowchart of a video editing method based on a two-dimensional Gaussian function provided in an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the structure of a video editing device based on a two-dimensional Gaussian function provided in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of a video editing device based on a two-dimensional Gaussian function, provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a video editing method, apparatus, device, and medium based on a two-dimensional Gaussian function. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the video editing method based on a two-dimensional Gaussian function in this invention includes:

[0021] 101. Obtain the video to be edited, the video frame sequence corresponding to the video to be edited, the prompt text, and the pre-trained diffusion model;

[0022] In this embodiment, the video frame sequence is a continuous set of visual frames extracted frame by frame from the video to be edited. Each frame contains information about the spatial structure of the scene, the appearance of objects, and temporal correlations. It serves as input data for 3D scene reconstruction, providing fundamental visual and geometric constraints for the subsequent generation of Gaussian distribution sets and the construction of the optimized 3D scene model. The prompt text is a text instruction defined by the user or editing task, used to clarify the target effect of the video editing, such as adjusting the color, style, or content of objects in the scene. It is the core semantic guidance for text-driven editing, providing a clear semantic direction for the diffusion model to generate the initial edited video frames. The pre-trained diffusion model can adopt mainstream text-to-image diffusion frameworks such as Stable Diffusion. Its training process relies on a large-scale text-image pair dataset, and through multiple rounds of iterative optimization, it learns the mapping relationship between text semantics and visual features, possessing the ability to generate high-quality images guided by text. In this solution, the pre-trained model is reused to generate the initial edited video frames, eliminating the need to train a dedicated video diffusion network separately. This avoids the problem of missing large-scale, high-quality video corpora and also avoids the waste of computing resources and the risk of video frame quality degradation caused by discarding mature image diffusers.

[0023] 102. Generate a Gaussian distribution set based on the video frame sequence, and construct an optimized 3D scene model based on the Gaussian distribution set;

[0024] In this embodiment, when generating a Gaussian distribution set based on a video frame sequence, the foreground and background regions of each frame are first distinguished by an automatic video object segmentation network. A mask cropping variant is then used to sparsely reconstruct the foreground and background to obtain foreground point clouds and background point clouds. The loose boundary volume is calculated to generate supplementary background points, which are then merged with the background point cloud to obtain the target background point cloud. A foreground Gaussian distribution subset and a background Gaussian distribution subset are then generated based on the foreground point cloud and the target background point cloud, respectively. The two subsets are then integrated to obtain the Gaussian distribution set.

[0025] In this embodiment, when constructing an optimized 3D scene model based on the Gaussian distribution set, a foreground 2D Gaussian scattering set is first constructed based on a subset of the foreground Gaussian distribution, and a background 2D Gaussian scattering set is constructed based on a subset of the background Gaussian distribution. Both sets are 2D Gaussian scattering (2DGS) components. Each Gaussian distribution is defined as an oriented planar disk, and the supporting plane is fixed in the world coordinate system to ensure the consistency of geometry under multiple viewpoints. Integrating these two sets yields an initial 3D scene model, which is a 3D scene representation model based on 2D Gaussian scattering. A differentiable renderer is used to perform differentiable rendering on the foreground and background 2D Gaussian scattering sets respectively, obtaining foreground and background rendered images. The differentiable renderer uses perspective... The rendering process is achieved through precise scattering. The intersection points of light rays with the plane containing the Gaussian distribution are calculated and their contributions are accumulated to generate a differentiable rendering result. A learnable blending map is obtained, which maintains an alpha channel mask for each video frame. An overall rendered image is generated based on the learnable blending map, the foreground rendered image, and the background rendered image, achieving seamless compositing of the foreground and background. The real video frame corresponding to the overall rendered image is obtained. The reconstruction loss is calculated based on the overall rendered image and the real video frame. The parameters of the initial 3D scene model are optimized based on the reconstruction loss to obtain an optimized 3D scene model. This optimized model is still a 3D scene representation model based on 2D Gaussian scattering, which can reconstruct the input video frame with high fidelity and provide a stable 3D scene prior for subsequent editing.

[0026] 103. Input the prompt text into the diffusion model to generate the initial edited video frame;

[0027] In this embodiment, after the prompt text is input into the diffusion model, the model relies on the text semantic and visual feature mapping relationship learned in the pre-training stage to perform generation operations frame by frame according to the video time sequence, targeting the video's [frame number]. The frame position is used to generate corresponding visual content based on the editing target semantics of the prompt text, resulting in the initial edited video frame, denoted as . Because the diffusion model uses frame-by-frame independent generation logic, although the generated initial edited video frames can match the visual creativity of the prompt text, they lack global spatiotemporal constraints and are prone to problems such as inter-frame style flickering and inconsistent object shape timing. This step provides an accurate visual fitting benchmark for subsequent optimization based on the two-dimensional Gaussian scattering model.

[0028] 104. Freeze the geometric parameters of the optimized 3D scene model to obtain a frozen 3D scene model, and render all Gaussian distributions of the frozen 3D scene model to obtain a rendered image;

[0029] In this embodiment, the geometric parameters of the optimized 3D scene model are frozen, specifically the position, normals, dimensions, and blending graph parameters of all Gaussian distributions, allowing only appearance parameters to change, resulting in a frozen 3D scene model. This model fully preserves the reconstructed foreground and background spatial configurations, ensuring that the same object in different frames is represented by the same set of Gaussian distributions, maintaining the spatiotemporal consistency of the scene structure. Subsequently, a differentiable renderer is used to render all Gaussian distributions of the frozen 3D scene model. The rendering process is based on fixed geometric parameters and current appearance parameters, calculating the intersection points of rays with the planes containing the Gaussian distributions and accumulating their contributions to generate a rendered image. Freezing the geometric parameters stabilizes the 3D structure of the scene, preventing the position and shape of objects from drifting or deforming during editing, providing reliable geometric constraints for subsequent appearance optimization. The rendered image based on the frozen model accurately reflects the visual effect corresponding to the current appearance parameters, enabling subsequent optimization to accurately anchor the text-guided editing goals while maintaining the scene structure unchanged. This effectively solves common problems in pure 2D diffusion editing, such as temporal discrepancies and object flickering, improving the temporal coherence and structural fidelity of video editing.

[0030] 105. Optimize the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model;

[0031] In this embodiment, a reconstruction loss objective function is constructed based on the rendered image and the initial edited video frame. This function quantifies the visual differences between the rendered image and the initial edited video frame, providing a clear fitting target for subsequent optimization. Subsequently, the color parameters and opacity parameters of all Gaussian distributions in the frozen 3D scene model are optimized according to the reconstruction loss objective function. During the optimization process, the geometric parameters and blending map are strictly kept frozen, and the gradient only acts on the color parameters and opacity parameters to ensure that the 3D geometric structure and spatial configuration of the scene do not change. After optimization, the target 3D scene model is obtained. The appearance parameters of this model accurately match the text-guided editing target, while fully preserving the scene structure and spatiotemporal consistency reconstructed in the first stage. This optimization process effectively transfers the editing effects of the diffusion model to the 3D scene model. By fixing the geometry, it ensures that the appearance updates of corresponding areas in different frames remain consistent, avoiding the style flickering and object deformation problems common in pure 2D diffusion editing. It adopts a fragmented optimization approach to process long videos and scenes with large actions, and achieves a smooth style transition by updating the appearance on the overlapping frames of adjacent segments, further improving temporal coherence. The optimized target 3D scene model can accurately respond to the visual creativity guided by the text, and can completely retain the motion trajectory and depth information of the original video. It only makes targeted adjustments to the appearance, effectively solving the object drifting and temporal flickering problems common in unconstrained diffusion editing, and providing a reliable guarantee for the editing of complex dynamic scenes.

[0032] 106. The video to be edited is preprocessed to obtain multiple preprocessed video segments, and each preprocessed video segment is input into the target 3D scene model for frame-by-frame rendering to obtain the target edited video.

[0033] In this embodiment, when preprocessing the video to be edited, a preset segmentation rule is first obtained. Specifically, the video to be edited is divided into continuous short segments according to the time dimension, and a fixed-length overlapping frame region is set between adjacent segments. Based on this rule, the video to be edited is divided into multiple video segments, which can reduce the processing complexity of long videos and scenes with large actions, and at the same time, reserve a foundation for subsequent temporal consistency optimization. Subsequently, an appearance parameter averaging algorithm is used to perform temporal smoothing processing on the overlapping frame region of adjacent segments in the video segments. By calculating the mean of all Gaussian distributed color parameters and opacity parameters on the overlapping frames, the appearance of adjacent segments in the overlapping region is unified, and inter-frame consistency constraints are enforced to obtain multiple preprocessed video segments, eliminating the style abruptness problem at the editing boundary. Each preprocessed video segment is input into the target 3D scene model for frame-by-frame rendering. The rendering process relies on the frozen geometric parameters and optimized appearance parameters of the target 3D scene model to complete single-frame rendering according to the time sequence of the video frames, generating multiple target editing video segments. Finally, the multiple target editing video segments are spliced ​​together in time sequence to obtain the target editing video. This effectively solves the core problems of object drift, temporal flicker, and abrupt editing boundaries in unconstrained diffusion editing. The final output target editing video has visual creativity, structural fidelity, and temporal consistency, and can efficiently adapt to the text-driven editing needs of complex scenarios such as long videos and large actions.

[0034] In this embodiment of the invention, the video frame sequence includes multiple video frames; generating a Gaussian distribution set based on the video frame sequence includes: using an automatic video object segmentation network to distinguish the foreground region and background region of each video frame; using a mask cropping variant to sparsely reconstruct the foreground region and background region of each video frame to obtain a foreground point cloud and a background point cloud; calculating a loose boundary volume based on the foreground point cloud and the background point cloud, and generating multiple sets of supplementary background points based on the loose boundary volume; merging the multiple sets of supplementary background points with the background point cloud to obtain a target background point cloud; generating a foreground Gaussian distribution subset based on the foreground point cloud, and generating a background Gaussian distribution subset based on the target background point cloud; integrating the foreground Gaussian distribution subset and the background Gaussian distribution subset to obtain the Gaussian distribution set.

[0035] In this embodiment, the video frame sequence includes multiple video frames. When generating a Gaussian distribution set based on the video frame sequence, an automatic video object segmentation network is first used to distinguish the foreground and background regions of each video frame. This network uses the OSVOS semi-supervised video object segmentation architecture, with the DAVIS training set (including multiple video frame sequences, video frame resolutions, pixel-level segmentation annotations, etc.) as the core samples. MS COCO is used to initialize the weights of the VGG-16 backbone network, and the Adam optimizer is used for training. The input video frame resolution is uniformly 480×854. Finally, the binary mask of the foreground object in each frame is output, providing accurate segmentation constraints for subsequent sparse reconstruction. Then, a mask cropping variant, namely MC-COLMAP customized for dynamic videos, is used to sparsely reconstruct the foreground and background regions of each video frame. The point cloud of each segmentation point is reconstructed on short overlapping segments of the video, significantly reducing motion and parallax in each COLMAP run, resulting in a foreground point cloud for dynamic foregrounds and a background point cloud for static backgrounds. Then, based on the foreground point cloud and background point cloud, a loose boundary volume is calculated. For background areas not captured by COLMAP, random background points distributed within the loose boundary volume are initialized, generating multiple sets of supplementary background points. These supplementary background points are then merged with the background point cloud to obtain the target background point cloud, ensuring the integrity of the point cloud coverage in the background region and avoiding scene representation loss due to occlusion or feature loss. Subsequently, a foreground Gaussian distribution subset is generated based on the foreground point cloud, and a background Gaussian distribution subset is generated based on the target background point cloud. Each Gaussian distribution is defined as an oriented planar disk with a 3D position, a unit normal vector defining the local plane, and anisotropic size parameters along two orthogonal directions of the plane. The supporting plane is fixed in the world coordinate system, ensuring the consistency of geometry under multiple views. This differs from the volume ellipsoid design of traditional 3D Gaussian functions, improving the stability of multi-view geometric representation. The foreground and background Gaussian distribution subsets are integrated to obtain a set of Gaussian distributions.

[0036] In this embodiment of the invention, the step of constructing an optimized 3D scene model based on the Gaussian distribution set includes: constructing a foreground two-dimensional Gaussian scattering set based on the foreground Gaussian distribution subset, and constructing a background two-dimensional Gaussian scattering set based on the background Gaussian distribution subset; integrating the foreground two-dimensional Gaussian scattering set and the background two-dimensional Gaussian scattering set to obtain an initial 3D scene model; using a differentiable renderer to perform differentiable rendering on the foreground two-dimensional Gaussian scattering set and the background two-dimensional Gaussian scattering set respectively to obtain a foreground rendered image and a background rendered image; obtaining a learnable blending map, and generating an overall rendered image based on the learnable blending map, the foreground rendered image, and the background rendered image; obtaining a real video frame corresponding to the overall rendered image, calculating a reconstruction loss based on the overall rendered image and the real video frame, and optimizing the parameters of the initial 3D scene model based on the reconstruction loss to obtain the optimized 3D scene model.

[0037] In this embodiment, a foreground 2D Gaussian scattering set is constructed based on a subset of the foreground Gaussian distribution, and a background 2D Gaussian scattering set is constructed based on a subset of the background Gaussian distribution. Each Gaussian distribution is defined as an oriented planar disk with a 3D position, a unit normal vector defining the local plane, and anisotropic size parameters along two orthogonal directions of the plane. The supporting plane is fixed in the world coordinate system to ensure the consistency of geometry under multiple viewpoints. The foreground and background 2D Gaussian scattering sets are integrated to obtain an initial 3D scene model, which is a 3D scene representation model based on 2D Gaussian scattering. A differentiable renderer is used to perform differentiable rendering on the foreground and background 2D Gaussian scattering sets respectively. The differentiable renderer achieves rendering through a perspective-accurate scattering process. For each camera ray, the intersection point of the ray with the plane containing the Gaussian distribution is calculated and the contribution is accumulated to generate a differentiable rendering result, resulting in a foreground rendered image and a background rendered image. A learnable blending map is obtained, which is an alpha channel mask maintained for each video frame. An overall rendered image is generated based on the learnable blending map, the foreground rendered image, and the background rendered image. The specific calculation expression is as follows:

[0038] ,

[0039] in, Represents the pixel index in the horizontal direction of the image. The pixel index represents the vertical direction of the image, and together they locate the spatial position of any pixel in the image; This represents the final composite rendered image in pixel coordinates. The visual output at that location is the complete pixel information after the foreground and background images are merged at that pixel position; This represents the learnable blending map in pixel coordinates. The weight value at this location, ranging from 0 to 1, is initialized by the segmentation mask and optimized during training. It controls the fusion ratio of the foreground and background rendered images at this pixel position. The closer to 1, the more foreground image information is fused at that pixel; conversely, the less foreground image information is fused, the more background image information is fused. This represents the foreground image rendered from a two-dimensional Gaussian scattering set in pixel coordinates. The visual information at that location represents the performance of the foreground area at that pixel position; This represents the background image rendered from a 2D Gaussian scattering set in pixel coordinates. The visual information at that location represents the appearance of the background area at that pixel position. It can naturally learn to emphasize the foreground area where the object is located and rely on the background of other areas to ensure the seamless composition of the entire image.

[0040] Obtain the real video frames corresponding to the overall rendered image. Calculate the reconstruction loss based on the overall rendered image and the real video frames. This loss compares the rendered output with the real video frames under the foreground and background masks. The specific calculation expression is as follows:

[0041] ,

[0042] in, This represents the reconstruction loss, used to measure the visual difference between the entire rendered image and the real video frame, and to guide the optimization of model parameters. Represents the square of the L2 norm; The binary foreground mask representing the frame is generated by an automatic video object segmentation network. Its dimensions are consistent with the rendered image, with a pixel value of 1 for the foreground region and 0 for the background region. It is used to constrain the optimization range of the foreground and background rendered images pixel by pixel. This represents an element-wise multiplication operation, which multiplies the mask by the difference between the rendered image and the real frame pixel by pixel, achieving precise constraints for optimization of different regions. This represents the corresponding real video frame, which serves as the target benchmark for optimizing the entire image. Indicates the foreground rendered image. The background and background rendered images are the fundamental branches that constitute the overall rendered image. Optimizing the Gaussian parameters and mixture map parameters of the initial 3D scene model based on reconstruction loss allows the overall rendered image to faithfully reproduce the visual content of the real video frame, ultimately resulting in an optimized 3D scene model. In this process, the directional planar disk design of the foreground and background 2D Gaussian scattering sets, combined with a differentiable renderer, ensures the consistency of geometry across multiple viewpoints. The learnable mixture map achieves seamless fusion of the foreground and background rendered images. The reconstruction loss, by separately constraining the foreground and background rendering branches, ensures a high degree of matching between the overall rendered image and the real video frame, providing a stable 3D scene prior for subsequent video editing. This solves the problems of temporal discrepancies and object drift in pure 2D editing, providing a solid foundation for text-guided video editing.

[0043] In this embodiment of the invention, optimizing the parameters of the initial 3D scene model based on the reconstruction loss to obtain the optimized 3D scene model includes: introducing depth distortion loss and normal consistency loss into the reconstruction loss to construct a total loss function; iteratively updating the parameters of the initial 3D scene model using an adaptive density enhancement mechanism based on the total loss function; and outputting the optimized 3D scene model when a preset iteration stopping condition is met.

[0044] In this embodiment, depth distortion loss and normal uniformity loss are introduced into the reconstruction loss to construct the total loss function. The depth distortion loss constrains the Gaussian depth concentration along the camera rays, while the normal uniformity loss enforces surface geometric orientation uniformity. The specific form of the depth distortion loss is as follows:

[0045] ,

[0046] ,

[0047] in, Indicates depth distortion loss. Represents a pixel, corresponding to a single camera ray; This represents the set of pixels used for training; This represents the set of Gaussian ellipse indices that the ray hits and that contribute to the pixel; Indicates the first The depth of each contributing element on that ray; Indicates the first The composition weight of each contributing fragment to a pixel; This represents the weighted average depth of all contributing fragments on the ray, used to measure depth concentration tendency.

[0048] The specific form of normal uniformity loss is as follows:

[0049] ,

[0050] in, Represents pixels The rendering normals are obtained by rendering the current Gaussian distribution; express transpose; This represents the set of pixels used for training; This represents the geometric normals calculated from the depth map, derived through the depth map gradient. The total loss function is a weighted combination of reconstruction loss, depth distortion loss, and normal consistency loss, in the form of... ,in, and These are learnable weight coefficients used to balance the influence of various loss terms. Based on the total loss function, an adaptive density enhancement mechanism is used to iteratively update the parameters of the initial 3D scene model. This mechanism dynamically adjusts the density of the Gaussian distribution according to the loss gradient and the local complexity of the scene, supplementing the Gaussian distribution in sparse regions or merging redundant Gaussian distributions, making the model more accurately fit the scene details. When the preset iteration stopping conditions are met, such as the total loss value converging to a preset threshold or reaching the maximum number of iterations, the optimized 3D scene model is output. This process addresses the issue of standard 3DGSalpha synthesis ignoring the actual distance of the Gaussian distribution along the light direction by constraining the depth concentration of the Gaussian distribution through depth distortion loss, thus avoiding excessive depth differences among multiple semi-transparent spots in the same light direction. It also enforces local consistency of surface geometry through normal consistency loss, ensuring the restored 3D shape remains stable in both depth and direction. An adaptive density enhancement mechanism improves the model's ability to represent complex scenes. The optimized 3D scene model, based on surface elements, possesses a fixed geometry and refined appearance, enabling low-error rendering of all video frames. This provides a unified 3D coordinate system for content spanning time and significantly improves the fidelity and geometric consistency of 3D scene reconstruction, providing a more stable and reliable 3D prior for subsequent video editing. This effectively solves common problems in pure 2D editing, such as temporal discrepancies and object drift, laying a more solid foundation for high-quality text-guided video editing.

[0051] In this embodiment of the invention, the step of iteratively updating the parameters of the initial 3D scene model using an adaptive density enhancement mechanism based on the total loss function includes: updating the parameters of the initial 3D scene model in one round based on the total loss function to obtain multiple current Gaussian distributions, and determining the detail missing metric for each current Gaussian distribution; obtaining preset filtering rules and preset segmentation rules, and based on the filtering rules, selecting at least one Gaussian distribution to be segmented from all the current Gaussian distributions according to the detail missing metric; segmenting the Gaussian distribution to be segmented based on the segmentation rules to obtain a new Gaussian distribution; introducing the new Gaussian distribution into the initial 3D scene model to replace the corresponding Gaussian distribution to be segmented, thereby obtaining an updated 3D scene model; and iteratively updating the parameters of the updated 3D scene model using an adaptive density enhancement mechanism based on the total loss function.

[0052] In this embodiment, the parameters of the initial 3D scene model are updated once based on the total loss function to obtain multiple current Gaussian distributions, and the detail missing metric for each current Gaussian distribution is determined. The specific calculation expression is as follows:

[0053] ,

[0054] in, Indicates the first A detail missing metric for the current Gaussian distribution, used to measure how poorly the Gaussian distribution fits the scene details; This represents the reconstruction loss and serves as the basis for gradient calculation. Indicates the first The set of pixels currently covered by a Gaussian distribution; and Representing the reconstruction loss pairs respectively and The partial derivative of the Gaussian distribution represents the distribution of the Gaussian distribution in the pixel. The luminosity gradient at the location; summing the absolute values ​​of the gradients eliminated directional cancellation, revealing whether there is unresolved net pressure on the Gaussian distribution, with higher... A value (above the preset threshold) indicates that the Gaussian distribution spans regions containing high-frequency content and requires further segmentation.

[0055] The process involves acquiring preset filtering and segmentation rules. Based on these rules, at least one Gaussian distribution to be segmented is selected from all current Gaussian distributions according to a detail loss metric. The filtering rule specifies that if the detail loss metric exceeds a preset threshold, the current Gaussian distribution corresponding to that metric is selected as the Gaussian distribution to be segmented. This threshold is dynamically adjusted based on scene complexity and the training phase to identify Gaussian distributions that do not adequately fit scene details. These distributions often span high-frequency areas such as edges or textures, leading to overly smooth rendering. The Gaussian distribution to be segmented is then segmented according to the segmentation rules. The segmentation rule involves segmenting along the maximum covariance axis of the Gaussian distribution to be segmented, generating two new Gaussian distributions whose means are slightly offset from the plane of the original Gaussian distribution. Each new distribution inherits some color and opacity from the original distribution, and its covariance radius is approximately half that of the original. This approach maintains the overall characteristics of the original distribution while improving the representation of local details and avoiding computational redundancy caused by over-segmentation. A new Gaussian distribution is introduced into the initial 3D scene model to replace the corresponding Gaussian distribution to be segmented, resulting in an updated 3D scene model. This replacement process ensures that the model maintains the efficiency of the total Gaussian distribution while enhancing the density of speckle clouds in complex regions, thus improving the ability to reproduce details. Based on the total loss function, an adaptive density enhancement mechanism is used to iteratively update the parameters of the updated 3D scene model. The above steps are executed periodically to gradually enhance the speckle clouds in complex regions, while identifying and merging over-reconstructed regions. This ensures that the model can still reconstruct fine details while significantly reducing blur. By introducing the same-direction view space gradient segmentation criterion, the gradient collision problem in high-frequency regions of traditional 3DGS is solved, avoiding overly smooth and blurry rendering. The calculation of the detail loss metric accurately identifies the Gaussian distribution that needs optimization, while the segmentation rules ensure the effectiveness and efficiency of segmentation. The iterative update process enables the model to gradually adapt to the complex details of the scene. The optimized 3D scene model can faithfully reconstruct each frame and encode the geometric structure and appearance features of the scene in a time-consistent 3D form, providing a more stable and reliable 3D prior for the subsequent video editing stage. This effectively solves the problems of temporal disorder and object drift commonly found in pure 2D editing, while improving the fidelity and detail of 3D scene reconstruction.

[0056] In this embodiment of the invention, the appearance parameters include color parameters and opacity parameters. The step of optimizing the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model includes: constructing a reconstruction loss objective function based on the rendered image and the initial edited video frame; and optimizing the color parameters and opacity parameters of all Gaussian distributions in the frozen 3D scene model according to the reconstruction loss objective function to obtain the target 3D scene model.

[0057] In this embodiment, a reconstruction loss objective function is constructed based on the rendered image and the initial edited video frames. This function is in the form of minimizing the sum of squared L2 norms between the rendered result and the initial edited video frames, and the specific formula is as follows:

[0058] ,

[0059] in, Indicates the freezing of the 3D scene model. A Gaussian distribution of color parameters, Indicates the first The opacity parameters are Gaussian distributed, and their initial values ​​are the results obtained from the first stage of pre-training. Indicates the video frame index. This indicates the segment currently being processed, i.e., a subset of frames; Indicates fixed geometric parameters Below, use color parameters and opacity parameter The rendered result Frame image; These represent fixed geometric parameters, including the location, normals, dimensions, and blending plot parameters of all Gaussian distributions, which remain unchanged during the optimization process; This indicates that the first generation, generated by the text-driven diffusion model based on text prompts, is... Initialize editing of video frames.

[0060] The optimization process freezes all Gaussian distributions for color and opacity parameters in the 3D scene model based on the reconstruction loss objective function. During optimization, all geometric parameters and blending graph parameters are frozen, allowing only color and opacity parameters to change. Gradients flow only to color and opacity parameters, not to position or normals. This freezing operation preserves the spatial configuration of the foreground and background content discovered in the first stage, ensuring that the same object in different frames is represented by the same set of Gaussian distributions, ultimately yielding the target 3D scene model. By updating the color and opacity of the Gaussian distributions, the modification effect of the diffusion model is transferred to the 3D model, keeping the basic 3D coordinates fixed. Corresponding regions in different frames are forced to uniformly present the edited appearance, significantly improving the temporal consistency of the final video. Freezing the geometry preserves the object identity and pose determined in the reconstruction stage, effectively solving the drift and flickering problems common in unconstrained diffusion editing. The separation of geometry and appearance ensures that no temporal artifacts are introduced when editing complex dynamic scenes. The edited video fully retains all motion and depth information of the input video, with only the appearance adjusted according to text prompts.

[0061] In this embodiment of the invention, the step of preprocessing the video to be edited to obtain multiple preprocessed video segments, and inputting each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain the target edited video, includes: obtaining a preset segment division rule; dividing the video to be edited into multiple video segments based on the segment division rule; using an appearance parameter averaging algorithm to perform temporal smoothing processing on the overlapping frame regions of adjacent segments in the video segments to obtain multiple preprocessed video segments; inputting each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain multiple target edited video segments; and splicing the multiple target edited video segments in chronological order to obtain the target edited video.

[0062] In this embodiment, a preset segmentation rule is obtained. This rule divides the video to be edited into continuous short-duration segments according to the time dimension. A fixed-length overlapping frame region is set between adjacent segments to adapt to the motion characteristics and parallax distribution of dynamic videos. Based on this rule, the video to be edited is divided into multiple video segments to reduce the processing complexity of long videos and scenes with large actions. An appearance parameter averaging algorithm is used to perform temporal smoothing processing on the overlapping frame region of adjacent segments. This can be implemented using an arithmetic averaging algorithm. This algorithm averages the update results of color parameters and opacity parameters on the overlapping frames of adjacent segments, enforces consistency constraints, and obtains multiple pre-processed video segments to ensure a smooth style transition at the editing boundary and avoid abrupt changes. Each pre-processed video segment is input into the target 3D scene model for frame-by-frame rendering. The target 3D scene model is a 2DGS model with frozen geometric parameters and only optimized appearance parameters. During the rendering process, a differentiable renderer is used to generate multiple target editable video segments. The rendering results of these segments are averaged in the overlapping frame region to ensure temporal consistency. Multiple target video clips are spliced ​​together chronologically to obtain the target edited video. The spliced ​​video fully preserves the motion and depth information of the input video, while the appearance is adjusted according to text prompts, achieving both high fidelity and temporal consistency. By dividing the video into short overlapping segments in the temporal dimension, it adapts to the motion characteristics of dynamic videos, reducing the processing difficulty of long videos and scenes with large actions. The temporal smoothing processing of the appearance parameter averaging algorithm effectively eliminates style abrupt changes at the editing boundaries, significantly improving the temporal coherence of the video. Frame-by-frame rendering relies on the geometric constraints of the target 3D scene model to ensure that the edited video fully preserves the motion trajectory and depth information of the original video. Only the appearance accurately responds to the text-guided editing goals, effectively solving common problems such as object drift and temporal flicker in unconstrained diffusion editing. The final output target edited video combines visual creativity, structural fidelity, and temporal consistency, and can efficiently adapt to the text-driven editing needs of complex scenes such as long videos and scenes with large actions.

[0063] The video editing method based on a two-dimensional Gaussian function in the embodiments of the present invention has been described above. The video editing device based on a two-dimensional Gaussian function in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the video editing device based on a two-dimensional Gaussian function in this invention includes:

[0064] Data acquisition module 201: used to acquire the video to be edited, the video frame sequence corresponding to the video to be edited, prompt text, and a pre-trained diffusion model;

[0065] Model building module 202: used to generate a Gaussian distribution set based on the video frame sequence, and to build an optimized 3D scene model based on the Gaussian distribution set;

[0066] Diffusion module 203: used to input the prompt text into the diffusion model and generate the initial edited video frame;

[0067] Freeze module 204: used to freeze the geometric parameters of the optimized 3D scene model to obtain a frozen 3D scene model, and render all Gaussian distributions of the frozen 3D scene model to obtain a rendered image;

[0068] Optimization module 205: used to optimize the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model;

[0069] Editing module 206: is used to preprocess the video to be edited to obtain multiple preprocessed video segments, and input each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain the target edited video.

[0070] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0071] above Figure 2 The video editing device based on the two-dimensional Gaussian function in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The video editing device based on the two-dimensional Gaussian function in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0072] Figure 3This is a schematic diagram of a video editing device based on a two-dimensional Gaussian function according to an embodiment of the present invention. The video editing device 300 based on the two-dimensional Gaussian function can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the video editing device 300 based on the two-dimensional Gaussian function. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the video editing device 300 based on the two-dimensional Gaussian function to implement the steps of the video editing method based on the two-dimensional Gaussian function provided in the above-described method embodiments.

[0073] The video editing device 300 based on a two-dimensional Gaussian function may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of a video editing device based on a two-dimensional Gaussian function does not constitute a limitation on a video editing device based on a two-dimensional Gaussian function. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0074] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a video editing method based on a two-dimensional Gaussian function.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video editing method based on a two-dimensional Gaussian function, characterized in that, include: The process involves acquiring the video to be edited, the video frame sequence corresponding to the video to be edited, the prompt text, and the pre-trained diffusion model, wherein the video frame sequence includes multiple video frames. An automatic video object segmentation network is used to distinguish the foreground and background regions of each video frame. A mask cropping variant is used to sparsely reconstruct the foreground and background regions of each video frame to obtain foreground point clouds and background point clouds. The loose boundary volume is calculated based on the foreground point cloud and the background point cloud, and multiple sets of supplementary background points are generated based on the loose boundary volume; The multiple sets of supplementary background points are merged with the background point cloud to obtain the target background point cloud; A foreground Gaussian distribution subset is generated based on the foreground point cloud, and a background Gaussian distribution subset is generated based on the target background point cloud. The foreground and background Gaussian distribution subsets are integrated to obtain a Gaussian distribution set, where each Gaussian distribution in the set is defined as an oriented planar disk. A foreground two-dimensional Gaussian scattering set is constructed based on the foreground Gaussian distribution subset, and a background two-dimensional Gaussian scattering set is constructed based on the background Gaussian distribution subset. The foreground and background two-dimensional Gaussian scattering sets are integrated to obtain an initial 3D scene model. A differentiable renderer is used to perform differentiable rendering on the foreground and background two-dimensional Gaussian scattering sets respectively to obtain a foreground rendered image and a background rendered image. A learnable blending map is obtained, and an overall rendered image is generated based on the learnable blending map, the foreground rendered image, and the background rendered image. Real video frames corresponding to the overall rendered image are obtained, a reconstruction loss is calculated based on the overall rendered image and the real video frames, and the parameters of the initial 3D scene model are optimized based on the reconstruction loss to obtain an optimized 3D scene model. The prompt text is input into the diffusion model to generate the initial edited video frame; Freeze the geometric parameters of the optimized 3D scene model to obtain a frozen 3D scene model, and render all Gaussian distributions of the frozen 3D scene model to obtain a rendered image; Based on the rendered image and the initial edited video frame, the appearance parameters of the frozen 3D scene model are optimized to obtain the target 3D scene model; The video to be edited is preprocessed to obtain multiple preprocessed video segments, and each preprocessed video segment is input into the target 3D scene model for frame-by-frame rendering to obtain the target edited video.

2. The video editing method based on a two-dimensional Gaussian function according to claim 1, characterized in that, The process of optimizing the parameters of the initial 3D scene model based on the reconstruction loss to obtain the optimized 3D scene model includes: Depth distortion loss and normal consistency loss are introduced into the reconstruction loss to construct the total loss function; Based on the total loss function, an adaptive density enhancement mechanism is used to iteratively update the parameters of the initial 3D scene model; When the preset iteration stopping condition is met, the optimized 3D scene model is output.

3. The video editing method based on a two-dimensional Gaussian function according to claim 2, characterized in that, The step of iteratively updating the parameters of the initial 3D scene model using an adaptive density enhancement mechanism based on the total loss function includes: The parameters of the initial 3D scene model are updated once based on the total loss function to obtain multiple current Gaussian distributions, and the detail missing metric of each current Gaussian distribution is determined. Obtain preset filtering rules and preset segmentation rules, and based on the filtering rules, select at least one Gaussian distribution to be segmented from all the current Gaussian distributions according to the detail missing metric. The Gaussian distribution to be segmented is segmented based on the segmentation rule to obtain a new Gaussian distribution; The new Gaussian distribution is introduced into the initial 3D scene model to replace the corresponding Gaussian distribution to be segmented, thereby obtaining an updated 3D scene model. Based on the total loss function, an adaptive density enhancement mechanism is used to iteratively update the parameters of the updated 3D scene model.

4. The video editing method based on a two-dimensional Gaussian function according to claim 1, characterized in that, The appearance parameters include color parameters and opacity parameters. The process of optimizing the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model includes: Construct a reconstruction loss objective function based on the rendered image and the initial edited video frame; The target 3D scene model is obtained by optimizing the color parameters and opacity parameters of all Gaussian distributions in the frozen 3D scene model according to the reconstruction loss objective function.

5. The video editing method based on a two-dimensional Gaussian function according to claim 1, characterized in that, The process involves preprocessing the video to be edited to obtain multiple preprocessed video segments, and then inputting each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain the target edited video, including: Obtain preset segment division rules, and divide the video to be edited into multiple video segments based on the segment division rules; An appearance parameter averaging algorithm is used to perform temporal smoothing on the overlapping frame regions of adjacent segments in the video segment, resulting in multiple preprocessed video segments. Each preprocessed video segment is input into the target 3D scene model for frame-by-frame rendering to obtain multiple target edited video segments. The multiple target edited video segments are then spliced ​​together in chronological order to obtain the target edited video.

6. A video editing device based on a two-dimensional Gaussian function, characterized in that, include: Data acquisition module: used to acquire the video to be edited, the video frame sequence corresponding to the video to be edited, prompt text and pre-trained diffusion model, wherein the video frame sequence includes multiple video frames; Model building module: used to distinguish the foreground and background regions of each video frame using an automatic video object segmentation network; A mask cropping variant is used to sparsely reconstruct the foreground and background regions of each video frame to obtain foreground point clouds and background point clouds. The loose boundary volume is calculated based on the foreground point cloud and the background point cloud, and multiple sets of supplementary background points are generated based on the loose boundary volume; The multiple sets of supplementary background points are merged with the background point cloud to obtain the target background point cloud; A foreground Gaussian distribution subset is generated based on the foreground point cloud, and a background Gaussian distribution subset is generated based on the target background point cloud. The foreground and background Gaussian distribution subsets are integrated to obtain a Gaussian distribution set, where each Gaussian distribution in the set is defined as an oriented planar disk. A foreground two-dimensional Gaussian scattering set is constructed based on the foreground Gaussian distribution subset, and a background two-dimensional Gaussian scattering set is constructed based on the background Gaussian distribution subset. The foreground and background two-dimensional Gaussian scattering sets are integrated to obtain an initial 3D scene model. A differentiable renderer is used to perform differentiable rendering on the foreground and background two-dimensional Gaussian scattering sets respectively to obtain a foreground rendered image and a background rendered image. A learnable blending map is obtained, and an overall rendered image is generated based on the learnable blending map, the foreground rendered image, and the background rendered image. Real video frames corresponding to the overall rendered image are obtained, a reconstruction loss is calculated based on the overall rendered image and the real video frames, and the parameters of the initial 3D scene model are optimized based on the reconstruction loss to obtain an optimized 3D scene model. Diffusion module: used to input the prompt text into the diffusion model and generate the initial edited video frame; Freeze module: used to freeze the geometric parameters of the optimized 3D scene model to obtain a frozen 3D scene model, and to render all Gaussian distributions of the frozen 3D scene model to obtain a rendered image; Optimization module: used to optimize the appearance parameters of the frozen 3D scene model based on the rendered image and the initial edited video frame to obtain the target 3D scene model; Editing module: Used to preprocess the video to be edited to obtain multiple preprocessed video segments, and input each preprocessed video segment into the target 3D scene model for frame-by-frame rendering to obtain the target edited video.

7. A video editing device based on a two-dimensional Gaussian function, characterized in that, The video editing device based on the two-dimensional Gaussian function includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the video editing device based on the two-dimensional Gaussian function to perform the steps of the video editing method based on the two-dimensional Gaussian function as described in any one of claims 1-5.

8. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the video editing method based on a two-dimensional Gaussian function as described in any one of claims 1-5.

Images