Video editing methods, apparatus, electronic devices, and storage media

Abstract

This application relates to the field of computer vision technology, and provides a video editing method, apparatus, electronic device, and storage medium. The method includes: performing a first inverse denoising process on a first original noise latent variable at the current time step based on original video prompts to obtain a second original noise latent variable, and using the second original noise latent variable as the first original noise latent variable for the next time step; performing a second inverse denoising process on a first target noise latent variable at the current time step based on target video prompts to obtain a second target noise latent variable; and obtaining a third target noise latent variable based on the background data region of the second original noise latent variable and the second target noise latent variable, and using the third target noise latent variable as the first target noise latent variable for the next time step. The video editing method, apparatus, electronic device, and storage medium provided by this application can perform video editing tasks efficiently and at low cost.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video editing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, video editing, as the process of editing and adding special effects to raw video footage to output high-quality video content, is widely used in film and television production, advertising, and many other fields. Its implementation methods are mainly divided into two categories: manual editing based on multimedia-assisted software and automated editing based on computer vision algorithms. Traditional manual editing relies on professionals using specialized software, heavily depending on the professional skills and experience of the personnel, resulting in high labor and time costs, making it difficult to meet large-scale editing needs, and the editing effect is easily affected by individual differences. Automated editing, on the other hand, relies on computer vision algorithms, which can automatically analyze various types of data resources and complete editing steps, reducing labor and time costs, simplifying the operation process, and providing pre-processed materials for manual editing.

[0003] As an emerging technology, diffusion models recover high-quality content by progressively denoising image or video data containing Gaussian noise. Taking the representative VACE video editing framework as an example, it is based on a diffusion model, using video, text, and user-input control signals (such as masks) as conditional inputs, and achieving controllable modification of video segments through context modeling. Its process includes three parts: context adaptation and optimization, video context understanding, and context embedding, achieving certain results in video editing. However, existing video editing frameworks based on diffusion-based video generation models still have the following problems: high dependence on manual user input, requiring frame-by-frame mask annotation, resulting in high labor costs for long videos or videos with frequent target movement; strong dependence on user text descriptions, requiring detailed textual definitions of modification targets, making it difficult for users to accurately convey their needs in complex scenarios, and inaccurate or insufficient text can lead to significant discrepancies between the generated results and expectations; reliance on training with the original generation model, unable to complete independently, with reduced editing performance when the pre-trained model differs from the target task, and high computational and storage requirements for training and inference, limiting its application scope.

[0004] Therefore, how to perform video editing tasks efficiently and at low cost is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a video editing method, apparatus, electronic device, and storage medium to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a video editing method, including: Obtain the initial noise latent variables for each video frame in the video to be edited; for each video frame, iteratively perform the following operations to obtain the target video: based on the original video prompts, perform a first inverse denoising process on the first original noise latent variable at the current time step to obtain a second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step; based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain a second target noise latent variable; based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain a third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; wherein, the first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0007] In one embodiment, the method further includes: for each video frame in the video to be edited, using the inversion process of the target diffusion video generation model, mapping the video frame to the latent space processed by the target diffusion video generation model, to obtain the initial noise latent variable of each video frame.

[0008] In one embodiment, obtaining a third target noise latent variable based on the background data region of the second original noise latent variable and the second target noise latent variable includes: replacing the background data region of the second target noise latent variable with the background data region of the second original noise latent variable corresponding to the current time step to obtain the third target noise latent variable; wherein the background data region of the second target noise latent variable corresponds to the non-editing region of the video to be edited.

[0009] In one embodiment, obtaining the third target noise latent variable based on the background data region of the second original noise latent variable and the second target noise latent variable includes: assigning a first weight to each data unit of the background data region of the second original noise latent variable; assigning a second weight to each data unit of the background data region of the second target noise latent variable; performing a weighted summation operation on each data unit of the background data region of the second original noise latent variable and each data unit of the background data region of the second target noise latent variable based on the first weight and the second weight to obtain the data of the background data region in the third target noise latent variable; and using the non-background data region of the second target noise latent variable as the non-background data region of the third target noise latent variable.

[0010] In one embodiment, the third target noise latent variable includes a background data sub-region, a transition data sub-region, and a non-background data sub-region; wherein the non-background data sub-region corresponds to the editing area of ​​the video to be edited; the background data sub-region and the transition data sub-region together correspond to the non-editing area of ​​the video to be edited, and the transition data sub-region is located between the background data sub-region and the non-background data sub-region to achieve a smooth transition of data features between the background data sub-region and the non-background data sub-region.

[0011] In one embodiment, obtaining the third target noise latent variable based on the background data region of the second original noise latent variable and the second target noise latent variable includes: fusing the first data region and the second data region to obtain the data of the transition data sub-region; wherein the first data region is the data region in the second original noise latent variable corresponding to the transition data sub-region; the second data region is the data region in the second target noise latent variable corresponding to the transition data sub-region; the third data region is used as the background data sub-region; and the fourth data region is used as the non-background data sub-region; wherein the third data region is the data region in the second original noise latent variable corresponding to the background data sub-region; and the fourth data region is the data region in the second target noise latent variable corresponding to the non-background data sub-region.

[0012] In one embodiment, fusing the first data region and the second data region to obtain the data of the transition data sub-region includes: determining the background weight of each data unit in the first data region; determining the generation weight of each data unit in the second data region; and performing a weighted summation operation on each data unit in the first data region and the second data region according to the background weight and the generation weight to obtain the data of the transition data sub-region.

[0013] In one embodiment, determining the background weight of each data unit in the first data region includes: assigning a first background sub-weight to each data unit in the first data region; wherein the first background sub-weight is associated with the time step of the second inverse denoising process, and the first background sub-weight corresponding to each time step gradually decreases from the initial time step; and determining the background weight of each data unit in the first data region based on the first background sub-weight of each data unit in the first data region.

[0014] In one embodiment, determining the background weight of each data unit in the first data region based on the first background sub-weight of each data unit in the first data region includes: assigning a second background sub-weight to each data unit in the first data region according to the minimum distance between the data unit and the non-background data region of the second original noise latent variable; wherein the non-background data region of the second original noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a larger second background sub-weight; and determining the background weight of each data unit in the first data region according to the first background sub-weight and the second background sub-weight of each data unit in the first data region.

[0015] In one embodiment, determining the background weight of each data unit in the first data region based on the first background sub-weight and the second background sub-weight of each data unit in the first data region includes: taking the sum of the first background sub-weight and the second background sub-weight of the data unit as the background weight of each data unit in the first data region.

[0016] In one embodiment, determining the generation weight of each data unit in the second data region includes: assigning a first generation sub-weight to each data unit in the second data region; wherein the first generation sub-weight is associated with the time step of the second inverse denoising process, and the first generation sub-weight corresponding to each time step gradually increases from the initial time step; and determining the generation weight of each data unit in the second data region based on the first generation sub-weight of each data unit in the second data region.

[0017] In one embodiment, determining the generation weight of each data unit in the second data region based on the first generation sub-weight of each data unit in the second data region includes: for each data unit in the second data region, assigning a second generation sub-weight to the data unit according to the minimum distance between the data unit and the non-background data region of the second target noise latent variable; wherein, the non-background data region of the second target noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a smaller second generation sub-weight; and determining the generation weight of each data unit in the second data region according to the first generation sub-weight and the second generation sub-weight of each data unit in the second data region.

[0018] In one embodiment, determining the generation weight of each data unit in the second data region based on the first generation sub-weight and the second generation sub-weight of each data unit in the second data region includes: taking the sum of the first generation sub-weight and the second generation sub-weight of each data unit in the second data region as the generation weight of each data unit in the second data region.

[0019] In one embodiment, the method further includes: Using a visual language model, the attribute features of the video to be edited are extracted to obtain the original video prompt words.

[0020] In one embodiment, the method further includes: Obtain editing instructions; using a large language model, determine the attribute features of the target video based on the original video prompts and the editing instructions, and obtain the target video prompts.

[0021] In one embodiment, the method further includes: Using a large language model, target object prompts are determined based on the original video prompts and the editing instructions; wherein, the target object prompts are used to identify the object to be edited in the video to be edited.

[0022] In one embodiment, the method further includes: Using a large language model, the original video prompts, the target video prompts, and the target object prompts are validated; if the confidence level of any one of the original video prompts, the target video prompts, and the target object prompts is determined to be lower than a preset threshold, it is marked as a low-confidence clause.

[0023] In one embodiment, the method further includes: using a target video tracking algorithm to determine the editable and non-editable regions of each image frame in the video to be edited based on the target object prompt.

[0024] In one embodiment, determining the editable and non-editable regions of each image frame in the video to be edited based on the target object prompt word using a target video tracking algorithm includes: determining the mask of each image frame in the video to be edited based on the target object prompt word using a target video tracking algorithm; and determining the editable and non-editable regions of each image frame in the video to be edited based on the mask of each image frame in the video to be edited.

[0025] In one embodiment, determining the mask of each image frame in the video to be edited based on the target object prompt word using the target video tracking algorithm includes: dividing the video to be edited into multiple video segments; wherein each video segment contains at least one video frame, and there are overlapping video frames between adjacent video segments; for each video segment, determining the mask of each image frame in the video segment based on the target object prompt word using the target video tracking algorithm; and obtaining the mask of each image frame in the video to be edited based on the mask of each image frame in each video segment.

[0026] In one embodiment, obtaining the mask of each image frame in the video to be edited based on the mask of each image frame in each video segment includes: successively determining the intersection-union ratio (IUR) of the masks of overlapping video frames between adjacent video segments; if all IURs are determined to be greater than a preset threshold; fusing the masks of each video frame in the adjacent video segments to obtain the mask of each image frame in the video to be edited.

[0027] Secondly, embodiments of this application provide a video editing apparatus, including: The acquisition module is used to: acquire the initial noise latent variables of each video frame in the video to be edited; the generation module is used to: iteratively perform the following operations for each video frame to obtain the target video: based on the original video prompts, perform a first inverse denoising process on the first original noise latent variable at the current time step to obtain a second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step; based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain a second target noise latent variable; based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain a third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; wherein, the first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0028] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the video editing method described in the first aspect.

[0029] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the video editing method described in the first aspect.

[0030] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the video editing method described in the first aspect.

[0031] The video editing method, apparatus, electronic device, and storage medium provided in this application embodiment perform a first reverse denoising process and a second reverse denoising process based on the original video prompts and the target video prompts, respectively. A third target noise latent variable is obtained by combining the background data region of the second original noise latent variable generated in the first reverse denoising process (corresponding to the non-editing area of ​​the video to be edited) with the second target noise latent variable generated in the second reverse denoising process. This process is used to progressively obtain the target video. Therefore, it can effectively combine the original information of the video to be edited with the target editing requirements, accurately modifying the editing area while preserving the accuracy of the content in the non-editing area, and efficiently and cost-effectively generating a high-quality target video that meets expectations. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the video editing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the method for determining the latent noise variables of a third target provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the method for determining the editable and non-editable regions of each image frame in a video to be edited, as provided in an embodiment of this application. Figure 4 This is a schematic diagram of the video editing system provided in an embodiment of this application; Figure 5 This is the video editing device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] Figure 1 This is a flowchart illustrating the video editing method provided in the embodiments of this application.

[0036] Reference Figure 1 This application provides a video editing method, which may include: Step 101: Obtain the initial noise latent variables of each video frame in the video to be edited.

[0037] The initial noise latent variable is a representation of a video frame in the latent space processed by the target diffusion video generation model, which contains the original noise characteristics of the video frame.

[0038] In the specific implementation process, for each video frame in the video to be edited, the inversion process of the target diffusion video generation model can be used to map the video frame to the latent space processed by the target diffusion video generation model, so as to obtain the initial noise latent variables of each video frame.

[0039] Targeted diffusion video generation models are generative models based on a probabilistic diffusion process. By progressively adding noise to the data and learning a reverse denoising process, they generate high-quality video frames from latent noise variables. By controlling the addition and removal of noise, Targeted Diffusion Video Generation Models generate temporally coherent and semantically sound video content, suitable for tasks such as video compositing and editing. Targeted diffusion video generation models can include, but are not limited to, DDIM (Denoising Diffusion Implicit Models) and VDM (Video Diffusion Models).

[0040] The latent space is an important concept in the diffusion video generation model. It is a low-dimensional space used to represent data within the diffusion video generation model. The representation of data in the latent space (i.e., noise latent variables) contains the essential characteristics of the data.

[0041] In some embodiments, the target diffusion video generation model is DDIM, an improved variant of the diffusion model that significantly improves generation efficiency by implicitly defining the inversion process (without explicitly sampling Markov chains). For example, in video generation tasks, DDIM can start from latent random noise variables and quickly generate frame sequences consistent with the style of the original video with a small number of denoising steps, while maintaining inter-frame motion coherence. Compared to traditional diffusion models, DDIM is more flexible in the inference phase, allowing for control of the balance between generation quality and speed by adjusting the step size.

[0042] The inversion process of the target diffusion video generation model is essentially a reverse diffusion process, which is the process of deriving the corresponding initial noise (or latent variables) from the generated samples (such as images or videos).

[0043] After obtaining the initial noise latent variables for each video frame in the video to be edited, the following operations are iteratively performed for each video frame until a preset number of steps are reached to obtain the target video: Step 102: Based on the original video prompts, perform the first inverse denoising process on the first original noise latent variable at the current time step to obtain the second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step.

[0044] The original video prompts are used to describe the characteristic attributes of each video frame in the video to be edited (such as current scene and environment information, actions, objects, etc.).

[0045] In the specific implementation process, based on the original video prompts, the first inverse denoising process is performed on the first original noise latent variable at the current time step; after removing some noise, a clearer second original noise latent variable is obtained; and this second original noise latent variable is set as the first original noise latent variable for the next time step.

[0046] The first original noise latent variable is a noise latent variable used as initial input to perform the first inverse denoising process. It represents the basic noise characteristics of the video frame at the current time step in the latent space. Specifically, the first original noise latent variable of the initial time step is initialized with the initial noise latent variable.

[0047] Step 103: Based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain the second target noise latent variable.

[0048] Target video cue words are used to describe the characteristic attributes of each video frame in the target video.

[0049] The first target noise latent variable corresponds to the first original noise latent variable and is used as the initial input for performing the second inverse denoising process at the same time step in the generation of the target video. Specifically, the first target noise latent variable at the initial time step is initialized with the initial noise latent variable.

[0050] In the specific implementation process, based on the target video prompts, a second inverse denoising process is performed on the first target noise latent variable at the current time step; after removing some noise, a clearer second target noise latent variable is obtained.

[0051] Step 104: Based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain the third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step.

[0052] Background data region refers to the noise latent variable region corresponding to the part of a video frame that does not need to be edited or modified. For example, when editing a video of a person, if the background is a static landscape, the noise latent variable region corresponding to this landscape is the background data region.

[0053] The background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited. The non-background data region of the second original noise latent variable corresponds to the editing region of the video to be edited.

[0054] In practice, the background data region and non-background data region of the second original noise latent variable can be separated using the masks of each video frame in the video to be edited. For details on how to obtain the masks of each video frame in the video to be edited, please refer to [link to relevant documentation]. Figure 3 The relevant content will not be repeated here.

[0055] In practice, the third target noise latent variable can be obtained in various ways based on the background data area of ​​the second original noise latent variable and the second target noise latent variable, without being limited by the description in this specification.

[0056] In some embodiments, the background data region of the second target noise latent variable can be replaced with the background data region of the second original noise latent variable corresponding to the current time step to obtain the third target noise latent variable.

[0057] The background data region of the second target noise latent variable corresponds to the non-editing region of the video to be edited. The non-background data region of the second target noise latent variable corresponds to the editing region of the video to be edited. In practice, the background data region and non-background data region of the second target noise latent variable can be separated using the mask of each video frame in the video to be edited.

[0058] In the specific implementation process, after identifying the background data region in the second target noise latent variable that corresponds to the non-editing area of ​​the video to be edited, the background data region at the same position in the second original noise latent variable is used to replace it, thus forming a new third target noise latent variable.

[0059] In some embodiments, a first weight may be assigned to each data unit of the background data region of the second original noise latent variable; a second weight may be assigned to each data unit of the background data region of the second target noise latent variable; based on the first weight and the second weight, a weighted summation operation may be performed on each data unit of the background data region of the second original noise latent variable and each data unit of the background data region of the second target noise latent variable to obtain the data of the background data region in the third target noise latent variable; and the non-background data region of the second target noise latent variable may be used as the non-background data region of the third target noise latent variable.

[0060] In the specific implementation process, according to the editing requirements and algorithm settings, corresponding weights can be assigned to the background data regions of the second original noise latent variable and the second target noise latent variable; a weighted summation operation can be performed on each data unit of these two regions to generate the background data region of the third target noise latent variable; at the same time, the non-background data region of the second target noise latent variable can be retained as the non-background part of the third target noise latent variable.

[0061] In the specific implementation process, it can be done by, for example Figure 4 The video generation mode adaptation module shown above, which requires no modification, performs the same steps.

[0062] In the embodiments provided in this application, a process of "target-diffusion video generation model inversion—dual-path denoising—boundary-consistent synthesis" is adopted to adapt and connect to existing diffusion video generation models with zero modifications (no change to structure, weights, or fine-tuning), thus efficiently and accurately performing video editing tasks. First, through the inversion process of the target-diffusion video generation model, the latent space starting point consistent with each video frame of the video to be edited is obtained: the initial noise latent variable.

[0063] The initial noise latent variables are used as common initial noise latent variables for both the reconstruction path (first inverse denoising process) and the generation / edit path (second inverse denoising process). This allows the two paths to share the same time axis and alignment rules, ensuring that they can be synthesized in the same coordinate system subsequently.

[0064] Two parallel denoising paths are employed: the reconstruction path injects only the original video cues during iteration, faithfully restoring the color, lighting, and texture of the source image; the generation / editing path injects the target video cues during iteration, enabling local semantic rewriting and appearance updates. In each denoising time step: the background data region represented by the intermediate noise variable of the reconstruction path (the second original noise latent variable) is injected as a "background reference" into the second target noise latent variable, resulting in the third target noise latent variable. Based on the third target noise latent variable, the generation / editing path can effectively edit the editing area while simultaneously utilizing the background information provided by the reconstruction path to precisely limit the scope of changes made by the generation / editing path and stabilize the unedited area. This effectively prevents issues such as screen tearing and discontinuity that may occur during editing, improving the quality and visual effects of video editing.

[0065] In some embodiments, to avoid the discontinuity in tone and harsh outline caused by the fusion of the background data area of ​​the second original noise latent variable and the second target noise latent variable, such as Figure 4 As shown, a soft-edge latent variable blending (SELB) mechanism can be introduced. Specifically, the third target noise latent variable includes a background data sub-region, a transition data sub-region, and a non-background data sub-region; wherein, the non-background data sub-region corresponds to the editing area of ​​the video to be edited; the background data sub-region and the transition data sub-region together correspond to the non-editing area of ​​the video to be edited, and the transition data sub-region is located between the background data sub-region and the non-background data sub-region, so as to achieve a smooth transition of data features between the background data sub-region and the non-background data sub-region. For a detailed description of how the third target noise latent variable is obtained based on the background data region of the second original noise latent variable and the second target noise latent variable in this embodiment, please refer to the following. Figure 2 The relevant content will not be repeated here.

[0066] Figure 2 This is a flowchart illustrating the method for determining the latent noise variables of a third target provided in an embodiment of this application. (Refer to...) Figure 2 This application provides a method for determining a third target noise latent variable, which may include: Step 201: Merge the first data region and the second data region to obtain the data of the transition data sub-region.

[0067] The first data region is the data region corresponding to the transition data sub-region in the second original noise latent variable; the second data region is the data region corresponding to the transition data sub-region in the second target noise latent variable.

[0068] In some embodiments, the background weight of each data unit in the first data region can be determined; the generation weight of each data unit in the second data region can be determined; and a weighted summation operation can be performed on each data unit in the first data region and the second data region based on the background weight and the generation weight to obtain the data of the transition data sub-region.

[0069] Background weights are used to reflect the degree to which data units contribute to the features of the original image.

[0070] Generative weights are used to reflect the degree to which data units contribute to the generation of new image features.

[0071] In the specific implementation process, the background weight of each data unit in the first data area can be determined in the following ways.

[0072] Assign a first background sub-weight to each data unit in the first data region.

[0073] The first background sub-weight is associated with the time step of the second inverse denoising process. Starting from the initial time step, the first background sub-weight corresponding to each time step gradually decreases.

[0074] The background weight of each data unit in the first data region is determined based on the first background sub-weight of each data unit in the first data region.

[0075] In some embodiments, the first background sub-weight of each data unit in the first data region can be used as the background weight of each data unit in the first data region.

[0076] In some embodiments, a second background sub-weight may be assigned to each data cell in the first data region based on the minimum distance between the data cell and the non-background data region of the second original noise latent variable.

[0077] The non-background data region of the second original noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a larger second background sub-weight; the background weight of each data unit in the first data region is determined according to the first background sub-weight and the second background sub-weight of each data unit in the first data region.

[0078] In the specific implementation process, the sum of the first background sub-weight and the second background sub-weight of the data unit can be used as the background weight of each data unit in the first data region.

[0079] In the specific implementation process, the generation weight of each data unit in the second data area can be determined in the following ways.

[0080] Assign the first generated sub-weight to each data unit in the second data region.

[0081] The first generator weight is associated with the time step of the second inverse denoising process. Starting from the initial time step, the first generator weight gradually increases for each time step.

[0082] The generation weight of each data unit in the second data region is determined based on the first generation sub-weight of each data unit in the second data region.

[0083] In some embodiments, the first generation sub-weight of each data unit in the second data region can be used as the generation weight of each data unit in the second data region.

[0084] In some embodiments, for each data unit in the second data region, a second generation sub-weight can be assigned to the data unit based on the minimum distance between the data unit and the non-background data region of the second target noise latent variable; wherein, the non-background data region of the second target noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a smaller second generation sub-weight; the generation weight of each data unit in the second data region is determined based on the first generation sub-weight and the second generation sub-weight of each data unit in the second data region.

[0085] In the specific implementation process, the sum of the first generation sub-weight and the second generation sub-weight of each data unit in the second data region can be used as the generation weight of each data unit in the second data region.

[0086] Step 202: Use the third data area as the background data sub-region.

[0087] The third data region is the data region in the second original noise latent variable that corresponds to the background data sub-region.

[0088] Step 203: Treat the fourth data region as a non-background data sub-region.

[0089] The fourth data region is the data region corresponding to the non-background data sub-region in the second target noise latent variable.

[0090] In the embodiments provided in this application, a weighted summation operation is performed on each data unit in the first and second data regions based on background weights and generation weights to obtain data in the transition data sub-region. This achieves pixel-level progressive mixing of intermediate noise latent variables in the reconstruction path and the generation / editing path: closer to the background data region and in earlier time steps, the focus shifts more towards the reconstruction path (stabilizing global color / lighting); further away from the background data region and in later time steps, the focus gradually shifts towards the generation / editing path (releasing details and semantic rewriting), ensuring smooth transition of high and low frequency information in the background data region and non-background data region, suppressing the "patchwork" effect. Under the dual effect of the noise intermediate amount constraint provided by the reconstruction path and SELB soft boundary fusion, the generation / editing path completes controlled local editing and blends naturally with the background, thus balancing controllability and temporal / visual consistency without altering the base model.

[0091] Figure 3 This is a flowchart illustrating the method for determining the editable and non-editable regions of each image frame in a video to be edited, as provided in an embodiment of this application.

[0092] Reference Figure 3 This application provides a method for determining the editable and non-editable regions of each image frame in a video to be edited, which may include: Step 301: Using a visual language model, extract the attribute features of the video to be edited to obtain the original video prompts.

[0093] In the specific implementation process, the editing instructions, including the video to be edited and a brief description in natural language, can be obtained; basic information (duration, frame rate, resolution, etc.) of the video to be edited can be parsed, and the parsed information can be used as a shared context for subsequent processing.

[0094] Editing instructions are text commands entered by the user to guide video editing operations, such as "replace cherries with strawberries".

[0095] In the specific implementation process, preset prompts can be used to guide the visual language model to perform a two-level semantic scan and structured induction of the video to be edited: the global level covers the subject category, scene elements, background composition, lighting and tone, and identifies the shot type (such as close-up, long shot, push-pull, pan, tilt, etc.) and overall style (such as realistic, cyberpunk, traditional Chinese style, film graininess) and atmosphere (such as warm, cold, tense); the local level supplements fine-grained attributes in the neighborhood of keyframes, such as the positional relationship of objects, local materials and textures, visible action stages, and physical contact relationships with the environment (handheld / occlusion / projection / reflection), etc. After redundancy suppression and consistency constraints, the two-level results form the original video prompts that can cover significant concepts while avoiding noise and off-topic details, serving as the contextual prior and semantic anchor points for subsequent target diffusion video generation models and mask retrieval.

[0096] Step 302: Using a large language model, determine the attribute features of the target video based on the original video prompts and editing instructions to obtain the target video prompts.

[0097] Target video cue words are textual information that describes the attributes of the edited video and can reflect the desired editing effect for the video to be edited.

[0098] Step 303: Using a large language model, determine the target object prompts based on the original video prompts and editing instructions; whereby the target object prompts are used to identify the object to be edited in the video to be edited.

[0099] In the specific implementation process, the large language model can be guided to align and map "current screen - user intent - editable slot (object, attribute, time period)" according to the original video prompts and editing instructions, combined with the preset prompt templates, so as to accurately output the target video prompts.

[0100] If the editing command is a local edit (such as adding or deleting a certain type of object, replacing a part of the appearance, or changing a single attribute), the large language model will also explicitly provide target object prompts so that the video tracking algorithm can be used to perform text retrieval and instance segmentation.

[0101] In some embodiments, to improve the accuracy of the generated target video cues, Chain-of-Thought (CoT) reasoning can be introduced to break down the generation process of target video cues and target object cues into three verifiable steps: First, locate and rewrite the target object and editing type in the editing instructions to ensure that the editing action is consistent with the object category / attribute; second, evaluate the collateral impact of the editing on the environment and relationships, such as changes in reflections and mirror images, updates to contact / occlusion relationships, lighting consistency, and scale perspective rationality; third, generate target video cues that conform to the physical and semantic logic of the scene after integrating the constraints of the first two steps, so that they satisfy the user's intent without disrupting the spatiotemporal coherence and stylistic consistency of the original image.

[0102] In some embodiments, a small number of high-quality, manually designed examples can also be input into the large language model as prompt words to stabilize the output style and field structure of the large language model and reduce ambiguity and omissions.

[0103] In some embodiments, to prevent the visual language model and the large language model from being unstable and affecting the quality of the output results, the large language model can also be used to verify the original video prompt words, the target video prompt words, and the target object prompt words; if it is determined that the confidence level of any one of the original video prompt words, the target video prompt words, and the target object prompt words is lower than a preset threshold, it is marked as a low confidence clause.

[0104] In the specific implementation process, a large language model can be used to perform a two-dimensional comparison and verification of the correctness and rationality of the original video prompts, target video prompts, and target object prompts. On the one hand, it verifies whether the editing instructions are strictly met, confirming that the target object specified in the editing instructions has been accurately covered, the editing type and intensity are consistent with the editing instructions, and that the semantic description of non-target objects has not been mistakenly modified. On the other hand, it performs a consistency check on the impact of the target object on the environment, such as whether the descriptions of potentially affected shadows, reflections, occlusion relationships, relative positions, and narrative coherence have been updated synchronously after replacing / adding / deleting objects. If necessary, it automatically proposes fine-tuning suggestions and writes back the revised target video prompts.

[0105] Through the above verification mechanism, a conservative strategy can be adopted while ensuring fluency: when the large language model detects uncertainty or potential conflict, it prioritizes maintaining the basic consistency between the target video prompt and the original video prompt, and marks low-confidence clauses so that when the subsequent video tracking algorithm determines the mask in each video frame of the video to be edited, it can reduce the intensity or trigger conservative synthesis.

[0106] In the embodiments provided in this application, a visual language model is used to extract attribute features of the video to be edited to obtain original video prompts; editing instructions are obtained; and a large language model is used to determine the attribute features of the target video based on the original video prompts and editing instructions to obtain target video prompts. By combining the visual language model and the large language model, users only need to input concise instructions (such as "replace the dog in the video with a cat" or "change the shirt of the man on the left to blue"), without writing long paragraphs or listing detailed attributes. This automatically generates original video prompts that match the video to be edited and target video prompts that are consistent with the modified target video. Thus, the user's natural language intent is automatically translated into executable editing semantics, which can be directly adapted to the input space of the subsequent target diffusion video generation model without any structural modifications or additional training to the original diffusion video generation model.

[0107] like Figure 4As shown, steps 301 to 303 above can be executed by the user instruction analysis module. The above method is implemented through the user instruction analysis module. The user instruction analysis module is completed collaboratively by three functional units: a Video Understanding Unit (VUU), used to execute step 301; a Target Video Prompt Generation Unit (TPGU), used to execute steps 302 and 303; and a Prompt Check and Rectification Unit (PCRU), used to verify the original video prompts, target video prompts, and target object prompts using a large language model.

[0108] Step 304: Using a target video tracking algorithm, determine the editing and non-editing regions of each image frame in the video to be edited based on the target object prompts.

[0109] In some embodiments, a target video tracking algorithm can be used to determine the mask of each image frame in the video to be edited based on the target object prompt words; and based on the mask of each image frame in the video to be edited, the editing area and non-editing area of ​​each image frame in the video to be edited can be determined.

[0110] Video tracking algorithms can continuously locate and track the motion trajectory of a specific target object in consecutive video frames.

[0111] In practice, various video tracking algorithms can be used to obtain the mask of each video frame in the video to be edited based on the target object prompts, without being limited by the description in this manual.

[0112] Taking Grounding-SAM (Grounding DINO-Segment Anything Model, a semantic segmentation-based figurative object detection and segmentation model) as an example, Grounding-SAM is an advanced model that integrates Grounding DINO and Segment Anything Model (SAM), primarily applied to object detection and segmentation tasks in images and videos. Grounding DINO is an object detection model that can accurately locate and identify corresponding target objects in images based on given object cues, such as words like "cat" or "car." Segment Anything Model (SAM) is an image segmentation model that can perform detailed segmentation of objects in an image based on their location information (e.g., bounding boxes), generating pixel-level masks.

[0113] Grounding-SAM combines the advantages of Grounding DINO and SAM. It first locates the target object in the video frame based on the target object cue word, and then uses SAM to accurately segment these target objects based on the location information obtained from the location, thereby obtaining the mask of the target object.

[0114] The editing area is the image region in a video frame that needs to be modified or processed; the non-editing area is the image region that does not need to be modified.

[0115] After obtaining the mask of the target object in each video frame, the video frame can be segmented based on the mask and the video frame to obtain the editable and non-editable regions of each video frame.

[0116] like Figure 4 As shown, this embodiment of the application achieves frame-by-frame location of the target object by integrating the target object prompts output by the Grounding-SAM and user command analysis modules, and automatically generates corresponding frame-by-frame mask markers. The mask markers can clearly and accurately distinguish between edited and non-edited areas in video frames, thereby accurately constraining the editing range spatially. This method not only eliminates the tedious step of manual frame-by-frame annotation by the user, but also ensures mask accuracy and consistency. Based on the background data regions and non-background data regions of various noise latent variables obtained from the mask division, high-quality modification of specified areas can be achieved while maintaining the overall structure and temporal consistency of the video.

[0117] In practice, video tracking algorithms (e.g., Grounding-SAM) may encounter the following problems: the model's overall mask generation for the video is unstable, such as the target object being completely occluded in the middle frame of the video, causing the generated mask to fail to propagate correctly, or the target object not appearing in the first frame of the video to be edited, causing the model to fail to detect the target object, etc.

[0118] To address the aforementioned issues, in some embodiments, the video to be edited can be divided into multiple video segments; each video segment contains at least one video frame, and there are overlapping video frames between adjacent video segments; for each video segment, a target video tracking algorithm is used to determine the mask of each image frame in the video segment based on the target object prompt; based on the mask of each image frame in each video segment, the mask of each image frame in the video to be edited is obtained.

[0119] When dividing a video to be edited into multiple video segments, adjacent video segments will have some identical video frames. These identical video frames are called overlapping video frames. For example, if a video to be edited is divided into segment A (containing video frames 1-5) and segment B (containing video frames 4-8), then video frames 4 and 5 are the overlapping video frames of segment A and segment B.

[0120] In the specific implementation process, the cross-union ratio (CUI) of the masks of overlapping video frames between adjacent video segments can be determined one by one; if all CUIs are determined to be greater than a preset threshold, the masks of each video frame in the adjacent video segments are fused to obtain the mask of each image frame in the video to be edited.

[0121] Intersection over Union (IoU) is a metric used to measure the similarity between two sets. In video editing, it can be used to measure the degree of overlap between two masks.

[0122] When the intersection-over-union (IoU) ratio of the overlapping video masks between all adjacent video segments is greater than a preset threshold, it indicates that the target objects corresponding to the adjacent video segments have high mask consistency in the overlapping part, and mask fusion can be performed. In the specific implementation process, the fusion weight can be determined first based on the IoU ratio. The larger the IoU ratio, the greater the weight of the corresponding segment mask. Then, the masks are fused frame by frame. For non-overlapping video frames that belong to only one segment, the mask corresponding to that segment is directly used as the final mask. For overlapping video frames, the adjacent masks are weighted and averaged according to the determined weights. Finally, after processing all video frames in the above way, masks with clear boundaries, cross-frame stability, and robustness to deformation / occlusion are obtained for each image frame of the video to be edited.

[0123] If it is determined that the cross-union ratio (CUI) of overlapping video masks between adjacent video segments is less than a preset threshold, the video tracking algorithm can be reused to initialize the mask near the abnormal video segment and limit the propagation span. Morphological post-processing (such as opening and closing operations for noise reduction and hole filling) can also be performed on the mask. If necessary, the mask generation of that video segment can be rerun until the CUI of overlapping video masks between all adjacent video segments is greater than the preset threshold.

[0124] In some embodiments, a large language model can also be used as a "quality inspector" to perform mask detection and correction tasks on each video frame of the video to be edited, as described above. This automatically reviews the mask from three perspectives: temporal integrity, spatial coverage, and temporal consistency (e.g., detecting multiple consecutive empty or abruptly changing mask areas, whether the mask completely covers the target without excessively intruding into the background, and whether the mask trajectory conforms to the target's motion pattern). If an anomaly is detected, an appropriate repair strategy is selected based on the problem type, including appropriately lowering or raising the confidence threshold, expanding or tightening the target object cue words, re-initializing the mask using the video tracking algorithm near the abnormal video segment and limiting its propagation span, and performing morphological post-processing on the mask (such as opening / closing operations for noise reduction and hole filling). If necessary, the mask segmentation of the video segment is re-run until the mask sequence passes the large language model's re-inspection. Through this verification process, some inherent problems of video tracking algorithms can be effectively alleviated, thereby stably generating high-quality video masks.

[0125] In the embodiments provided in this application, the video to be edited is divided into multiple video segments with overlapping frames. A target video tracking algorithm is used, with each video segment as a unit, to determine the mask of the image frames in each video segment based on the target object prompts. The cross-union ratio (CUI) of the masks of overlapping frames in adjacent video segments is determined sequentially. If all CUIs are greater than a preset threshold, the masks of each frame in the adjacent segments are fused to obtain the final result. This allows for the output of a mask sequence with clear boundaries and consistent cross-frames without manual frame-by-frame annotation, effectively improving data processing efficiency and enhancing the spatial accuracy and temporal stability of video editing. This effectively avoids the content collapse problem in the generated target video caused by factors such as inversion accuracy and text description quality when using training-free frameworks for video editing in related technologies. For example, the originally clear and reasonable object structure or scene layout in the video becomes chaotic; the semantic information expressed by the video is lost or becomes difficult to understand; a large amount of texture information in the video is lost, and the object surface becomes smooth and lacks detail.

[0126] In the specific implementation process, such as Figure 4 As shown, step 304 can be executed through the automatic mask generation module to generate accurate and time-consistent masks for the target objects in each video frame.

[0127] The video editing apparatus provided in the embodiments of this application is described below. The video editing apparatus described below can be referred to in correspondence with the video editing method described above.

[0128] Figure 5 This is the video editing device provided in the embodiments of this application. For example... Figure 5 As shown, the video editing device 500 includes the following modules.

[0129] The acquisition module 510 is used to: acquire the initial noise latent variables of each video frame in the video to be edited.

[0130] Generation module 520 is used to iteratively perform the following operations for each video frame to obtain the target video: Based on the original video prompts, a first inverse denoising process is performed on the first original noise latent variable at the current time step to obtain a second original noise latent variable, which is then used as the first original noise latent variable for the next time step. Based on the target video prompts, a second inverse denoising process is performed on the first target noise latent variable at the current time step to obtain a second target noise latent variable. Based on the background data region of the second original noise latent variable and the second target noise latent variable, a third target noise latent variable is obtained, which is then used as the first target noise latent variable for the next time step. The background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited. The first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0131] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call a computer program stored in the memory 630 to execute steps of a video editing method, such as: Obtain the initial noise latent variables for each video frame in the video to be edited; for each video frame, iteratively perform the following operations to obtain the target video: based on the original video prompts, perform a first inverse denoising process on the first original noise latent variable at the current time step to obtain a second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step; based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain a second target noise latent variable; based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain a third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; wherein, the first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0132] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the video editing methods provided in the above embodiments, such as: Obtain the initial noise latent variables for each video frame in the video to be edited; for each video frame, iteratively perform the following operations to obtain the target video: based on the original video prompts, perform a first inverse denoising process on the first original noise latent variable at the current time step to obtain a second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step; based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain a second target noise latent variable; based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain a third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; wherein, the first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0134] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including: Obtain the initial noise latent variables for each video frame in the video to be edited; for each video frame, iteratively perform the following operations to obtain the target video: based on the original video prompts, perform a first inverse denoising process on the first original noise latent variable at the current time step to obtain a second original noise latent variable, and use the second original noise latent variable as the first original noise latent variable for the next time step; based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain a second target noise latent variable; based on the background data region of the second original noise latent variable and the second target noise latent variable, obtain a third target noise latent variable, and use the third target noise latent variable as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; wherein, the first original noise latent variable and the first target noise latent variable at the initial time step are initialized with the initial noise latent variable.

[0135] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video editing method characterized by, The method comprises: obtaining initial noise latent variables of each video frame in a video to be edited; for each video frame, iteratively performing the following operations to obtain a target video: performing a first reverse denoising process on the first original noise latent variable of the current time step according to the original video prompt to obtain a second original noise latent variable, and taking the second original noise latent variable as the first original noise latent variable of the next time step; performing a second reverse denoising process on the first target noise latent variable of the current time step according to the target video prompt to obtain a second target noise latent variable; obtaining a third target noise latent variable according to the background data region of the second original noise latent variable and the second target noise latent variable, and taking the third target noise latent variable as the first target noise latent variable of the next time step; wherein the background data region of the second original noise latent variable corresponds to a non-editing region of the video to be edited; wherein the initial time step of the first original noise latent variable and the first target noise latent variable is initialized by the initial noise latent variable.

2. The video editing method of claim 1, wherein, The method further comprises: for each video frame in the video to be edited, mapping the video frame into a latent space processed by the target diffusion-based video generation model using an inversion process of the target diffusion-based video generation model to obtain the initial noise latent variable of each video frame.

3. The video editing method of claim 1, wherein, The method further comprises: replacing the background data region of the second target noise latent variable with the background data region of the second original noise latent variable corresponding to the current time step to obtain the third target noise latent variable; wherein the background data region of the second target noise latent variable corresponds to a non-editing region of the video to be edited.

4. The video editing method of claim 1, wherein, The method further comprises: allocating a first weight to each data unit of the background data region of the second original noise latent variable; allocating a second weight to each data unit of the background data region of the second target noise latent variable; performing a weighted sum operation on each data unit of the background data region of the second original noise latent variable and each data unit of the background data region of the second target noise latent variable based on the first weight and the second weight to obtain data in the background data region of the third target noise latent variable; taking the non-background data region of the second target noise latent variable as the non-background data region of the third target noise latent variable.

5. The video editing method of claim 1, wherein, The third target noise latent variable comprises a background data sub-region, a transition data sub-region and a non-background data sub-region. The non-background data sub-region corresponds to the editing area of ​​the video to be edited; the background data sub-region and the transition data sub-region together correspond to the non-editing area of ​​the video to be edited; the transition data sub-region is located between the background data sub-region and the non-background data sub-region to achieve a smooth transition of data features between the background data sub-region and the non-background data sub-region.

6. The video editing method of claim 5, wherein, The step of obtaining the third target noise latent variable based on the background data region of the second original noise latent variable and the second target noise latent variable includes: The first data region and the second data region are merged to obtain the data of the transition data sub-region; wherein, the first data region is the data region in the second original noise latent variable that corresponds to the transition data sub-region; and the second data region is the data region in the second target noise latent variable that corresponds to the transition data sub-region. The third data region is used as the background data sub-region; The fourth data region is designated as the non-background data sub-region. The third data region is the data region in the second original noise latent variable that corresponds to the background data sub-region; the fourth data region is the data region in the second target noise latent variable that corresponds to the non-background data sub-region.

7. The video editing method of claim 6, wherein, The process of merging the first data region and the second data region to obtain the data of the transition data sub-region includes: Determine the background weight of each data unit in the first data region; Determine the generation weight of each data unit in the second data region; Based on the background weight and the generation weight, a weighted summation operation is performed on each data unit in the first data region and the second data region to obtain the data of the transition data sub-region.

8. The video editing method of claim 7, wherein, Determining the background weight of each data unit in the first data region includes: Assign a first background sub-weight to each data unit in the first data region; wherein, the first background sub-weight is associated with the time step of the second inverse denoising process, and the first background sub-weight corresponding to each time step gradually decreases from the initial time step. The background weight of each data unit in the first data region is determined based on the first background sub-weight of each data unit in the first data region.

9. The video editing method of claim 8, wherein, The step of determining the background weight of each data unit in the first data region based on the first background sub-weight of each data unit in the first data region includes: For each data unit in the first data region, a second background sub-weight is assigned to the data unit based on the minimum distance between the data unit and the non-background data region of the second original noise latent variable; wherein, the non-background data region of the second original noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a larger second background sub-weight. The background weight of each data unit in the first data region is determined based on the first background sub-weight and the second background sub-weight of each data unit in the first data region.

10. The video editing method of claim 9, wherein, Determining the background weight of each data unit in the first data region based on the first background sub-weight and the second background sub-weight of each data unit in the first data region includes: The sum of the first background sub-weight and the second background sub-weight of the data unit is used as the background weight of each data unit in the first data region.

11. The video editing method of claim 7, wherein, Determining the generation weight of each data unit in the second data region includes: Assign a first generated sub-weight to each data unit in the second data region; wherein, the first generated sub-weight is associated with the time step of the second reverse denoising process, and the first generated sub-weight gradually increases for each time step starting from the initial time step; Based on the first generation sub-weight of each data unit in the second data region, the generation weight of each data unit in the second data region is determined.

12. The video editing method of claim 11, wherein, The step of determining the generation weight of each data unit in the second data region based on the first generation sub-weight of each data unit in the second data region includes: For each data unit in the second data region, a second generation sub-weight is assigned to the data unit based on the minimum distance between the data unit and the non-background data region of the second target noise latent variable; wherein, the non-background data region of the second target noise latent variable corresponds to the editing region of the video to be edited; the data unit farther away from the non-background data region has a smaller second generation sub-weight; The generation weight of each data unit in the second data region is determined based on the first generation sub-weight and the second generation sub-weight of each data unit in the second data region.

13. The video editing method of claim 12, wherein, The step of determining the generation weight of each data unit in the second data region based on the first generation sub-weight and the second generation sub-weight of each data unit in the second data region includes: The sum of the first and second generation sub-weights of each data unit in the second data region is used as the generation weight of each data unit in the second data region.

14. The video editing method of claim 1, wherein, The method further includes: Using a visual language model, the attribute features of the video to be edited are extracted to obtain the original video prompt words.

15. The video editing method of claim 14, wherein, The method further includes: Get editing instructions; Using a large language model, the attribute features of the target video are determined based on the original video prompts and the editing instructions, thereby obtaining the target video prompts.

16. The video editing method of claim 15, wherein, The method further includes: Using a large language model, target object prompts are determined based on the original video prompts and the editing instructions; wherein, the target object prompts are used to identify the object to be edited in the video to be edited.

17. The video editing method of claim 16, wherein, The method further includes: The original video prompts, the target video prompts, and the target object prompts are validated using a large language model. If the confidence level of any one of the original video prompt, the target video prompt, and the target object prompt is determined to be lower than a preset threshold, it is marked as a low-confidence clause.

18. The video editing method of claim 16, wherein, The method further includes: Using a target video tracking algorithm, the editable and non-editable regions of each image frame in the video to be edited are determined based on the target object prompts.

19. The noise control based video editing method of claim 18, wherein, The step of using a target video tracking algorithm to determine the editable and non-editable regions of each image frame in the video to be edited, based on the target object prompts, includes: Using a target video tracking algorithm, the mask of each image frame in the video to be edited is determined based on the target object prompt words; Based on the mask of each image frame in the video to be edited, the editing area and non-editing area of ​​each image frame in the video to be edited are determined.

20. The noise control based video editing method of claim 19, wherein, The step of using a target video tracking algorithm to determine the mask of each image frame in the video to be edited based on the target object prompts includes: The video to be edited is divided into multiple video segments; each video segment contains at least one video frame, and there are overlapping video frames between adjacent video segments; For each video segment, the target video tracking algorithm is used to determine the mask of each image frame in the video segment based on the target object prompt words; The mask of each image frame in the video to be edited is obtained based on the mask of each image frame in each video segment.

21. The noise control based video editing method of claim 20, wherein, The step of obtaining the mask of each image frame in the video to be edited based on the mask of each image frame in each video segment includes: The intersection-union ratio of the masks of overlapping video frames between adjacent video segments is determined successively; If it is determined that all intersection-union ratios are greater than a preset threshold; The masks of each video frame in the adjacent video segments are fused to obtain the mask of each image frame in the video to be edited.

22. A video editing apparatus characterized by comprising: include: The acquisition module is used to: acquire the initial noise latent variables of each video frame in the video to be edited; The generation module is used to iteratively perform the following operations on each video frame to obtain the target video: Based on the original video prompts, the first inverse denoising process is performed on the first original noise latent variable at the current time step to obtain the second original noise latent variable, and the second original noise latent variable is used as the first original noise latent variable at the next time step. Based on the target video prompts, perform a second inverse denoising process on the first target noise latent variable at the current time step to obtain the second target noise latent variable; Based on the background data region of the second original noise latent variable and the second target noise latent variable, a third target noise latent variable is obtained, and the third target noise latent variable is used as the first target noise latent variable for the next time step; wherein, the background data region of the second original noise latent variable corresponds to the non-editing region of the video to be edited; Specifically, the first original noise latent variable and the first target noise latent variable of the initial time step are initialized with the initial noise latent variable.

23. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that When the processor executes the computer program, it implements the steps of the video editing method according to any one of claims 1 to 21. 24.A non-transitory computer-readable storage medium having stored thereon a computer program. When the computer program is executed by a processor, it implements the video editing method as described in any one of claims 1 to 21.

25. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video editing method according to any one of claims 1 to 21.

Images