Video special effect editing method and system based on text video large model

Abstract

The invention provides a video special effect editing method and system based on a text video large model, and the method comprises the steps: constructing a space-time sparse condition marking sequence through a source video, so as to reduce the calculation redundancy and reserve key dynamic information; a position coding correction and causal attention mechanism is introduced into the text video large model to isolate the interference of potential noise marks, the text video large model is firstly finely adjusted into a general video editing model by using large-scale video data through a double-stage fine adjustment strategy, and then the fine adjustment is further performed on a special effect editing data set based on an EffectLoRA module to obtain a special effect editing model; and obtaining a video special effect editing model with a text controllable special effect generation capability. And finally, special effect injection is realized through a space-time sparse condition construction and text instruction combined driving model, so that a generated result strictly keeps the space-time structure consistency of an original video while meeting text description. The method has the advantages of being accurate in editing effect, high in universality and high in real-time performance.

Description

Technical Field

[0001] This invention relates to the field of video editing within the field of artificial intelligence technology, and more specifically, to a video effects editing method and system based on a large-scale textual video model. Background Technology

[0002] Visual effects (VFX) aim to create or edit existing videos by incorporating eye-catching visual elements such as fire, cartoon characters, or particle effects. As a core technology in filmmaking, gaming, and virtual reality, VFX enriches visual storytelling, highlights key elements, and creates immersive experiences. However, traditional video VFX production workflows rely on complex animation design, computer-generated imagery, and professional post-production compositing. These workflows result in high production costs, long lead times, and significant human intervention, hindering personalized or real-time applications.

[0003] In recent years, advancements in text-to-video (T2V) generation technology have opened up new possibilities for automated VFX creation. However, the application of existing technologies in video VFX editing (i.e., automatically adding or modifying effects in existing videos) remains largely unexplored.

[0004] As a unique and higher-level video editing task, video VFX editing differs fundamentally from video VFX generation. Its core objective is to seamlessly and realistically integrate visual effects into the source video while strictly preserving the spatial structure and temporal coherence of the original content. Although recent video editing models have made significant progress in various editing tasks, they still struggle to meet the stringent requirements of video effects editing. Existing video editing models typically allow for some degree of background or appearance variation, making it difficult to ensure pixel-level consistency with the source video. This limitation is unacceptable in video effects editing, as the background must remain completely unchanged. Furthermore, unlike general video editing methods that leverage large-scale data to improve performance, generating high-quality paired effects data is extremely challenging, limiting the scalability of model training. Efficient video effects editing must learn unique patterns of effects injection from these high-quality paired samples to achieve a physically consistent fusion of effects and real-world scenes.

[0005] The aforementioned challenges mean that automated video VFX editing remains largely an unsolved problem. Therefore, a solution capable of automating video effects editing is urgently needed. Summary of the Invention

[0006] In view of the problems in the prior art, the purpose of this invention is to provide a video special effects editing method and system based on a large model of textual video. Addressing the challenges of video special effects editing methods, this invention represents the source video as a spatiotemporally sparse context condition and utilizes the inherent context learning capability of the DiT architecture to inject visual effects into the source video while keeping the spatiotemporal characteristics of the source video unchanged, thereby achieving automated video special effects editing.

[0007] On the one hand, this invention provides a video effects editing method based on a large model of textual content, comprising the following three stages: Spatiotemporal sparse conditional label sequence construction stage: Based on the source video, obtain the spatial sparse label sequence and the temporal sparse label sequence, and concatenate the spatial sparse label sequence and the temporal sparse label sequence along the label dimension to obtain the spatiotemporal sparse conditional label sequence; The two-stage fine-tuning stage includes: obtaining a target video corresponding to the source video based on a preset target video database, obtaining a target marker sequence of the target video using a pre-trained video encoder, adding noise to the target marker sequence to obtain a noisy target marker sequence, so as to fine-tune the preset raw video model into a general video editing model for performing general video editing tasks; and, using video effect editing data containing paired visual effect examples, fine-tuning the general video editing model for performing general video editing tasks based on a preset EffectLoRA module, so as to give the general video editing model a specific visual effect editing mode. The inference phase includes: constructing a spatiotemporally sparse conditional label sequence of the source video to be edited, and obtaining a noise target label sequence of the source video to be edited; concatenating the noise target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited along the label dimension, and feeding them into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

[0008] Alternatively, the step of obtaining spatially sparse and temporally sparse label sequences based on the source video includes: Spatial downsampling and first frame extraction are performed on the source video to obtain low-resolution video and first frame; The first frame is encoded as a spatially sparse marker sequence, and the downsampled low-resolution video is encoded as a temporally sparse marker sequence.

[0009] Alternatively, the step of obtaining spatially sparse and temporally sparse label sequences based on the source video includes: The source video is encoded using 3D VAE to obtain a source marker sequence; Extract the marker corresponding to the first frame of the source marker sequence as a spatially sparse marker sequence; and perform sparse sampling on the source marker sequence in the spatial dimension to obtain a temporally sparse marker sequence.

[0010] In addition, an optional approach is that, before fine-tuning the preset text-based video model into a general video editing model for performing general video editing tasks during the two-stage fine-tuning stage, the approach further includes adding a position encoding correction and causal attention module to the pre-trained text-based video model based on the spatiotemporal sparse conditional label sequence. The target marker sequence is subjected to noise processing to fine-tune the textual video model with positional encoding correction and causal attention modules into a general video editing model that can perform general video editing tasks.

[0011] Alternatively, an optional approach is to add positional encoding correction and causal attention modules to the pre-trained textual video large model based on the spatiotemporally sparse conditional label sequence, including: Position coding correction is inserted into the textual video large model to avoid spatiotemporal misalignment between the spatiotemporally sparse label sequence and the generation space; wherein, for the temporally sparse label sequence, a real correspondence between the temporally sparse label sequence and the target region is established to prevent spatial misalignment; for the spatially sparse label sequence, the position coding of the first frame of the target label sequence is used to avoid temporal errors. By designing an attention mask, the bidirectional attention mechanism of the textual video large model is changed to a causal attention mechanism to avoid the influence of noisy target label sequences on the temporally sparse conditional label sequence.

[0012] Alternatively, the attention mask can be designed as follows: in, This represents the mask at row index i and column index j. This represents a sequence of noise target markers.

[0013] Alternatively, a noise-adding process can be applied to the target labeled sequence to fine-tune the textual video model with positional encoding correction and causal attention modules into a general video editing model capable of performing general video editing tasks, including: The target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension and fed into the pre-trained textual video large model; Noise is added only to the target labeled sequence, while the spatiotemporally sparse conditional labeled sequence remains unchanged; the loss function used is as follows: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, For time-sparse labeled sequences, It is a spatially sparse labeled sequence. The velocity field predicted by the large model for the video. For the target vector field, Indicates noise.

[0014] Alternatively, the preferred approach is to fine-tune the general video editing model for performing general video editing tasks based on the preset EffectLoRA module, including: Insert a low-rank LoRA into the general video editing model that performs the general video editing task; The target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension and fed into a general video editing model with low-rank LoRA inserted; Keeping the spatiotemporally sparse conditional label sequence unchanged, noise is added only to the target label sequence, using the following loss function: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, It is a time-sparse labeled sequence. It is a spatially sparse labeled sequence. The velocity field predicted by the large model for the video. For the target vector field, Indicates noise.

[0015] Meanwhile, this invention also provides a video effects editing system based on a large-scale video model, applicable to electronic devices, including: A spatiotemporal sparse conditional label sequence construction unit is used to obtain spatial sparse label sequences and temporal sparse label sequences based on the source video, and to concatenate the spatial sparse label sequences and the temporal sparse label sequences along the label dimension to obtain a spatiotemporal sparse conditional label sequence; a two-stage fine-tuning unit is used to obtain a target video corresponding to the source video based on a preset target video database, and to obtain a target label sequence of the target video using a pre-trained video encoder, and to add noise to the target label sequence to obtain a noisy target label sequence, so as to fine-tune the preset text-based video model into a general video editing model for performing general video editing tasks; and, using video effect editing data containing paired visual effect examples, to fine-tune the general video editing model for performing general video editing tasks based on a preset EffectLoRA module, so as to give the general video editing model a specific visual effect editing mode; The inference unit is used to construct a spatiotemporally sparse conditional label sequence of the source video to be edited, and obtain the noise target label sequence of the source video to be edited; concatenate the noise target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited along the label dimension, and feed them into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

[0016] In another aspect, the present invention also provides an electronic device, the electronic device including a memory, a processor, and a video effects editing program based on a large-scale video model stored in the memory and executable on the processor, wherein the video effects editing program based on a large-scale video model implements the video effects editing method based on a large-scale video model as described above when executed by the processor.

[0017] The video effects editing method and system based on a text-based video large model according to the present invention utilizes source videos to construct spatiotemporally sparse conditional label sequences. This reduces computational redundancy and preserves key dynamic information by performing spatiotemporal sparse labeling on the source videos. Furthermore, a two-stage fine-tuning strategy is employed to achieve efficient, stable, and controllable video effects editing. Specifically, the two-stage training strategy first fine-tunes the text-based video large model into a general video editing model using large-scale video data. Then, based on the EffectLoRA module, it further fine-tunes the model on the effects editing dataset to obtain a video effects editing model with text-controllable effects generation capabilities. Finally, in the inference stage, effects injection is achieved by jointly driving the model through spatiotemporally sparse conditional construction and text instructions, ensuring that the generated results strictly maintain the spatiotemporal structural consistency of the original video while satisfying the text description.

[0018] Compared with existing technologies, the video special effects editing method and system based on the large model of textual video provided by this invention has the characteristics of accurate editing effect, strong versatility and high real-time performance, and can be widely used in film and television production, short video creation and interactive content generation and other scenarios.

[0019] To achieve the foregoing and related objectives, one or more aspects of the invention include the features which will be described in detail below and specifically pointed out in the claims. The following description and accompanying drawings illustrate certain exemplary aspects of the invention. However, these aspects indicate only a few of the various ways in which the principles of the invention can be used. Furthermore, the invention is intended to include all such aspects and their equivalents. Attached Figure Description

[0020] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings and the contents of the claims, and with a more complete understanding of the invention. In the drawings: Figure 1 This is a schematic diagram of the video effects editing method based on a large-scale video model according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of a video effects editing method based on a large-scale video model according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the video effects editing method based on a large-scale text-based video model according to Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the logic structure of an electronic device according to an embodiment of the present invention; Figure 5 The image shows an example of the video effects editing effect achieved by applying the video effects editing method and system based on the large-scale video model of the present invention. Figure 6 This diagram illustrates the comparison of the effects of applying the present invention with other existing video editing solutions.

[0021] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation

[0022] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.

[0023] Due to current limitations, it is difficult to meet the requirements. Therefore, this invention proposes a video effects editing method and system based on a large-scale textural video model.

[0024] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0025] Video effects editing is a process based on a text-based video model to edit source videos according to text instructions, giving the source video visual effects that are aligned with the text instructions. At the same time, the visual effects are seamlessly integrated with the source video, and the spatiotemporal feature information of the edited video is strictly consistent with the spatiotemporal feature information of the source video.

[0026] Text-generated video models are those that generate corresponding videos from simple text descriptions (in English). Existing text-generated video models include Wan 2.2 and CogVideoX. Wan 2.2 is an advanced generative large model in the field of computer vision, capable of generating videos aligned with the text.

[0027] DiT (Diffusion Transformer) is a generative model that applies the Transformer architecture to diffusion models, aiming to improve the performance and quality of video generation tasks. In DiT, the input video is divided into several fixed-size patches, and each patch is linearly embedded into a vector, forming a series of labeled sequences. These labeled sequences are fed into a Transformer encoder, which captures the spatiotemporal features of the video through a self-attention mechanism. This design enables the model to handle complex video generation tasks more effectively. The denoising network in Wan 2.2 consists of several DiT modules.

[0028] A 3D variational autoencoder (3D VAE) is a generative model used to learn data distributions. It consists of an encoder and a decoder; the encoder maps the input data to a latent space, and the decoder reconstructs the input data from the latent space. In the Wan 2.2 model, the 3D VAE encodes the input video information into a sequence of latent tags, performs denoising, and then generates the corresponding video through the decoder. This method enables the model to capture the complex distribution of data during generation, improving the quality and diversity of the generated videos.

[0029] LoRA (Low-Rank Adaptation) is an efficient parameter fine-tuning technique designed to optimize the adaptability of large-scale pre-trained models for specific tasks. Its core idea is to freeze the weights of the pre-trained model and then insert trainable low-rank matrices into key parts of the model to achieve task-specific adaptation. This method significantly reduces the number of parameters and computational resources required for fine-tuning while maintaining or improving model performance. Compared to traditional full-parameter fine-tuning, LoRA only needs to train the newly added low-rank matrix, avoiding retraining the entire model. This not only reduces computational and storage costs but also reduces the risk of overfitting, enabling the model to adapt to new tasks more efficiently.

[0030] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] Figure 1 , Figure 2 The process and principle of the video special effects editing method based on the large model of textual video according to the embodiments of the present invention are illustrated by way of example.

[0032] like Figure 1 and Figure 2 As shown in the figure, the video effects editing method based on the large model of textual content provided in this embodiment mainly includes the following steps: S110: Spatiotemporal sparse conditional label sequence construction stage; used to construct the source video into a spatiotemporal sparse conditional label sequence.

[0033] In the embodiments provided by this invention, there are two ways to construct the source video into a spatiotemporally sparse conditional label sequence: 1) Downsample the source video and extract the first frame. Use 3D VAE to encode the downsampled video and the first frame into a temporally sparse label sequence and a spatially sparse label sequence, respectively. Then, stitch them together along the label dimension to obtain a spatiotemporally sparse conditional label sequence.

[0034] 2) The source video is encoded using 3D VAE to obtain the source tag sequence. The tag corresponding to the first frame of the source tag sequence is extracted as the spatial sparse tag sequence. The source tag sequence is sparsely sampled in the spatial dimension to obtain the temporal sparse tag sequence. The tags are then spliced ​​along the tag dimension to obtain the spatiotemporally sparse conditional tag sequence.

[0035] The aforementioned 3D VAE is a pre-trained video encoder used in this invention.

[0036] Specifically, in one embodiment of the present invention, the process of constructing a spatiotemporally sparse conditional label sequence from the source video further includes the following steps: S111: Spatial downsampling and first frame extraction are performed on the source video to obtain the low-resolution video and first frame of the source video; S112: Encode the first frame into a spatially sparse marker sequence; and encode the downsampled low-resolution video into a temporally sparse marker sequence; S113: The spatial sparse label sequence and the temporal sparse label sequence are concatenated along the label dimension to form a spatiotemporal sparse conditional label sequence.

[0037] The text-based video model used in this embodiment can be implemented using Wan 2.2. For ease of explanation, Wan 2.2 will be used as an example of the implementation of the text-based video model in the following embodiments for specific explanations.

[0038] Specifically, as an example, the specific process for constructing the above spatiotemporally sparse conditional label sequence includes: processing the source video... Downsampling to low-resolution video Using 3D VAE to convert low-resolution video Encoded as a time-sparse labeled sequence Extract source video The first frame Using 3D VAE to capture the first frame Encoded as a spatially sparse label sequence ; Concatenate time-sparse label sequences along the label dimension and spatially sparse labeled sequences As a spatiotemporally sparse conditional marker sequence .

[0039] After converting the source video into a spatiotemporally sparse conditional label sequence, step S120, the two-stage fine-tuning stage, is executed. Specifically, the two-stage fine-tuning stage in this embodiment consists of a general video editing fine-tuning stage and an EffectLoRA module fine-tuning stage. In the improved architecture of the raw video large model, positional encoding correction and causal attention modules are added to the pre-trained DiT-based raw video large model based on the spatiotemporally sparse conditional label sequence.

[0040] Furthermore, in a preferred embodiment of the present invention, in order to obtain better special effects editing results, before fine-tuning the preset text-based video model into a general video editing model for performing general video editing tasks during the aforementioned two-stage fine-tuning stage, the architecture of the text-based video model can be improved first. This includes: adding positional encoding correction and causal attention modules to the pre-trained text-based video model based on the spatiotemporal sparse conditional label sequence; introducing positional encoding correction and causal attention mechanisms into the text-based video model, where positional encoding correction is used to align sparse conditional labels with the target generation space, and causal attention is used to isolate interference from potential noise labels, ensuring clear conditional dependencies and correct information flow direction.

[0041] As an example, the improvements to the Wensheng video big data model architecture include the following steps: S1211: Insert positional encoding correction into the Wensheng video large model to avoid spatiotemporal misalignment between the spatiotemporally sparse label sequence and the generation space; the correction includes temporal correction and spatial correction. Specifically, for the temporally sparse label sequence... To prevent spatial misalignment, a real-world correspondence is established between the temporally sparse labeled sequence and the target region. in, Position encoding representing a time-sparse labeled sequence Represents the target label sequence Location encoding, i and j These represent the original location indices of the target video. n Indicates the downsampling factor of the source video.

[0042] For spatially sparse labeled sequences Using target marker sequences The position encoding of the first frame is used to avoid timing errors.

[0043] In this embodiment, the target tag sequence is a tag sequence of target videos stored in a preset target video database, used to pair with source videos according to preset rules. Specifically, as an example, in the preset target video database, target videos can be obtained from general editing model data in the following way: high-resolution source videos are obtained from Pexels, and the obtained high-resolution source videos are processed to construct target videos in five categories: addition, deletion, replacement, attribute modification, and style migration.

[0044] For adding and deleting, GPT-4o can be used to obtain the subject category of each video segment, and the Grounding DINO and SAM models can be used to obtain the masked region of the subject in the video. The masked region and the subject deletion model DiffuEraser are then used to obtain the video after deleting the subject from the source video. The video after deleting the subject and the source video are each a pair of additions (the video after deleting the subject is the source video, and the source video is the target video) and deletions (the source video remains the source video, and the video after deleting the subject is the target video).

[0045] For subject replacement and attribute modification, VACE is used to edit the source video to obtain the corresponding video with modified subject and video with changed subject attributes. These videos are the target videos corresponding to the source video.

[0046] For style migration, FLUX-kontext is used to edit the first frame of the source video and extract its depth map. The edited first frame and depth map are then fed into VACE to obtain the target video with the modified style.

[0047] S1212: By designing an attention mask, the bidirectional attention mechanism of the Wensheng video large model is changed to a causal attention mechanism to avoid the temporally sparse conditional label sequence being affected by the noisy target label sequence, i.e., the noisy target label sequence. The impact of this. The attention mask design in this embodiment is as follows: in, This represents the mask at row index i and column index j. Represents a noisy target label sequence. Represents a time-sparse labeled sequence. This represents a spatially sparse labeled sequence.

[0048] After completing the architectural improvements to the text-based video model, the first stage of fine-tuning can be performed: general video editing fine-tuning. Specifically, in the general video editing fine-tuning stage, target videos corresponding to the source videos are obtained based on a preset target video database, and target marker sequences of the target videos are obtained using a pre-trained video encoder. Noise is added to the target marker sequences to fine-tune the text-based video model with positional encoding correction and causal attention modules into a general video editing model capable of performing general video editing tasks. More specifically, this general video editing fine-tuning stage includes the following steps: S1221: Obtain the target video corresponding to the source video based on a preset target video database; S1222: Encode the target video into a target marker sequence using 3D VAE. And along the marker dimension, the target marker sequence and the spatiotemporal sparse conditional label sequence The images are pieced together and fed into a pre-trained large-scale text-based video model. S1223: When fine-tuning the pre-trained text-based video model, only the target labeled sequence is adjusted. Add noise to the spatiotemporally sparse conditional label sequence. It remains unchanged. The loss function used is as follows: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, It is a time-sparse labeled sequence. It is a spatially sparse labeled sequence. This is a velocity field predicted by a large video model for Wensheng. It can generate a noise-free target marker sequence based on a noisy target marker sequence at time step t. Indicates noise. Given a target vector field, it can directly generate a noise-free target label sequence based on a noisy target sequence. This loss function forces the prediction results of the large-scale model for text-based videos to closely approximate the real data.

[0049] After the general video editing fine-tuning stage, the process moves to the EffectLoRA module fine-tuning stage. Using video effect editing data containing a small number of paired visual effect examples, the general video editing model performing the general video editing task is fine-tuned based on the preset EffectLoRA module to give the model editing modes for specific visual effects. Specifically, the EffectLoRA module fine-tuning stage includes the following steps: S1231: Insert a low-rank LoRA into the general video editing model that performs the general video editing task.

[0050] S1232: The target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension and fed into a general video editing model with low-rank LoRA inserted; First, the source video is constructed into a spatiotemporally sparse conditional label sequence using the aforementioned spatiotemporally sparse conditional label sequence. Then, using 3D VAE, the source video paired with the target video is encoded into a target marker sequence. And along the marker dimension, the target marker sequence and spatiotemporal sparse conditional label sequence They are assembled together and then fed into the model.

[0051] S1233: Keep the spatiotemporal sparse conditional label sequence unchanged, and only add noise to the target label sequence.

[0052] Specifically, the aforementioned video effects editing data can be obtained through preset effects-adding software. For each category of effects data, effects-adding software such as After Effects can be used to modify the source video, adding visual effects to create a video with visual effects. Correspondingly, for the source video before adding effects, the video with visual effects is the target video paired with that source video.

[0053] When fine-tuning using the EffectLoRA module, only the target marker sequence is considered. Add noise to the spatiotemporally sparse conditional label sequence. It remains unchanged. The loss function used is as follows: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, It is a time-sparse labeled sequence. It is a spatially sparse labeled sequence. The velocity field predicted by the large model for the video. For the target vector field, Indicates noise.

[0054] It is evident that the two fine-tuning processes in the two-stage fine-tuning use the same loss function, the difference being that the training data is inconsistent.

[0055] After the two-stage fine-tuning is completed, step S130, the inference stage, is executed: the fine-tuned model is used to perform the video effects editing task. This includes: constructing the source video to be edited as a spatiotemporally sparse conditional label sequence, and obtaining the noise target label sequence of the source video to be edited; concatenating the noise target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited along the label dimension, and feeding them into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

[0056] More specifically, the reasoning stage includes the following steps: S131: Constructing a spatiotemporally sparse conditional label sequence from the source video to be edited. ; S132: A sequence of latent noise labels randomly sampled along the label dimension and spatiotemporal sparse conditional label sequence To splice; S133: The concatenated data is fed into a general editing model with EffectLoRA for denoising. During the denoising process, only the data is processed... Denoising is performed on the spatiotemporally sparse calibrated sequence. It remains unchanged.

[0057] The denoised Z is then decoded by a pre-trained video decoder to obtain a video with visual effects.

[0058] Combination Figure 2 As shown, using the video effects editing method based on the Wensheng video large model, the low-resolution video after downsampling of the source video is encoded into a temporally sparse labeled sequence using 3D VAE, and the first frame is encoded into a spatially sparse labeled sequence using 3D VAE. The temporally sparse labeled sequence and the spatially sparse labeled sequence are concatenated along the labeled dimension to obtain a spatiotemporally sparse conditional labeled sequence. The target video paired with the source video is encoded into a target labeled sequence using 3D VAE, and noise is added to the target labeled sequence to obtain a noisy target labeled sequence, i.e., a noisy target labeled sequence. The noisy target labeled sequence and the spatiotemporally sparse labeled sequence are concatenated along the labeled dimension and fed into the Wensheng video large model for denoising. After obtaining the output, the spatiotemporally sparse labeled sequence is discarded, and noise is calculated only on the target labeled sequence.

[0059] In one embodiment of the present invention, in the fine-tuning stage of the general video editing model in the above-mentioned two-stage fine-tuning stage, the dataset used contains 40,000 paired general video editing data, including five task types: subject deletion, subject addition, subject swapping, attribute modification, and style transfer. The pre-trained raw video model is fine-tuned using LoRA with a rank of 96, with 50,000 training iterations. The trained LoRA weights are then merged into the weights of the pre-trained raw video model to obtain the general video editing model. In the EffectLoRA module fine-tuning stage, the paired video effects editing dataset contains 15 effects editing types, each containing paired data of 20 source videos and target videos with visual effects. The EffectLoRA module with a rank of 32 is inserted into the general video editing model for fine-tuning, with 1,000 training iterations. In the inference stage, the denoising processing time step is 50.

[0060] Compared with existing technologies, the video effects editing method based on textual video big model provided by the present invention constructs the source video as a spatiotemporally sparse conditional label sequence and uses a two-stage fine-tuning strategy to gradually convert the pre-trained textual video big model into a video effects editing model, so as to inject visual effects into the source video while following text instructions, and keep the spatiotemporal information of the source video unchanged.

[0061] The above describes a video effects editing method based on a textual video model according to an embodiment of the present invention. It can be determined that using the above-described video effects editing method based on a textual video model of the present invention for video effects editing can ensure that the generated result strictly maintains the spatiotemporal structure consistency of the original video while satisfying the textual description. The present invention features accurate editing effects, strong versatility, and high real-time performance, and can be widely applied in film and television production, short video creation, and interactive content generation scenarios. The above-described video effects editing method based on a textual video model of the present invention can be implemented in software, hardware, or a combination of both.

[0062] Corresponding to the above-mentioned video effects editing method based on the large model of textual content video, the present invention also provides a video effects editing system based on the large model of textual content video.

[0063] Figure 3 The framework structure of a video effects editing system 500 based on a large-scale video model according to an embodiment of the present invention is shown. Figure 3 As shown, the video effects editing system based on a large-scale textual video model provided by this invention generates videos using the video effects editing method based on a large-scale textual video model as described above. According to the functions implemented, the video effects editing system 500 based on a large-scale textual video model mainly includes a spatiotemporal sparse conditional label sequence construction unit 510, a two-stage fine-tuning unit 520, and an inference unit 530.

[0064] The unit of this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0065] Furthermore, in one possible implementation of the present invention, the functions of each module / unit are as follows: The spatiotemporal sparse conditional label sequence construction unit 510 is used to construct a spatiotemporal sparse conditional label sequence from a source video using 3D VAE. This unit acquires a spatial sparse label sequence and a temporal sparse label sequence from the source video, and concatenates the spatial and temporal sparse label sequences along the label dimension to obtain the spatiotemporal sparse conditional label sequence.

[0066] In a specific embodiment of the present invention, the above-mentioned spatiotemporal sparse conditional labeling sequence construction unit 510 further includes: The sampling and extraction unit 511 is used to perform spatial downsampling and first frame extraction on the source video; The encoding unit 512 is used to encode the first frame into a spatially sparse marker sequence using a pre-trained video encoder, and to encode the downsampled low-resolution video into a temporally sparse marker sequence. The splicing unit 513 splices the spatial sparse label sequence and the temporal sparse label sequence into a spatiotemporal sparse conditional label sequence; A two-stage fine-tuning unit 520 is used to fine-tune the pre-trained raw video model into a video effects editing model. Specifically, the two-stage fine-tuning unit 520 further includes: The video editing fine-tuning unit 522 is used to obtain the target video corresponding to the source video based on the preset target video database, and to obtain the target label sequence of the target video using a pre-trained video encoder, and to add noise to the target label sequence to obtain a noisy target label sequence, so as to fine-tune the preset raw video model into a general video editing model that performs general video editing tasks. EffectLoRA module fine-tuning unit 523 uses video effect editing data containing a small number of paired visual effect examples to fine-tune the general video editing model that performs the general video editing task based on the preset EffectLoRA module, so as to give the general video editing model a specific visual effect editing mode.

[0067] In another preferred embodiment of the present invention, the dual-stage fine-tuning unit 520 may further include an architecture improvement unit 521, which is used to add position encoding correction and causal attention modules to the pre-trained DiT-based textual video large model based on the spatiotemporal sparse conditional label sequence. After the two-stage fine-tuning is completed, the fine-tuned model can be used for video effects editing tasks. Specifically, through the inference unit 530, the source video to be edited is constructed into a spatiotemporally sparse conditional label sequence, and the noise target label sequence of the source video to be edited is obtained; the noise target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension, and then fed into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

[0068] The video effects editing system based on the large-scale video model described above is an implementation method corresponding to the video effects editing method based on the large-scale video model described above. Its specific execution steps can be referred to in the specific implementation of the video effects editing method based on the large-scale video model described above, and will not be described in detail here.

[0069] Figure 5 and Figure 6 The effect of video effects editing based on the text-based video large model of the present invention is shown. Among them, Figure 5 This is a video editing example demonstrating the application of the video effects editing method of the present invention. Figure 6 This diagram illustrates a comparison of the effects of applying the present invention and existing video editing methods.

[0070] like Figure 5 As shown, the first row of each section is the source video, and the second row is the result of video effects editing using the video effects editing method and system based on the text-based video large model of this invention. Figure 6 The left side shows a comparison of general video editing, and the right side shows a comparison of video effects editing. The first row of images is the source video, the second to fifth rows are the results of other comparison methods, and the last row, "Ours," represents the results of video effects editing using the text-based video large model-based video effects editing method and system of this invention. Other comparison schemes are as follows: InsV2V: Paper Consistent video to-video transfer using synthetic dataset; InsViE: Paper Insvie-1m: Effective instruction-based video editing with elaborate dataset construction; VACE: Paper Vace: All-in-one video creation and editing; Lucy Edit: Paper Lucy edit: Open-weight text-guided video editing..

[0071] like Figure 4 As shown, the present invention also provides an electronic device 1 for implementing a video effects editing method based on a large model of textual video.

[0072] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a video effects editing program 12 based on a Wensheng video model. The memory 11 may include both internal storage units of the aforementioned video effects editing system and / or the video effects editing system based on a Wensheng video model, and external storage devices. The memory 11 can be used not only to store application software and various types of data installed, such as the code of the video effects editing program based on a Wensheng video model, but also to temporarily store data that has been output or will be output.

[0073] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for video effects editing programs based on the Wensheng video large model, but also to temporarily store data that has been output or will be output.

[0074] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., video effects editing programs based on a large-scale video model), and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0075] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0076] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0077] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management system, thereby enabling functions such as charging management, discharging management, and power consumption management through the power management system. The power supply may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0078] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0079] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0080] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0081] The video effects editing program 12 based on the Wensheng video large model stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When it runs in the processor 10, it can realize the video effects editing method based on the Wensheng video large model as described above.

[0082] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 2The descriptions of the relevant steps in the corresponding embodiments are not repeated here. Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0083] This invention also provides a computer-readable storage medium, which may be non-volatile or volatile, storing a computer program that, when executed by a processor, implements the video effects editing method based on the Wensheng video large model as described above.

[0084] Specifically, the specific implementation method of the computer program when executed by the processor can be referred to the description of the relevant steps in the video special effects editing method based on the large model of textual video in the embodiment, and will not be repeated here.

[0085] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0086] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0088] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware.

[0089] The video effects editing method and system based on a large-scale video model according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the video effects editing method and system based on a large-scale video model proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the contents of the appended claims.

Claims

1. A video special effects editing method based on a large-scale textual video model, characterized in that, It includes the following three stages: Spatiotemporal sparse conditional label sequence construction stage: Based on the source video, obtain the spatial sparse label sequence and the temporal sparse label sequence, and concatenate the spatial sparse label sequence and the temporal sparse label sequence along the label dimension to obtain the spatiotemporal sparse conditional label sequence; Two-stage fine-tuning stage; This includes: obtaining target videos corresponding to the source videos based on a preset target video database, obtaining target marker sequences of the target videos using a pre-trained video encoder, adding noise to the target marker sequences to create noisy target marker sequences, thereby fine-tuning the preset raw video model into a general video editing model for performing general video editing tasks; and using video effect editing data containing paired visual effect examples, fine-tuning the general video editing model for performing general video editing tasks based on a preset EffectLoRA module, thereby giving the general video editing model editing modes for specific visual effects. The inference phase includes: constructing a spatiotemporally sparse conditional label sequence of the source video to be edited, and obtaining a noise target label sequence of the source video to be edited; concatenating the noise target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited along the label dimension, and feeding them into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

2. The video special effects editing method based on a large-scale textual video model as described in claim 1, characterized in that, The acquisition of spatially sparse and temporally sparse labeled sequences based on source video includes: Spatial downsampling and first frame extraction are performed on the source video to obtain low-resolution video and first frame; The first frame is encoded as a spatially sparse marker sequence, and the downsampled low-resolution video is encoded as a temporally sparse marker sequence.

3. The video effects editing method based on a large-scale textual video model as described in claim 1, characterized in that, The acquisition of spatially sparse and temporally sparse labeled sequences based on source video includes: The source video is encoded using 3D VAE to obtain a source marker sequence; Extract the marker corresponding to the first frame of the source marker sequence as a spatially sparse marker sequence; and perform sparse sampling on the source marker sequence in the spatial dimension to obtain a temporally sparse marker sequence.

4. The video special effects editing method based on a large-scale textual video model as described in claim 1, characterized in that, Before fine-tuning the preset text-based video model into a general video editing model for performing general video editing tasks in the two-stage fine-tuning stage, the method further includes adding position encoding correction and causal attention modules to the pre-trained text-based video model based on the spatiotemporal sparse conditional label sequence. The target marker sequence is subjected to noise processing to fine-tune the textual video model with positional encoding correction and causal attention modules into a general video editing model that can perform general video editing tasks.

5. The video effects editing method based on a large-scale textual video model as described in claim 4, characterized in that, Based on the spatiotemporally sparse conditional label sequence, positional encoding correction and causal attention modules are added to the pre-trained textual video large model, including: Position coding correction is inserted into the textual video large model to avoid spatiotemporal misalignment between the spatiotemporally sparse label sequence and the generation space; wherein, for the temporally sparse label sequence, a real correspondence between the temporally sparse label sequence and the target region is established to prevent spatial misalignment; for the spatially sparse label sequence, the position coding of the first frame of the target label sequence is used to avoid temporal errors. By designing an attention mask, the bidirectional attention mechanism of the textual video large model is changed to a causal attention mechanism to avoid the influence of noisy target label sequences on the temporally sparse conditional label sequence.

6. The video effects editing method based on a large-scale textual video model as described in claim 5, characterized in that, The attention mask design is as follows: in, This represents the mask at row index i and column index j. This represents the sequence of noise target markers.

7. The video special effects editing method based on a large-scale textual video model as described in claim 6, characterized in that, The target marker sequence is subjected to noise processing to fine-tune the text-based video model with positional encoding correction and causal attention modules into a general video editing model that performs general video editing tasks, including: The target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension and fed into the pre-trained textual video large model; Noise is added only to the target labeled sequence, while the spatiotemporally sparse conditional labeled sequence remains unchanged; the loss function used is as follows: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, It is a time-sparse labeled sequence. It is a spatially sparse labeled sequence. The velocity field predicted by the large model for the video. For the target vector field, Indicates noise.

8. The video special effects editing method based on a large-scale textual video model as described in claim 4, characterized in that, The fine-tuning of the general video editing model for performing general video editing tasks based on the preset EffectLoRA module includes: Insert a low-rank LoRA into the general video editing model that performs the general video editing task; The target label sequence and the spatiotemporally sparse conditional label sequence of the source video to be edited are concatenated along the label dimension and fed into a general video editing model with low-rank LoRA inserted; Keeping the spatiotemporally sparse conditional label sequence unchanged, noise is added only to the target label sequence, using the following loss function: in, Indicates the conditional probability path at time step t. and The expected value under, For the target labeled sequence, For time steps, For text instructions, It is a time-sparse labeled sequence. It is a spatially sparse labeled sequence. The velocity field predicted by the large model for the video. For the target vector field, Indicates noise.

9. A video special effects editing system based on a large-scale video model, applied to electronic devices, characterized in that, include: A spatiotemporal sparse conditional label sequence construction unit is used to obtain spatial sparse label sequences and temporal sparse label sequences based on the source video, and to concatenate the spatial sparse label sequences and the temporal sparse label sequences along the label dimension to obtain a spatiotemporal sparse conditional label sequence; a two-stage fine-tuning unit is used to obtain a target video corresponding to the source video based on a preset target video database, and to obtain a target label sequence of the target video using a pre-trained video encoder, and to add noise to the target label sequence to obtain a noisy target label sequence, so as to fine-tune the preset text-based video model into a general video editing model for performing general video editing tasks; and, using video effect editing data containing paired visual effect examples, to fine-tune the general video editing model for performing general video editing tasks based on a preset EffectLoRA module, so as to give the general video editing model a specific visual effect editing mode; The inference unit is used to construct a spatiotemporally sparse conditional label sequence from the source video to be edited, and to obtain the noise target label sequence of the source video to be edited; The noise target marker sequence and the spatiotemporally sparse conditional marker sequence of the source video to be edited are concatenated along the marker dimension and fed into a general editing model with EffectLoRA to inject visual effects into the source video to be edited.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a video effects editing program based on a large-scale video model stored in the memory and executable on the processor. When the large-scale video effects editing program is executed by the processor, it implements the video effects editing method based on a large-scale video model as described in claims 1 to 8.

Images