Special effect video generation method and system

Abstract

The invention relates to a special effect video generation method and system, and the method comprises the following steps: S1, carrying out video content understanding based on a Vision-Language Model, inputting an original video frame sequence to the Vision-Language Model, and outputting a structured content label and a special effect content candidate list to the Vision-Language Model; s2, performing special effect style injection on the video generation model by adopting a LoRA fine tuning technology so as to realize personalized and content-driven special effect animation generation; and S3, fusing the special effect layer output by the video generation model with the original video frame to generate a final special effect video with visual impact, natural transition sense and dynamic consistency. According to the invention, the video content can be automatically identified and the special effect video conforming to the video content is generated.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for generating special effects videos. Background Technology

[0002] Although short video platforms and editing tools with video effects functions have emerged in the market, such as Douyin, Meitu Xiu Xiu, CapCut, and Jianying, these products still have many shortcomings in video effects generation, mainly in the following aspects:

[0003] It relies heavily on templates and manual editing, lacking intelligent features.

[0004] Most mainstream video effects production tools currently rely on preset templates. Users need to manually select templates and adjust parameters, which is complex, inefficient, and difficult to meet users' needs for fast and personalized creation.

[0005] For novice users, understanding and using these templates presents a certain learning curve, and manual editing makes it difficult to create high-quality dynamic effects in a short time. Templates often suffer from poor matching and material compatibility, lacking flexibility. Templates are typically statically constructed and cannot be flexibly adjusted according to different scenes, character actions, or emotions in the video, resulting in a lack of semantic consistency and visual harmony between effects and content.

[0006] When video content is complex or changes rapidly, traditional templates often fail to adaptively generate effects, easily resulting in distortions such as "misalignment" and "clipping." They lack deep content understanding and semantically driven effect generation.

[0007] Traditional video effects systems often process low-level image features (such as color and outlines), lacking an understanding of semantic information in the video (such as character identity, emotional state, and scene category), making it difficult to generate effects that are highly relevant to the content. This limits the system's potential for expansion in terms of automation, aesthetics, and expressiveness.

[0008] Weak dynamic coherence processing capability:

[0009] Most existing methods fail to consider the semantic and visual coherence between consecutive frames when processing video frame by frame, which can easily cause special effects to "jump" or "drift" on the timeline, affecting the viewing experience.

[0010] In practical applications, this timing inconsistency can severely reduce the professionalism and visual appeal of video effects.

[0011] The algorithm has poor generalization ability:

[0012] Current special effects generation systems often rely heavily on specific scenes and input video types, lacking cross-domain and cross-style transfer and adaptation capabilities, making it difficult to meet users' diverse and personalized creative needs.

[0013] Therefore, it is necessary to provide a method and system for generating special effects videos, so as to automatically identify video content and generate special effects videos that conform to the video content. Summary of the Invention

[0014] The purpose of this invention is to provide a method and system for generating special effects videos, which can automatically identify video content and generate special effects videos that conform to the video content.

[0015] To address the problems existing in the prior art, this invention provides a method for generating special effects videos, comprising the following steps:

[0016] S1: Based on the Vision-Language Model, video content understanding is performed. The input to the Vision-Language Model is the original video frame sequence, and the output of the Vision-Language Model is a structured content tag and a candidate list of special effects content.

[0017] S2: LoRA fine-tuning technology is used to inject special effects styles into the video generation model to achieve personalized, content-driven special effects animation generation;

[0018] S3: The effects layer output by the video generation model is merged with the original video frames to generate a final effects video with visual impact, natural transitions, and dynamic consistency.

[0019] Optionally, in the method for generating the special effects video, the abbreviation for Vision-Language Model is VLM, which stands for Multimodal Vision-Language Model.

[0020] Optionally, in the method for generating the special effects video, the steps for understanding the video content based on the Vision-Language Model are as follows:

[0021] Frame-level sampling is performed on the video, and every N frames are extracted into a keyframe sequence;

[0022] Keyframes are input into the VLM, and a set of multimodal embeddings is generated by combining the temporal context of the video with the prompts input by the user.

[0023] VLM performs semantic analysis on multimodal embeddings to identify semantic features in the video, including subject, scene, action, and emotion.

[0024] The system has a built-in special effects content matching strategy library. By mapping keywords to special effects templates and combining video semantic information, it automatically determines the appropriate special effects type.

[0025] Optionally, in the method for generating the special effects video, LoRA is an abbreviation for Low-Rank Adaptation.

[0026] Optionally, in the method for generating special effects videos, the video generation model is customized based on existing diffusion-type video generation models, and the input of the video generation model consists of text prompts, video frame sequences, and special effects semantics to be injected.

[0027] Optionally, in the method for generating special effects videos, in S2, after receiving the special effects tags or prompts output by the video content understanding module, the large-scale pre-trained video generation model is quickly adapted through efficient fine-tuning to generate video special effects materials that conform to the semantic scene.

[0028] Optionally, in the method for generating the special effects video, LoRA is used to train the target special effects, and the steps are as follows:

[0029] Prepare multiple training images or short video clips that represent the style of special effects;

[0030] With the main model parameters fixed, only the LoRA insertion module is trained to generate lightweight style adaptation parameters;

[0031] Each effect type corresponds one-to-one with a LoRA weight package, and only the relevant LoRA module needs to be loaded during the generation stage;

[0032] During inference, the semantic content of the original input video is used as the main prompt word, and the LoRA module is responsible for injecting special effects style into the generated image, realizing the decoupling and fusion of semantics and style.

[0033] Optionally, in the method for generating the special effects video, S3 combines black cutout and transparent blending.

[0034] Optionally, in the method for generating the special effects video, the fusion processing steps are as follows:

[0035] The RGB values ​​of the special effects image are analyzed pixel by pixel, a threshold is set, and the background is determined based on the threshold.

[0036] Set the transparency to 0 for pixels that meet the criteria;

[0037] Set up an effects layer that includes an alpha channel for later compositing;

[0038] For special effects footage that already has an alpha channel, the standard Alpha Blending method is used for compositing.

[0039] This invention also provides a system for generating special effects videos, the system being formed according to the method described above, the system comprising:

[0040] The video content understanding module is configured to perform video content understanding based on the Vision-Language Model. The input to the Vision-Language Model is the original video frame sequence, and the output of the Vision-Language Model is a structured content tag and a candidate list of special effects content.

[0041] The special effects generation module is configured to use LoRA fine-tuning technology to inject special effects styles into the video generation model in order to achieve personalized, content-driven special effects animation generation.

[0042] The special effects compositing module is configured to merge the special effects layers output by the video generation model with the original video frames to generate a final special effects video with visual impact, natural transitions, and dynamic consistency.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] (1) Intelligent content understanding: Through VLM, a deeper understanding of video semantics is achieved, making the selection of special effects more context-sensitive;

[0045] (2) High-efficiency special effects generation: LoRA technology is used to achieve rapid fine-tuning, which greatly reduces the generation cost and supports high-precision style control;

[0046] (3) Natural visual integration: The efficient compositing method of black cutout + transparency blending is adopted, resulting in a natural effect and good edge blending;

[0047] (4) High scalability: The entire process supports modular replacement, and VLM or video generation models can be flexibly deployed according to different platforms;

[0048] (5) User-friendly interaction: Supports prompts and enhances personalized control experience by combining VLM judgment. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation

[0050] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0051] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0052] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0053] Currently, the optimization of diffusion models increasingly relies on high-quality human preference data to improve the aesthetic quality, semantic consistency, and user satisfaction of generated images. However, existing human preference datasets are mainly generated by manual annotation or large-scale multimodal models (MLLMs), but these methods still have many limitations:

[0054] To address the problems existing in the prior art, this invention provides a method for generating special effects videos.

[0055] By introducing a visual-language multimodal model (VLM) to intelligently analyze video content and combining it with LoRA low-rank adaptation technology to train the video generation model in a lightweight manner, special effects synthesis that highly matches the video content can be achieved. This is applicable to various scenarios such as short video editing and digital human special effects.

[0056] like Figure 1 As shown, the method includes the following steps:

[0057] S1: The video content understanding module in this invention aims to automatically determine the appropriate special effects type in the video, and is a crucial step in the entire special effects generation system. Video content understanding is based on the Vision-Language Model (using Alibaba QWen-VL in this invention). The input to the Vision-Language Model is the original video frame sequence, and the output is a structured content tag and a candidate list of special effects content. The abbreviation for Vision-Language Model is VLM, which stands for Multimodal Visual-Language Model.

[0058] Specifically, the steps for video content understanding based on the Vision-Language Model are as follows:

[0059] Frame-level sampling is performed on the video, and every N frames are extracted into a keyframe sequence;

[0060] Keyframes are input into the VLM, and combined with the temporal context of the video and the prompts input by the user, a set of multimodal embeddings is generated.

[0061] VLM performs semantic analysis on multimodal embeddings to identify semantic features in the video. Semantic features include subjects (such as people, animals, and objects), scenes (such as night, forest, and science fiction world), actions (such as jumping, running, and waving), and emotions (such as intense, romantic, and mysterious).

[0062] The system has a built-in special effects content matching strategy library. By mapping keywords to special effects templates and combining video semantic information, it automatically determines the appropriate special effects type. For example, when the model recognizes "a person waving + a science fiction scene", it can recommend generating special effects such as "energy fluctuation" or "beam emission"; when it recognizes "intense running + a fire scene", it can associate dynamic effects such as "flame trajectory" and "explosion dust".

[0063] In addition, this step supports interactive guidance from users, who can enhance the accuracy of semantic recognition and special effects matching by entering keywords or prompts (such as "add cyber style" or "cool entrance"), thereby achieving flexible switching between semi-automatic and fully automatic modes.

[0064] The above methods provide high-quality semantic instruction input for subsequent special effects generation, significantly improving the relevance and visual consistency between special effects and content.

[0065] S2: This step employs LoRA (Low-Rank Adaptation) fine-tuning technology to inject special effects styles into the video generation model, enabling personalized, content-driven special effects animation generation. After receiving special effects tags or prompts from the video content understanding module, this step uses efficient fine-tuning to quickly adapt the large-scale pre-trained video generation model, thereby generating video special effects materials that conform to the semantic scene. LoRA is a parameter-efficient fine-tuning method that achieves the transfer and control of special effects styles by inserting low-rank matrix branches at specific locations in the original model (such as the Query / Key linear layer in a Transformer or a convolutional block in a UNet).

[0066] Furthermore, the video generation model is customized based on existing diffusion-based video generation models (such as Wan2.1, HunyuanVideo, etc.). The input to the video generation model consists of a text prompt, a video frame sequence, and the semantic effects to be injected. For example, when the video content tag is "person running + fire effect", the model input will include a "running person" frame sequence and the prompt: a person running surrounded by fire trail, cinematic, glowing.

[0067] Furthermore, LoRA is used to train the target effect, and the steps are as follows:

[0068] Prepare multiple training images or short video clips that represent special effects styles (such as flame eruptions, lightning explosions, neon lines, etc.);

[0069] With the main model parameters fixed, only the LoRA insertion module is trained to generate lightweight style adaptation parameters;

[0070] The special effects types correspond one-to-one with the LoRA weight packs. During the generation stage, only the relevant LoRA modules need to be loaded to quickly switch between different special effects styles.

[0071] During inference, the semantic content of the input original video is used as the main prompt word, and the LoRA module is responsible for injecting special effects styles into the generated image, thereby achieving decoupling and fusion of semantics and style.

[0072] S3: The effects layer output by the video generation model is merged with the original video frames to generate a final effects video with visual impact, natural transitions, and dynamic consistency.

[0073] In the video effects generation system of this invention, this step is responsible for fusing the effects layer output by the video generation model with the original video frames to generate a final effects video with strong visual impact, natural transitions, and dynamic consistency. This module combines two methods: "black keying" and "alpha transparency blending," balancing compositing efficiency and visual quality.

[0074] Black masking is a background removal technique suitable for special effects images output by generative models where black is typically used as the background color. This method identifies black areas in the special effects image by setting a pixel threshold and makes them transparent, retaining only brighter foreground areas such as glowing lights, flames, lightning, and particles.

[0075] Further, the fusion processing steps are as follows:

[0076] The RGB values ​​of the special effects image are analyzed pixel by pixel, a threshold is set, and the background is determined to be black based on the threshold (e.g., RGB < [20, 20, 20]).

[0077] For pixels that meet the criteria, set the transparency to 0 (i.e., alpha = 0); otherwise, the alpha is linearly mapped based on the difference in brightness or distance from black (to enhance the smoothness of the blended edges).

[0078] Set up an effects layer (RGBA format) that includes an alpha channel for later compositing;

[0079] For effects footage that already has an alpha channel (such as RGBA frames directly generated from a model), the standard alpha blending method is used for compositing. The core idea is to control the pixel-level blending ratio through the transparency of the effects layer, ensuring that the effects are naturally superimposed on the original video without obvious boundaries or tearing. The alpha value can come from the channels of the generated image, preset fade-in / fade-out control curves, or be manually set by the user (e.g., "effect strength = 0.8").

[0080] This invention also provides a system for generating special effects videos, the system being formed according to the method described above, the system comprising:

[0081] The video content understanding module is configured to perform video content understanding based on the Vision-Language Model. The input to the Vision-Language Model is the original video frame sequence, and the output of the Vision-Language Model is a structured content tag and a candidate list of special effects content.

[0082] The special effects generation module is configured to use LoRA fine-tuning technology to inject special effects styles into the video generation model in order to achieve personalized, content-driven special effects animation generation.

[0083] The special effects compositing module is configured to merge the special effects layers output by the video generation model with the original video frames to generate a final special effects video with visual impact, natural transitions, and dynamic consistency.

[0084] Specifically, (1) Video content understanding module (special effects content judgment):

[0085] The video content understanding module in this invention aims to automatically determine the appropriate special effects type in a video, serving as a crucial pre-processor and key component of the entire special effects generation system. This module is based on a multimodal vision-language model (VLM), specifically Alibaba's QWen-VL, taking the original video frame sequence as input and outputting structured content tags and a candidate list of special effects content.

[0086] First, the system performs frame-level sampling on the video, extracting a keyframe sequence from every N frames. Then, the keyframes are input into the VLM model, which, combined with the video's temporal context and potential user prompts, generates a set of multimodal embeddings. The model performs semantic analysis on these embeddings, identifying semantic features such as the subject (e.g., people, animals, objects), scene (e.g., night, forest, science fiction world), actions (e.g., jumping, running, waving), and emotions (e.g., intense, romantic, mysterious) within the video.

[0087] Building upon this foundation, the system incorporates a library of special effects content matching strategies. By mapping keywords to special effects templates and combining this with video semantic information, it automatically determines the appropriate type of special effects. For example, when the model identifies "a person waving + a sci-fi scene," it can recommend generating special effects such as "energy fluctuations" or "beam emission." When it identifies "intense running + a fire scene," it can associate it with dynamic effects such as "flame trails" or "explosion dust."

[0088] In addition, the module supports interactive user guidance, meaning that users can enhance the accuracy of semantic recognition and special effects matching by inputting keywords or prompts (such as "add cyber style" or "cool entrance"), thereby achieving flexible switching between semi-automatic and fully automatic modes.

[0089] Through the above methods, the video content understanding module provides high-quality semantic instruction input for subsequent special effects generation, significantly improving the correlation and visual consistency between special effects and content.

[0090] (2) Special Effects Generation Module (LoRA-based Special Effects Fine-tuning)

[0091] We referenced relevant research and employed LoRA (Low-Rank Adaptation) fine-tuning technology to inject special effects styles into the video generation model, enabling personalized, content-driven special effects animation generation. After receiving special effects tags or Prompt instructions from the video content understanding module, this module rapidly adapts the large-scale pre-trained video generation model through efficient fine-tuning, thereby generating video special effects materials that conform to the semantic scene.

[0092] Custom development is possible based on existing diffusion-based video generation models (such as Wan2.1, HunyuanVideo, etc.). The model input consists of a text prompt, a video frame sequence, and the semantic effects to be injected. For example, when the video content tag is "person running + fire effects," the model input will include a "running person" frame sequence and the prompt: a person running surrounded by fire trail, cinematic, glowing.

[0093] LoRA is a parameter-efficient fine-tuning method that achieves the transfer and control of special effects styles by inserting low-rank matrix branches at specific locations in the original model (such as the Query / Key linear layer in Transformer or the convolutional block in UNet).

[0094] This invention utilizes LoRA to train target effects, and the process is as follows:

[0095] Prepare several training images or short video clips that represent special effects styles (such as flame eruptions, lightning explosions, neon lines, etc.);

[0096] With the main model parameters fixed, only the LoRA insertion module is trained to generate lightweight style adaptation parameters;

[0097] The special effects types correspond one-to-one with the LoRA weight packs. During the generation stage, only the relevant LoRA modules need to be loaded to quickly switch between different special effects styles.

[0098] During inference, the semantic content of the original input video serves as the main prompt, while the LoRA module is responsible for injecting special effects styles into the generated image, achieving decoupling and fusion of semantics and style.

[0099] (3) Special Effects Compositing Module (Black Extraction + Alpha Fusion)

[0100] In the video effects generation system of this invention, the effects compositing module is responsible for fusing the effects layer output by the video generation model with the original video frames to generate a final effects video with strong visual impact, natural transitions, and dynamic consistency. This module adopts a combination of "black keying" and "alpha transparency blending," balancing compositing efficiency and visual quality.

[0101] Black masking is a background removal technique suitable for special effects images output by generative models where black is typically used as the background color. This method identifies black areas in the special effects image by setting a pixel threshold and makes them transparent, retaining only brighter foreground areas such as glowing lights, flames, lightning, and particles.

[0102] The processing flow is as follows:

[0103] 1. Analyze the RGB values ​​of the special effects image pixel by pixel, and set a threshold (e.g., RGB < [20, 20, 20]) to determine whether it is a black background;

[0104] 2. Set the transparency to 0 for pixels that meet the conditions (i.e., alpha = 0); otherwise, linearly map alpha based on the difference between brightness and black (to enhance the smoothness of blended edges).

[0105] 3. Effects layers (RGBA format) containing alpha channels for subsequent compositing.

[0106] For effects footage that already has an alpha channel (such as RGBA frames directly generated from a model), the system uses standard alpha blending for compositing. The core idea is to control the pixel-level blending ratio through the transparency of the effects layer, ensuring that the effects are naturally superimposed on the original video without obvious boundaries or tearing. The alpha value can come from the channels of the generated image, preset fade-in / fade-out control curves, or be manually set by the user (e.g., "effect strength = 0.8").

[0107] Compared with the prior art, the present invention has the following advantages:

[0108] (1) Intelligent content understanding: Through VLM, a deeper understanding of video semantics is achieved, making the selection of special effects more context-sensitive;

[0109] (2) High-efficiency special effects generation: LoRA technology is used to achieve rapid fine-tuning, which greatly reduces the generation cost and supports high-precision style control;

[0110] (3) Natural visual integration: The efficient compositing method of black cutout + transparency blending is adopted, resulting in a natural effect and good edge blending;

[0111] (4) High scalability: The entire process supports modular replacement, and VLM or video generation models can be flexibly deployed according to different platforms;

[0112] (5) User-friendly interaction: Supports prompts and enhances personalized control experience by combining VLM judgment.

[0113] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for generating special effects videos, characterized in that, Includes the following steps: S1: Based on the Vision-Language Model, video content understanding is performed. The input to the Vision-Language Model is the original video frame sequence, and the output of the Vision-Language Model is a structured content tag and a candidate list of special effects content. S2: LoRA fine-tuning technology is used to inject special effects styles into the video generation model to achieve personalized, content-driven special effects animation generation; S3: The effects layer output by the video generation model is merged with the original video frames to generate a final effects video with visual impact, natural transitions, and dynamic consistency.

2. The method for generating special effects videos as described in claim 1, characterized in that, The abbreviation for Vision-Language Model is VLM, which stands for Multimodal Vision-Language Model.

3. The method for generating special effects videos as described in claim 2, characterized in that, The steps for video content understanding based on the Vision-LanguageModel are as follows: Perform frame-level sampling on the video and extract each N frames into a keyframe sequence; Keyframes are input into the VLM, and a set of multimodal embeddings is generated by combining the temporal context of the video with the prompts input by the user. VLM performs semantic analysis on multimodal embeddings to identify semantic features in the video, including subject, scene, action, and emotion. The system has a built-in special effects content matching strategy library. By mapping keywords to special effects templates and combining video semantic information, it automatically determines the appropriate special effects type.

4. The method for generating special effects videos as described in claim 1, characterized in that, LoRA is an abbreviation for Low-Rank Adaptation.

5. The method for generating special effects videos as described in claim 1, characterized in that, The video generation model is a customized development based on the existing diffusion-type video generation model. The input of the video generation model consists of text prompts, video frame sequences, and special effects semantics to be injected.

6. The method for generating special effects videos as described in claim 1, characterized in that, In S2, after receiving the special effects tags or prompts from the video content understanding module, the large-scale pre-trained video generation model is quickly adapted through efficient fine-tuning to generate video special effects materials that conform to the semantic scene.

7. The method for generating special effects videos as described in claim 6, characterized in that, The steps for training the target effect using LoRA are as follows: Prepare multiple training images or short video clips that represent the style of special effects; With the main model parameters fixed, only the LoRA insertion module is trained to generate lightweight style adaptation parameters; Each effect type corresponds one-to-one with a LoRA weight package, and only the relevant LoRA module needs to be loaded during the generation stage; During inference, the semantic content of the original input video is used as the main prompt word, and the LoRA module is responsible for injecting special effects style into the generated image, realizing the decoupling and fusion of semantics and style.

8. The method for generating special effects videos as described in claim 1, characterized in that, S3 uses a combination of black cutout and transparent blending.

9. The method for generating special effects videos as described in claim 8, characterized in that, The fusion process is as follows: The RGB values ​​of the special effects image are analyzed pixel by pixel, a threshold is set, and the background is determined based on the threshold. Set the transparency to 0 for pixels that meet the criteria; Set up an effects layer that includes an alpha channel for later compositing; For special effects footage that already has an alpha channel, the standard Alpha Blending method is used for compositing.

10. A system for generating special effects videos, characterized in that, The system is formed by the method according to any one of claims 1-9, the system comprising: The video content understanding module is configured to perform video content understanding based on the Vision-Language Model. The input to the Vision-Language Model is the original video frame sequence, and the output of the Vision-Language Model is a structured content tag and a candidate list of special effects content. The special effects generation module is configured to use LoRA fine-tuning technology to inject special effects styles into the video generation model in order to achieve personalized, content-driven special effects animation generation. The special effects compositing module is configured to merge the special effects layers output by the video generation model with the original video frames to generate a final special effects video with visual impact, natural transitions, and dynamic consistency.

Images