Video-oriented intelligent animation generation method

Abstract

The invention discloses a video-oriented intelligent animation generation method, and the model of the intelligent animation generation method employs an encoder and decoder architecture, and comprises a shallow feature extraction module which is used for carrying out the feature extraction of an input video sequence and a style reference image, and obtaining a video shallow feature and a style shallow feature; and the feature embedding and coding module is used for carrying out block linear embedding on the video shallow layer features, adding space-time position codes and coding the style shallow layer features to obtain style feature vectors. According to the invention, through the space-time dynamic attention path in the STSFA module, the combined modeling of the space structure and the time dynamic is realized, the long-time-history space-time dependency relationship in the video can be accurately captured, and for the shielding scene, the dynamic state of the shielded area can be deduced through the motion trail of the adjacent frames.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, computer vision, and computer graphics, specifically to an intelligent animation generation method for video. Background Technology

[0002] With the explosive growth in demand for digital content creation, efficiently converting real-world videos into artistic animations has become a core technological challenge for industries such as film, gaming, and short videos. Existing technological solutions mainly suffer from three major bottlenecks: First, poor dynamic consistency: Convolutional Neural Network (CNN)-based solutions struggle to model long-term temporal dependencies, leading to flickering and stuttering between animation frames. While Recurrent Neural Network (RNN)-based solutions can handle temporal dependencies, they suffer from poor parallelism and the vanishing gradient problem over long distances.

[0003] Second, the style adaptability is weak: existing methods often treat style transfer and motion capture as two independent modules connected in series (for example, first perform frame-by-frame stylization, and then perform temporal smoothing), which leads to the stylization effect being disconnected from the motion logic. For example, the style lines in fast-moving areas cannot reflect the dynamic tension.

[0004] Third, it has low computational efficiency: the computational complexity of the standard Transformer's self-attention mechanism is quadratic with the sequence length, making it difficult to handle high-resolution, long-sequence video inputs and unable to meet the needs of real-time or near-real-time applications. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent animation generation method for video, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a smart animation generation method for video, wherein the smart animation generation method model adopts an encoder-decoder architecture, comprising: The shallow feature extraction module is used to extract features from the input video sequence and style reference image respectively, to obtain shallow video features and shallow style features; The feature embedding and encoding module is used to perform block-based linear embedding of shallow video features and add spatiotemporal location encoding, as well as to encode the shallow style features to obtain a style feature vector. The hierarchical spatiotemporal style fusion encoder consists of multiple cascaded encoding stages. Each encoding stage includes at least one spatiotemporal style fusion attention STSFA module and a feature downsampling layer. The STSFA module is used to perform spatiotemporal dynamic attention calculation and style feature attention calculation in parallel, and outputs the fused features through an adaptive fusion mechanism. The decoder is used to upsample the fusion features output by the hierarchical spatiotemporal-style fusion encoder to generate an animation feature map; The style rendering layer is used to perform stylized rendering of the animation feature map based on the style feature vector, and output the final stylized animation sequence.

[0007] Preferably, the STSFA module includes: The spatiotemporal dynamic attention path is used to perform window self-attention calculations in the spatial dimension and the temporal dimension on the input feature sequence to capture spatiotemporal dynamic information in the video. The style feature attention path is used to map the style feature vector into attention weights, modulate the features processed by the spatiotemporal dynamic attention path, and realize the adaptive interaction between style features and dynamic features. The multimodal adaptive fusion module is used to perform learnable weighted fusion of the output features of the spatiotemporal dynamic attention path and the style feature attention path to generate the fused features.

[0008] Preferably, the spatial dimension window self-attention calculation in the spatiotemporal dynamic attention path adopts a movable window mechanism to achieve cross-window information interaction.

[0009] Preferably, the style feature attention path achieves style modulation through the following steps: The style feature vector is mapped to a style weight vector through a linear layer; the style weight vector is multiplied element-wise with the input features after layer normalization to obtain preliminary modulation features; the similarity matrix between the preliminary modulation features and the style feature vector is calculated to obtain style attention weights; the preliminary modulation features are weighted and aggregated using the style attention weights.

[0010] Preferably, the multimodal adaptive fusion module is implemented through a two-layer neural network. The input of the neural network is the concatenation of the output features of the spatiotemporal dynamic attention path and the style feature attention path, and its output is the fusion weight, which is used to perform weighted summation of the two features.

[0011] Preferably, the model is trained using a composite loss function, which includes dynamic consistency loss, style transfer loss, and generation quality loss.

[0012] Preferably, the dynamic consistency loss includes joint consistency loss and inter-frame smoothing loss; the style transfer loss is calculated based on the difference in feature Gram matrices between the generated animation and the style reference image.

[0013] Preferably, the generated quality loss includes adversarial loss and L1 reconstruction loss.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This intelligent animation generation method for videos achieves joint modeling of spatial structure and temporal dynamics through the spatiotemporal dynamic attention path within the STSFA module. It can accurately capture long-term spatiotemporal dependencies in videos. For occluded scenes, the dynamics of the occluded area can be inferred from the motion trajectories of adjacent frames. For fast-moving scenes, it can effectively suppress interference caused by motion blur, ensuring the continuity and physical rationality of the animation motion. The style feature attention path and adaptive fusion mechanism achieve deep adaptation between style features and dynamic information, support diverse style transfer, and can dynamically adjust style parameters according to the motion state, so that the style expression is highly matched with the motion logic. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the process structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0018] In the description of this patent, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "setting" should be interpreted broadly. For example, they can refer to a fixed connection or setting, a detachable connection or setting, or an integrated connection or setting. Those skilled in the art can understand the specific meaning of the above terms in this patent according to the specific circumstances.

[0019] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a number" means two or more, unless otherwise explicitly specified.

[0020] Please see Figure 1 As shown, this invention provides a smart animation generation method for video. This smart animation generation method model adopts an encoder-decoder architecture, including: The shallow feature extraction module is used to extract features from the input video sequence and style reference image respectively, to obtain shallow video features and shallow style features; The feature embedding and encoding module is used to perform block-based linear embedding of shallow video features and add spatiotemporal location encoding, as well as to encode the shallow style features to obtain a style feature vector. The hierarchical spatiotemporal style fusion encoder consists of multiple cascaded encoding stages. Each encoding stage includes at least one spatiotemporal style fusion attention STSFA module and a feature downsampling layer. The STSFA module is used to perform spatiotemporal dynamic attention calculation and style feature attention calculation in parallel, and outputs the fused features through an adaptive fusion mechanism. The decoder is used to upsample the fusion features output by the hierarchical spatiotemporal-style fusion encoder to generate an animation feature map; The style rendering layer is used to perform stylized rendering of the animation feature map based on the style feature vector, and output the final stylized animation sequence.

[0021] The STSFA module includes: The spatiotemporal dynamic attention path is used to perform window self-attention calculations in the spatial dimension and the temporal dimension on the input feature sequence to capture spatiotemporal dynamic information in the video. The style feature attention path is used to map the style feature vector into attention weights, modulate the features processed by the spatiotemporal dynamic attention path, and realize the adaptive interaction between style features and dynamic features. The multimodal adaptive fusion module is used to perform learnable weighted fusion of the output features of the spatiotemporal dynamic attention path and the style feature attention path to generate the fused features.

[0022] The spatial dimension window self-attention calculation in the spatiotemporal dynamic attention path adopts a movable window mechanism to achieve cross-window information interaction.

[0023] The style feature attention path achieves style modulation through the following steps: The style feature vector is mapped to a style weight vector through a linear layer; the style weight vector is multiplied element-wise with the input features after layer normalization to obtain preliminary modulation features; the similarity matrix between the preliminary modulation features and the style feature vector is calculated to obtain style attention weights; the preliminary modulation features are weighted and aggregated using the style attention weights.

[0024] The multimodal adaptive fusion module is implemented through a two-layer neural network. The input of the neural network is the concatenation of the output features of the spatiotemporal dynamic attention path and the style feature attention path, and its output is the fusion weight, which is used to perform weighted summation of the two features.

[0025] The model is trained using a composite loss function, which includes dynamic consistency loss, style transfer loss, and generation quality loss.

[0026] The dynamic consistency loss includes joint consistency loss and inter-frame smoothing loss; the style transfer loss is calculated based on the difference in the feature Gram matrix between the generated animation and the style reference image.

[0027] The generated quality loss includes adversarial loss and L1 reconstruction loss.

[0028] The workflow of this model is as follows: Input: Input an RGB video containing T frames (e.g., 256x256 resolution) and a style reference image (e.g., a cartoon image); Shallow feature extraction: Shallow features of video frames and images are extracted using MobileNetV3. Feature embedding and encoding: Video features are segmented and embedded, and learnable positional encodings are added. Style features are encoded into vectors through fully connected layers; Hierarchical encoding: The embedded feature input consists of an encoder consisting of 4 stages, each stage containing 2-6 STSFA modules and a downsampling layer. During the encoding process, the spatiotemporal information and style information are continuously and deeply fused by the STSFA modules. Decoding and rendering: The encoded features are upsampled to the original resolution by the decoder, and then the final stylized animation sequence is output through the style rendering layer.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for video-oriented intelligent animation generation, characterized in that: The intelligent animation generation method model adopts an encoder and a decoder architecture, and comprises: a shallow feature extraction module, configured to respectively perform feature extraction on an input video sequence and a style reference image to obtain video shallow features and style shallow features; a feature embedding and encoding module, configured to perform block linear embedding on the video shallow features and add spatial-temporal position encoding, and encode the style shallow features to obtain a style feature vector; a hierarchical spatial-temporal and style fusion encoder, which is composed of a plurality of cascaded encoding stages, each encoding stage comprising at least one spatial-temporal and style fusion attention (STSFA) module and one feature down-sampling layer; the STSFA module is configured to perform parallel spatial-temporal dynamic attention calculation and style feature attention calculation, and output fused features through an adaptive fusion mechanism; a decoder, configured to up-sample the fused features output by the hierarchical spatial-temporal and style fusion encoder to generate an animation feature map; a style rendering layer, configured to perform stylized rendering on the animation feature map according to the style feature vector to output a final stylized animation sequence.

2. The method of claim 1, wherein: The STSFA module comprises: a spatial-temporal dynamic attention path, configured to perform window self-attention calculation in the spatial dimension and window self-attention calculation in the temporal dimension on an input feature sequence to capture spatial-temporal dynamic information in the video; a style feature attention path, configured to map the style feature vector to an attention weight, modulate the features processed by the spatial-temporal dynamic attention path, and realize adaptive interaction between the style feature and the dynamic feature; a multi-modal adaptive fusion module, configured to perform learnable weighted fusion on the output features of the spatial-temporal dynamic attention path and the style feature attention path to generate the fused features.

3. The method of claim 1, wherein: The window self-attention calculation in the spatial dimension in the spatial-temporal dynamic attention path adopts a movable window mechanism to realize information interaction across windows.

4. The method of claim 3, wherein: The style feature attention path realizes style modulation through the following steps: mapping the style feature vector to a style weight vector through a linear layer; and performing element-wise multiplication between the style weight vector and the input features after layer normalization to obtain preliminary modulation features; calculating a similarity matrix of the preliminary modulation features and the style feature vector to obtain a style attention weight; and performing weighted aggregation on the preliminary modulation features using the style attention weight.

5. The method of claim 2, wherein: The multi-modal adaptive fusion module is realized through a two-layer neural network, the input of the neural network being a concatenation of the output features of the spatial-temporal dynamic attention path and the style feature attention path, and the output of the neural network being a fusion weight used for weighted summation of the two features.

6. The method of claim 1, wherein: The model training adopts a composite loss function, and the composite loss function comprises a dynamic consistency loss, a style transfer loss, and a generation quality loss.

7. The method of claim 1, wherein: The dynamic consistency loss comprises a joint point consistency loss and an inter-frame smoothness loss; and the style transfer loss is calculated based on the feature Gram matrix difference between the generated animation and the style reference image.

8. The method of claim 1, wherein: The generation quality loss comprises an adversarial loss and an L1 reconstruction loss.

Images