An animation generation method, apparatus, device, medium, and program product

Abstract

Embodiments of the present application provide an animation generation method and device, equipment, medium and program product, which are used for improving the capturing fineness of facial dynamic details, thereby improving the facial control accuracy of the portrait in the animation. It can be applied to the fields of computer, image processing, artificial intelligence, etc. Including: obtaining a reference video and a reference image; performing feature extraction on the pixel points in the reference video and the reference image to obtain a first feature representation; performing action feature extraction on the reference video to obtain a first action feature representation and a second action feature representation of the reference video, the first action feature representation being used for indicating the global action feature corresponding to the reference video, and the second action feature representation being used for indicating the facial action feature corresponding to the reference video; and performing decoding processing based on the first feature representation, the first action feature representation and the second action feature representation to obtain a simulation animation.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to an animation generation method, apparatus, device, medium, and program product. Background Technology

[0002] With the popularization of short videos and the rapid development of the animation industry, systems for intelligently generating animations based on videos and images have been widely used. In this process, users acquire images or videos containing human figures through a camera or other image capture tools, and then select an animation template (i.e., an animated video) to synthesize the human figures from the images or videos into virtual animations. For example, in one scenario, when a user is chatting with a friend through an instant messaging application, they can send emoticons to express their meaning; these emoticons can be generated based on the above method.

[0003] Current solutions typically control facial expressions by using explicit facial landmarks, but this not only leads to a loss of detail but also heavily relies on the accuracy of landmark extraction technology.

[0004] Therefore, how to improve the facial dynamic details and control precision of human figures in animation is an urgent problem that needs to be solved. Summary of the Invention

[0005] This application provides an animation generation method, apparatus, device, medium, and program product to improve the precision of capturing facial dynamic details, thereby improving the facial control accuracy of human figures in animation.

[0006] In view of this, this application provides an animation generation method, comprising: acquiring a reference video and a reference image, wherein the reference video is used to indicate the behavioral actions corresponding to a target object in a simulated animation, and the reference image is used to indicate the portrait information corresponding to the target object in the simulated animation; performing feature extraction on pixels in the reference video and the reference image to obtain a first feature representation, wherein the first feature representation is a latent feature of the reference video and the reference image; performing motion feature extraction on the reference video to obtain a first motion feature representation and a second motion feature representation of the reference video, wherein the first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video; and performing decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

[0007] Another aspect of this application provides an animation generation apparatus, comprising: an acquisition module for acquiring a reference video and a reference image, wherein the reference video is used to indicate the behavior and actions corresponding to a target object in a simulated animation, and the reference image is used to indicate the portrait information corresponding to the target object in the simulated animation;

[0008] The processing module is configured to extract features from pixels in the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; extract motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; and perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

[0009] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call a first variational network to extract features from the pixels of each video frame in the reference video to obtain a first pixel signal.

[0010] The second variational network is invoked to extract features from the pixels in the reference image to obtain the second pixel signal;

[0011] The first pixel signal and the second pixel signal are modeled in the latent space and their features are concatenated to obtain the first feature representation.

[0012] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the pose extraction network to perform global motion feature extraction on the reference video to obtain the first motion feature representation;

[0013] The intensity-aware action extraction network is invoked to extract facial features from the reference video to obtain the second action feature representation.

[0014] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform facial cropping on each video frame in the reference video to obtain multiple facial images.

[0015] A convolutional network is invoked to extract features from the multiple facial images to obtain facial action feature representations;

[0016] The facial action feature representation is added to the intensity number to obtain the second input feature representation of the intensity-aware action extraction network. The intensity number is used to indicate the intensity feature representation corresponding to the facial key point.

[0017] The intensity-aware action extraction network is invoked to extract features from the second input feature representation to obtain the second action feature representation.

[0018] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the first normalization layer in the intensity-aware action extraction network to perform feature processing on the second input feature to obtain a first intermediate feature representation;

[0019] The multi-head attention network in the intensity-aware action extraction network is invoked to process the first intermediate feature representation to obtain the second intermediate feature representation;

[0020] The second normalization layer in the intensity-aware action extraction network is invoked to process the second intermediate feature representation to obtain the third intermediate feature representation.

[0021] The multilayer perceptual network in the intensity-aware action extraction network is invoked to process the third intermediate feature representation to obtain the second action feature representation.

[0022] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to obtain a learnable action feature representation, which is a feature representation already stored in an action memory.

[0023] The multi-head attention network in the intensity-aware action extraction network is invoked to perform feature processing on the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation.

[0024] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to extract identity features from the reference image to obtain an identity encoding representation;

[0025] Global feature extraction is performed on the reference image to obtain a global feature representation;

[0026] Based on the identity encoding representation and the global feature representation, the character identity feature representation of the parameter image is determined, and the character identity feature representation is used to generate the simulated animation.

[0027] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to perform facial cropping on the reference image to obtain a first facial image;

[0028] The convolutional network is invoked to encode the identity of the first facial image, so as to obtain the identity encoding representation of the corresponding human image in the reference image.

[0029] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call a multi-layer convolutional network to perform multi-scale feature extraction and concatenation on the global feature representation to obtain a fourth intermediate feature representation;

[0030] A multi-head attention network is invoked to process the fourth intermediate feature representation and the identity encoding representation to obtain the identity feature representation of the person.

[0031] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call multiple convolutional modules of the denoising network to perform feature learning on the person's identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer.

[0032] The decoder of the third variational network is invoked to decode the denoised feature representation to obtain the simulated animation.

[0033] In one possible design, in another implementation of another aspect of the embodiments of this application, the processing module is used to call the spatial convolutional layer in the denoising network to perform feature learning on the first feature representation and the first action feature representation to obtain a fifth intermediate feature representation;

[0034] The spatiotemporal convolutional layer in the denoising network is invoked to learn features from the fifth intermediate feature representation in order to obtain the sixth intermediate feature representation.

[0035] The spatiotemporal attention layer in the denoising network is invoked to learn features from the sixth intermediate feature representation in order to obtain the seventh intermediate feature representation.

[0036] The appearance attention layer in the denoising network is invoked to learn features from the seventh intermediate feature representation and the character identity feature representation to obtain the eighth intermediate feature representation;

[0037] The action attention layer in the denoising network is invoked to learn features from the eighth intermediate feature representation and the second action feature representation to obtain the ninth intermediate feature representation.

[0038] The remaining convolutional modules then learn the features to obtain the denoised feature representation.

[0039] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0040] The memory is used to store programs;

[0041] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0042] Bus systems are used to connect memory and processor to enable communication between them.

[0043] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0044] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0045] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Implicit feature vectors of images are extracted from reference images and reference videos. These feature vectors contain high-dimensional information such as facial expressions and poses. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, learning facial features and action features separately enables good decoupling of facial features and action features under different image styles, improving the generalization ability of the solution. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of an application scenario for animation generation in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of an interface for generating emoticons in an instant messaging software according to an embodiment of this application.

[0048] Figure 3 This is a schematic diagram of an emoticon template in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of one embodiment of animation generation in this application.

[0050] Figure 5 This is a schematic diagram of a network structure of the intensity-sensing action extraction network in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of a network structure of the fine-grained appearance extraction network in an embodiment of this application;

[0052] Figure 7This is a schematic diagram of a network structure of the identity-aware adaptation network in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the processing flow of the animation generation model in the embodiments of this application;

[0054] Figure 9 This is a schematic diagram of one embodiment of the animation generation device in this application;

[0055] Figure 10 This is a schematic diagram of one embodiment of the server in this application;

[0056] Figure 11 This is a schematic diagram of one embodiment of the terminal in this application. Detailed Implementation

[0057] This application provides an animation generation method, apparatus, device, medium, and program product to improve the precision of capturing facial dynamic details, thereby improving the facial control accuracy of human figures in animation.

[0058] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0059] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0060] With the popularization of short videos and the rapid development of the animation industry, systems for intelligently generating animations based on videos and images have been widely used. In this process, users acquire images or videos containing human figures through cameras or other image acquisition tools, and then select an animation template (i.e., animated video) to synthesize virtual animations from the images or videos. For example, in one scenario, when a user is chatting with a friend through an instant messaging application, they can send emojis to express their meaning; these emojis can be generated based on the above method. Current solutions typically control facial expressions through explicit facial key points, but this not only leads to a loss of detail but also heavily relies on the accuracy of landmark extraction technology. Therefore, how to improve the facial dynamic details and control precision of human figures in animations is a problem that urgently needs to be solved.

[0061] To address this technical problem, this application provides the following technical solution: Acquiring a reference video and a reference image, wherein the reference video indicates the behavioral actions of a target object in a simulated animation, and the reference image indicates the facial information of the target object in the simulated animation; extracting features from pixels in the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; adding random noise feature representation to the first feature representation to obtain another first feature representation; extracting motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; and performing decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation. This extracts latent feature vectors from the reference image and reference video. These feature vectors contain high-dimensional information such as facial expressions and postures. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, learning facial features and motion features separately allows for good decoupling of facial features and motion features under different image styles, thereby improving the generalization ability of the solution.

[0062] Given that this application involves some technical terms, these terms will be introduced below.

[0063] Stable video diffusion model: This refers to a deep learning-based video generation model that achieves efficient and stable video generation through diffusion and denoising processes in the latent space. It can be understood as the video constructed using the variational network and denoising network in this application.

[0064] Implicit video-driven animation refers to the process of using implicitly represented motion information as control signals to drive the generation of human portrait animation, without directly using explicit facial key points or facial markers.

[0065] Human portrait animation generation: refers to the technology of converting static single-person images into video content with dynamic expressions and head postures through algorithms.

[0066] Identity information decoupling: The process of separating the identity information of a person from their motion information in a video, so as to allow for independent control and adjustment during animation generation.

[0067] Implicit representation: refers to a method that does not directly describe data or features, but indirectly defines data or features through a series of deep learning network operators. In this embodiment, it is used to encode human motion information and serve as a control signal.

[0068] Control signal injection: refers to integrating implicitly represented motion information into a denoising network through an attention mechanism to achieve precise control over the animation generation process.

[0069] Denoising network: A type of deep learning network used to learn the inverse denoising process in a latent space, which includes self-attention and cross-attention mechanisms.

[0070] Spatiotemporal consistency refers to the preservation of spatial details and temporal continuity in the generated video, ensuring that the movements and facial expressions of the characters in the video are natural and smooth.

[0071] Identity consistency: In the process of generating human portrait animation, maintain the consistency of the identity features (such as facial features) of the person in the source image.

[0072] Action memory: A technique for enhancing implicit motion feature representation by storing and retrieving contextual information between different frames to improve the continuity and context-awareness of motion features.

[0073] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0074] The animation generation method provided in this application embodiment can be applied to, for example... Figure 1 The system shown includes a terminal 100, a server 200, a database 300, and a network 400. The terminal 100, server 200, and database 300 are connected via the network 400. The database 300 is used to provide the system with reference videos, reference images, and generative models. Figure 1 The number of terminals, servers, and databases in the system shown is merely an example. For instance, there may be multiple terminals, servers, and databases. This application does not limit the number of terminals, servers, and databases.

[0075] The aforementioned system is used in the fields of image processing, animation simulation, emoji generation, and computer technology, and this application embodiment does not limit these applications. In this solution, server 200 can receive reference video and reference image sent by terminal 100, and then perform feature extraction on the pixels in the reference video and reference image to obtain a first feature representation, which is a latent feature of the reference video and reference image; add random noise feature representation to the first feature representation to obtain a second feature representation; extract motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, where the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation; and finally send the simulated animation to terminal 100. It should be noted that the reference video, reference image, and generation model are authorized by the user or fully authorized by all parties.

[0076] In this process, terminal 100 communicates with server 200 via a network. Database 300 can be integrated onto server 200, or it can be located in the cloud or on another server. During the animation generation process, interaction can occur between terminal 100 and server 200. For example, a user generates an animation generation request through terminal 100, including a reference video and reference image. Terminal 100 then sends the animation generation request, along with the reference video and reference image, to server 200. Server 200 generates a corresponding simulated animation based on the reference video and reference image and sends it back to terminal 100.

[0077] Terminal 100 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these. Server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), big data, and artificial intelligence platforms.

[0078] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously.

[0079] It is understood that in the specific embodiments of this application, data related to the reference video, reference image, generation model and motion memory library are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0080] Based on the above description, the animation application scheme of this application will be explained below using a specific application scenario. For example, when a user chats with a friend through an instant messaging application, they can send emoticons to express their meaning, and the emoticons can be generated based on the above scheme. Figure 2 As shown, in the chat interface of an instant messaging application, after a user selects the emoticon option, the following will appear: Figure 2 The image shown is an AI-powered emoji creation interface. After the user clicks "upload photo," they are redirected to the photo upload interface, where they can upload reference images by selecting the appropriate upload controls. For example, ... Figure 2 As shown, selecting "Upload Photo" allows you to directly capture a portrait image of yourself using your camera. Selecting "Upload from Album" directly opens the photo album application, allowing you to select a portrait image to upload. After uploading the portrait image, you can then select a relevant reference video (i.e.,...). Figure 3 (The emoticon template shown). Figure 3As shown, emoticon template 1 is an animated video of "saluting"; emoticon template 2 is an animated video of "happy"; emoticon template 3 is an animated video of "sending a heart"; and emoticon template 4 is an animated video of "refusing". Then, after the user selects the corresponding emoticon template, using that template as a reference video and the user's portrait image as a reference image, a simulated animation is generated. This generates a representation of the user's portrait information, but with the actions indicated in the emoticon template.

[0081] Based on the above introduction, and taking the server as the execution entity, the animation generation method in this application is described below. Please refer to [link / reference]. Figure 4 One embodiment of the animation generation method in this application includes:

[0082] 401. Obtain a reference video and a reference image. The reference video is used to indicate the behavior and actions of the target object in the simulated animation, and the reference image is used to indicate the human image information of the target object in the simulated animation.

[0083] In this embodiment, when applied to a scenario of simulated animation generation, the server can receive a simulated animation generation request from the terminal, and obtain the corresponding reference video and reference image based on the generation request.

[0084] Reference videos refer to dynamic video sequences containing the actions and behaviors of the target object, used to provide information on the motion trajectory and movement trends of the target object in the simulated animation. In other words, the main function of reference videos is to provide the behavioral patterns of the target object in the simulated animation, including key features such as limb movements, posture changes, movement speed, and rhythm. These features will be directly used to simulate the behavior of the target object in the animation.

[0085] Reference videos are a key input for generating dynamic simulation animations, and their sources typically include: 1. Actual video footage: Capturing the natural behavior of the reference object by filming its actual movements with camera equipment (such as mobile phones or camcorders). 2. Third-party video resources: Selecting corresponding reference videos from a stored video library as behavioral templates for generating animations. 3. Motion capture systems: Generating motion data of the reference object using motion capture technology (such as optical capture or inertial capture), which can usually be converted into reference video formats. 4. Virtually generated videos: Virtual videos generated or fitted by computers, such as synthetic videos generated using existing motion data.

[0086] Reference images are static images used to indicate the appearance characteristics (human image information) of a target object. They primarily provide visual attribute information about the target object in the simulated animation, such as facial features, expressions, and appearance style. The purpose of reference images is to provide clear guidance for the appearance information of the target object, including hairstyle, clothing, facial features, and expressions. These features will be used to construct the image of the target object in the simulated animation, giving it unique appearance characteristics.

[0087] Reference images can come from the following sources: 1. Actual photographs of the target object: Still photographs of the target object acquired through a camera or other capture equipment, accurately reflecting its appearance. 2. Static video frame extraction: Extracting a frame from other videos as a reference image, such as selecting the keyframe with the most prominent features of the target object from a video containing the target object. 3. Design sketches or illustrations: Sketches or design drawings of the target object created by artists, used to provide visual reference for animation. 4. Virtually generated images: Target portraits generated through artificial intelligence, or virtual reference images selected from external resource libraries.

[0088] In summary, the reference video provides motion information in the temporal dimension, ensuring that the simulated behavior of the target object conforms to the real or expected motion trajectory. The reference image provides appearance features in the spatial dimension, ensuring that the image of the target object is accurately reproduced in the animation.

[0089] Depending on the application scenario, the reference video and reference image can have different sources. For example, in an application scenario based on generating emojis from portraits, the reference video can be an emoji template, and the reference image can be any portrait image. In an animation capture scenario, the reference video can be a video generated by capturing the actual movements of the reference object through a camera device (such as a mobile phone, camcorder, etc.), and the reference image can be a portrait image of another object. In other words, no specific limitations are imposed here.

[0090] 402. Extract features from the pixels in the reference video and the reference image to obtain a first feature representation, which is the latent feature of the reference video and the reference image.

[0091] In this embodiment, after receiving the reference video and the reference image, the server extracts features from the pixels in the reference video and the reference image respectively to generate pixel signals; then, it performs latent space modeling on the pixel signals to obtain corresponding feature representations; then, it concatenates the feature representations corresponding to the reference video and the reference image to obtain the first feature representation.

[0092] It should be understood that, when extracting features from the reference video and the reference image, in order to ensure that the feature representations of the reference image and the reference video can be efficiently learned and generated in a low-dimensional, continuous latent space, the server can call a variational network to extract features from the reference video and the reference image. The process can be as follows: the server calls a first variational network to extract features from the pixels of each video frame in the reference video to obtain a first pixel signal; it calls a second variational network to extract features from the pixels in the reference image to obtain a second pixel signal; the first pixel signal and the second pixel signal are then used for latent space modeling and feature concatenation to obtain the first feature representation. It should be understood that the first pixel signal includes dynamic behavioral features in the reference video, such as motion trajectory, posture changes, and movement speed; the second pixel signal includes static appearance features of the human figure in the reference image, such as facial texture, expression information, and clothing features.

[0093] Variational networks, which combine deep learning with variational inference, are widely used to solve inverse problems, such as medical image reconstruction (MRI, CT), image denoising, and image deblurring. Their goal is to leverage the expressive power of neural networks to embed prior knowledge into the optimization problem, thereby improving the quality and efficiency of the solution. The first and second variational networks can have the same network structure or differ in design.

[0094] The extraction process of the first pixel signal by the first variational network is described below: Each video frame in the reference video is used as the input image of the first variational network; then, the convolutional neural network in the first variational network is called to extract the local features (such as edge information, texture, shape, etc.) of the pixels of each video frame; then, the extracted features are processed by a non-linear activation function (such as ReLU) and mapped to a higher-level feature space; finally, a latent space representation corresponding to each frame is generated, which is a low-dimensional, continuous latent feature.

[0095] Similarly, the second variational network uses the same process when processing the reference image, which will not be elaborated here.

[0096] 403. Extract motion features from the reference video to obtain a first motion feature representation and a second motion feature representation of the reference video. The first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video.

[0097] In this embodiment, in order to capture more accurate global motion features and facial motion features, the server can use different feature extraction networks to extract motion features from the reference video.

[0098] In one exemplary scheme, the server invokes a pose extraction network to perform global motion feature extraction on the reference video to obtain the first motion feature representation; then it invokes an intensity-aware motion extraction network to perform facial feature extraction on the reference video to obtain the second motion feature representation.

[0099] In one exemplary scheme, the pose extraction network may include a spatial feature extraction module and a temporal feature extraction module. The feature extraction process of the spatial feature extraction module can be as follows: each frame of the reference video is processed by a convolutional neural network in the spatial feature extraction module to extract spatial features of various body parts (such as the position and angle of the head, arms, and legs). The convolutional neural network extraction process includes multiple convolutional layers and pooling layers, progressively capturing low-level features (such as edges and contours) to high-level features (such as the overall shape of the limbs). The processing flow of the temporal feature extraction module can be as follows: the spatial feature sequence extracted by the spatial feature extraction module is further processed by a temporal modeling network (such as LSTM, GRU, or Transformer) to capture the dynamic relationships between frames. At this point, the temporal feature extraction module is responsible for modeling the movement patterns in the temporal dimension, such as the temporal relationships of the start, duration, and end of the movement. Finally, the frame-level features are fused into a high-dimensional global feature representation to obtain the first action feature representation. This first action feature representation reflects the global action information of the person in the reference video, such as walking trajectory, overall body movement direction, and pose change patterns.

[0100] In one exemplary scheme, the intensity-aware action extraction network can extract fine-grained features of the facial region of a person in a reference video, particularly action information related to facial expressions and muscle movements. Its feature extraction process for facial action features can be as follows: Figure 5As shown: Facial images are cropped from each video frame in the reference video to obtain multiple facial images; a convolutional network (i.e., the action encoder) is invoked to extract features from these multiple facial images to obtain facial action feature representations; these facial action feature representations are added to intensity numbers to obtain the second input feature representation of the intensity-aware action extraction network, where the intensity number indicates the intensity feature representation corresponding to the facial key points; the first normalization layer in the intensity-aware action extraction network is invoked to process the second input feature to obtain a first intermediate feature representation; the multi-head attention network in the intensity-aware action extraction network is invoked to process the first intermediate feature representation to obtain a second intermediate feature representation; the second normalization layer in the intensity-aware action extraction network is invoked to process the second intermediate feature representation to obtain a third intermediate feature representation; and the multilayer perceptron in the intensity-aware action extraction network is invoked to process the third intermediate feature representation to obtain the second action feature representation.

[0101] In the above scheme, when performing facial pre-cropping on each video frame of the reference video, the server can use a face detection model (such as a detector based on Haar features, Dlib, or deep learning) to locate the facial region of the person in the video; then, the facial region in each frame is cropped out to reduce background interference and focus on the feature extraction of facial movements.

[0102] Simultaneously, when adding the intensity number to the facial motion feature representation, the server also needs to calculate the intensity number. In this embodiment, the intensity number is an encoding mechanism that characterizes the intensity of facial motion changes by calculating the positional offset of facial key points in consecutive video frames. Each number can describe the amplitude of motion in a specific facial region (such as the tip of the nose, corner of the mouth, eyebrows, etc.). The process can be as follows: In the initial frame of the video, the initial position of the facial key point is recorded as a reference, and then the offset distance of the key point in subsequent frames is calculated; then, based on the offset distance of the facial key point, each frame is assigned an motion intensity number. A larger offset distance corresponds to a higher motion intensity, and vice versa.

[0103] In this solution, the server can use the following common models for detection: 1. Dlib model, which is based on HOG features and regression trees and can detect 68 facial keypoints. 2. MediaPipe Face Mesh model, a deep learning-based facial keypoint detection model that can detect 468 high-precision keypoints. 3. OpenPose model, which supports multimodal keypoint detection for the human body and face.

[0104] In one exemplary scheme, the server can calculate the Euclidean distance of each facial keypoint relative to its initial position in a two-dimensional plane as the offset distance. The calculated offset distance is then quantized to obtain the intensity number. For example, several intensity intervals can be defined, dividing the offset distance into multiple discrete intervals. For instance, an offset distance of [0, 1] might be assigned an intensity number of 0 (indicating no significant movement), an offset distance of (1, 3] an intensity number of 1 (indicating minor movement), an offset distance of (3, 5] an intensity number of 2 (indicating moderate movement), and an offset distance greater than 5 an intensity number of 3 (indicating significant movement).

[0105] To further learn the temporal information of the context within video frames, an action memory database can be added to the intensity-aware action extraction network during action extraction. For example... Figure 5 As shown, when the server calls the multi-head attention network in the intensity-aware action extraction network to process the first intermediate feature representation to obtain the second intermediate feature representation, the following process can be adopted: Obtain a learnable action feature representation, which is a feature representation already stored in the action memory; call the multi-head attention network in the intensity-aware action extraction network to process the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation. It should be understood that the action memory is a knowledge base or storage system specifically used to store and manage action data, typically for action-related generation, recognition, analysis, or simulation tasks. It can be viewed as a database containing various action information, enabling models or systems to retrieve or utilize this action information when needed.

[0106] A typical action memory bank usually includes the following components:

[0107] (1) Action feature data

[0108] Key point data: Stores the trajectory and positional changes of key points of a target object (such as the human body, face, or hand). Key points may include: two-dimensional or three-dimensional coordinate sequences of body joints (such as knees, elbows, etc.); motion trajectories of facial key points (such as corners of the eyes, corners of the mouth, tip of the nose, etc.); and dynamic changes of finger joints.

[0109] Action feature vector: A high-dimensional feature representation used to describe an action, which may be generated by neural networks, PCA (principal component analysis) or other feature extraction methods.

[0110] (2) Action Tags and Classification

[0111] Action tags: Add semantic tags to each action data, such as "walking", "running", "waving", "nodding", etc.

[0112] Action classification: Action data is classified according to the type, speed, intensity or other characteristics of the action to facilitate efficient retrieval.

[0113] (3) Time series data

[0114] Action memory banks typically store time-series information to describe the dynamic changes in actions. For example:

[0115] The movement trajectory of the target object over a period of time (such as the process from standing to walking).

[0116] The time segments of an action (such as the start, duration, and end time of the action).

[0117] (4) Action video or sample data

[0118] The motion memory bank may also contain video clips, images, or animations related to the motion as reference material.

[0119] (5) Action-related information

[0120] The relationships between actions, for example:

[0121] The continuity of certain actions (such as "jumping" followed by "landing").

[0122] The intensity of actions (such as the difference in intensity between "smiling" and "laughing").

[0123] 404. Decode the simulation animation based on the first feature representation, the first action feature representation, and the second action feature representation.

[0124] After obtaining the first feature representation, the first action feature representation, and the second action feature representation, the server inputs the first feature representation, the first action feature representation, and the second action feature representation into a denoising network for learning, and finally performs decoding processing to reconstruct the simulated animation.

[0125] To extract and process fine-grained features from the reference image, ensuring that the generated simulated animation accurately reproduces background details while preserving the character's identity, thereby improving the overall visual realism of the simulated animation, the server can also add a fine-grained appearance extraction network to extract identity features from the reference image to obtain a character identity feature representation. Simultaneously, in the development of the fine-grained appearance extraction network, to achieve more refined appearance control, a multi-scale adapter for identity perception (also referred to as an identity-aware adapter in this application) is proposed to enhance the model's ability to preserve portrait identity information and background details. One network structure of this fine-grained appearance extraction network can be as follows: Figure 6As shown, based on this Figure 6 The network structure shown illustrates the following process for the server to extract the person's identity feature representation: The server performs facial cropping on the reference image to obtain a first facial image; it calls a convolutional network to encode the first facial image to obtain the identity encoding representation of the corresponding person in the reference image; simultaneously, it performs global feature extraction on the reference image to obtain a global feature representation; based on the identity encoding representation and the global feature representation, it determines the person's identity feature representation of the parameter image, which is used to generate the simulated animation.

[0126] In such Figure 6 In the network structure shown, when determining the person's identity feature representation of the parameter image based on the identity encoding representation and the global feature representation, the global feature representation can be adapted to the identity encoding representation through the identity perception adapter to obtain an intermediate feature representation; at the same time, the global feature representation will also be input into the multilayer perception network for feature learning to obtain a middle intermediate feature representation; finally, the two intermediate feature representations are added together to obtain the person's identity feature representation.

[0127] In one exemplary scheme, the network structure of the identity-aware adapter can be as follows: Figure 7 As shown, based on this Figure 7 The network structure shown illustrates the following processing flow of the identity-aware adapter: The global feature representation is downsampled using a downsampling layer to obtain the corresponding latent features; then, multi-size convolutional layers are used to extract features from these latent features to obtain multi-size feature representations, which are then concatenated to obtain an intermediate feature representation; this intermediate feature representation, along with the identity feature (i.e., the identity encoding representation), is used as input to the multi-head attention layer for feature learning to obtain another intermediate feature representation; finally, this intermediate feature representation is upsampled to obtain a multi-size feature fusion intermediate feature representation. This approach maintains consistency in character identity during the generation of simulated animations and improves the ability to capture facial details, enabling precise control over character identity and background in simulated animations.

[0128] Based on the above scheme, the process of the denoising network can be as follows: The server calls multiple convolutional modules of the denoising network to learn features from the character identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer. The decoder of the variational network is called to decode the denoised feature representation to obtain the simulated animation.

[0129] The following section illustrates the feature learning process of this denoising network using the execution process of a specific convolutional module:

[0130] The server calls the spatial convolutional layer in the denoising network to learn features from the first feature representation and the first action feature representation to obtain a fifth intermediate feature representation; it calls the spatiotemporal convolutional layer in the denoising network to learn features from the fifth intermediate feature representation to obtain a sixth intermediate feature representation; it calls the spatiotemporal attention layer in the denoising network to learn features from the sixth intermediate feature representation to obtain a seventh intermediate feature representation; it calls the appearance attention layer in the denoising network to learn features from the seventh intermediate feature representation and the person's identity feature representation to obtain an eighth intermediate feature representation; it calls the action attention layer in the denoising network to learn features from the eighth intermediate feature representation and the second action feature representation to obtain a ninth intermediate feature representation; and so on, through the feature learning of the remaining convolutional modules, to obtain the denoised feature representation.

[0131] As described above, this embodiment, based on a stable video diffusion model, integrates identity and motion information through appearance and motion attention mechanisms, eliminating the need for fine-tuning the image diffusion model and separately training the motion module. Simultaneously, it reduces background jitter and portrait blurring when the identities of the individuals in the source image (i.e., the reference image) and the driving video (reference video) differ, thus improving video quality.

[0132] Based on the above scheme, the following will be used as an example. Figure 8 The network structure (i.e., the generative model) shown illustrates an animation generation method in this application: Figure 8 The generative model shown includes a variational network, a denoising network, a fine-grained appearance extraction network, a pose extraction network, and an intensity-aware action extraction network. The action memory database serves as an auxiliary to the intensity-aware action extraction network, enhancing the implicit representation of motion features and supplementing temporal information based on contextual information.

[0133] The variational network is used to extract pixel signals from the raw video and convert them into modeling information in the latent space. An intensity-aware motion extraction network encodes motion information and serves as a control signal for the animation stage. The denoising network models features from the appearance and motion extraction networks in the latent space. The fine-grained appearance extraction network manages the identities of characters and the background in the generated video. The identity-aware adapter in the fine-grained appearance extraction network enhances the visual appearance representation. (The text then repeats the description of the intensity-aware motion extraction network.)

[0134] The network parameters of the aforementioned variational network, denoising network, fine-grained appearance extraction network, pose extraction network, and intensity-aware action extraction network can all be trained. The training process can be as follows:

[0135] Construct training samples. In this embodiment, the training samples include training videos and training images, wherein the training image should be a video frame extracted from the training video. Furthermore, to improve the generalization ability of the generative model, the training samples should be diversified.

[0136] In this embodiment, a series of data augmentation and training strategies are also set up to adapt to different animation styles, and spatial control condition mapping is used to improve the stability of areas other than the face during the inference stage.

[0137] In one exemplary approach, to maximize the diversity of training samples within a limited pool of samples, data color enhancement and processing based on different style transfer networks can be employed to increase the number of training samples.

[0138] Data color enhancement is a method that generates new samples by altering image color attributes (such as brightness, contrast, and saturation). It manipulates the pixel values ​​of an image to generate samples with different color distributions while preserving the semantic information of the image. This method can simulate various scenarios, including different lighting conditions and camera settings, thereby improving the model's robustness to color changes. For example, suppose the original training sample is a face image. The original image is a frontal photograph of a face under normal lighting.

[0139] The color-enhanced samples can be categorized as follows:

[0140] Adjust brightness: Generate a brighter image and a darker image.

[0141] Adjust contrast: Generate a high-contrast image (with sharper facial contours) and a low-contrast image (with a softer overall look).

[0142] Change the saturation: Generate a brightly colored image and a near-grayscale image.

[0143] Add noise: Generate an image with random noise.

[0144] Style transfer is a deep learning-based technique that generates new images with different visual styles by applying a target style (such as artistic style or specific texture) to the original image. Style transfer networks not only increase the number of training samples but also introduce diverse image styles, thereby enhancing the model's adaptability to style changes. For example, suppose the original training sample is a face image. The original image is a photograph of a face taken under normal conditions.

[0145] The style-transferred samples can include the following categories:

[0146] Oil painting style: Transform facial images into the style of Van Gogh or Monet oil paintings.

[0147] Sketch Style: Transforms facial images into black and white sketches.

[0148] Cartoon style: Uses an animation style transfer network to generate face images with a cartoon effect.

[0149] Retro Style: Generates images with retro filter effects, simulating the style of old photos.

[0150] Cyberpunk Style: Generates images with neon lighting effects and high contrast, simulating the visual effects of cyberpunk scenes.

[0151] Then, a variational network is used to extract features from the video and image data in the training samples to obtain latent features. Random noise is then added to the latent features as input to the denoising network. At the same time, other networks are called to perform corresponding feature extraction to obtain corresponding feature representations. The corresponding feature representations are then input into the corresponding attention layers for denoising learning. Finally, the variational network is used for decoding to obtain the predicted animation.

[0152] The loss value is calculated based on the predicted animation and the training video of the training sample; finally, the generative model is trained based on the loss value to obtain a generative model that can be applied online.

[0153] It should be understood that when generative models are applied to online inference, the stability of regions other than the face can be improved through spatial control conditional mapping. For example, to generate better simulated animations with skeletal differences in different body shapes and spatial positions, certain strategies can be used to adjust positions and proportions. That is, in animation generation or motion transfer tasks, characters in reference images and reference videos may have different body shapes (such as height, build, limb proportions, etc.). Differences in skeletal structure can lead to distortions and unnatural effects when directly transferring the driven action. For example, transferring the action of a long arm to a short-armed character may cause the action to exceed the natural range. On the other hand, characters in reference videos and reference images may be located in different spatial positions (e.g., the driving character may be standing, while the source character may be sitting or leaning). If the differences in spatial position are not considered, motion transfer may lead to misalignment or distortion in the generated results.

[0154] To address the aforementioned issues, appropriate reasoning strategies can be set during the online generation process.

[0155] In one exemplary approach, the inference strategy involves adjusting using standardized skeletal proportions. This means normalizing the skeletons in the reference image and video, mapping different bone lengths to a standardized proportional space. For example, the bone proportions can be calculated using the coordinates of keypoints (such as shoulders, elbows, and wrists); then, the motion in the reference video is scaled according to the standardized proportions to fit the skeletal structure of the reference image.

[0156] In another exemplary scheme, the inference strategy is to perform non-linear scaling adjustments. That is, for certain specific body shapes (such as babies or cartoon characters), a non-linear scaling strategy needs to be introduced to adjust different parts (such as the head, limbs, and torso) independently.

[0157] In another exemplary approach, the inference strategy is body shape matching. That is, the skeleton in the reference video is deformed using a shape matching algorithm (such as Thin-Plate Spline transform) to make it consistent with the skeleton shape of the reference image.

[0158] In another exemplary scheme, the inference strategy is keypoint alignment. That is, by translating and rotating the keypoint coordinates of the reference video and reference image, the skeletal center points (such as the midpoint of the pelvis or spine) of the two are made to coincide.

[0159] In another exemplary approach, the inference strategy is pose normalization. This involves mapping the pose of the skeleton in the reference video to the initial pose of the reference image, for example, by calculating a rotation matrix or affine transformation, to adjust the overall position of the skeleton in the reference image to a coordinate system consistent with that of the skeleton in the reference image.

[0160] In another exemplary approach, the inference strategy is flexible alignment. That is, if the initial postures of the two are significantly different (e.g., the driving character is standing while the source character is sitting), the action needs to be flexibly transformed, for example, by gradually transitioning the standing action to the sitting action.

[0161] The beneficial effects of this application are illustrated below with specific experimental results:

[0162] Experimental results (as shown in Table 1) demonstrate that the solution provided in this application outperforms existing technologies in both Frechet sensing distance and Frechet video distance, proving its superiority in generating video quality.

[0163] The results shown in Table 1 are objective comparison results on the VFHQ dataset. Indicators marked with ↓ are better the lower they are, and indicators marked with ↑ are better the higher they are.

[0164] Table 1

[0165]

[0166] Secondly, in terms of enhancing identity preservation and facial dynamic generation capabilities, the technical solution provided in this application effectively preserves the identity features of the person in the source image while more accurately capturing facial dynamics through the application of an identity-aware multi-scale adapter and an action memory bank. As shown in Tables 1 and 2, the technical solution provided in this application shows significant improvements over existing methods in terms of identity similarity, facial motion quality, image visual quality, and image smoothness, demonstrating its superior performance in processing complex portraits.

[0167] Secondly, regarding the improvement of video detail capture capabilities, the technical solution provided in this application, through a fine-grained appearance extraction network and an intensity-aware motion extraction network, is able to capture richer facial details and dynamics. As shown in Table 2, the technical solution provided in this application performs better in capturing facial movements such as eye movement direction, eye rotation, and lip synchronization. These details are often difficult to capture in previous technologies, therefore, these methods show a significant gap compared to the technical solution provided in this application in terms of facial movement quality.

[0168] The results shown in Table 1 are subjective comparison results on the VFHQ dataset. Indicators marked with ↓ are better the lower they are, and indicators marked with ↑ are better the higher they are.

[0169] Table 2

[0170]

[0171] The animation generation apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 9 , Figure 9This is a schematic diagram of one embodiment of the animation generation device in this application. The animation generation device 20 includes:

[0172] The acquisition module 201 is used to acquire a reference video and a reference image. The reference video is used to indicate the behavior and actions of the target object in the simulated animation, and the reference image is used to indicate the portrait information of the target object in the simulated animation.

[0173] The processing module 202 is configured to extract features from pixels in the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; extract motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; and perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

[0174] This application provides an animation generation apparatus. Using this apparatus, implicit feature vectors of images are extracted from reference images and reference videos. These feature vectors contain high-dimensional information such as facial expressions and poses. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, facial features and motion features are learned separately, which enables good decoupling of facial features and motion features under different image styles, improving the generalization ability of the solution.

[0175] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0176] The processing module 202 is used to call the first variational network to extract features from the pixels of each video frame in the reference video to obtain the first pixel signal;

[0177] The second variational network is invoked to extract features from the pixels in the reference image to obtain the second pixel signal;

[0178] The first pixel signal and the second pixel signal are modeled in the latent space and their features are concatenated to obtain the first feature representation.

[0179] This application provides an animation generation apparatus. Using this apparatus, feature extraction is performed on reference images and reference videos based on pixel points, and latent space modeling is then performed and concatenated to learn effective feature representations of the reference images and videos. These feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. The use of variational networks enables the efficient learning and generation of image and video feature representations in a low-dimensional, continuous latent space.

[0180] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to call the pose extraction network to perform global motion feature extraction on the reference video in order to obtain the first motion feature representation;

[0181] The intensity-aware action extraction network is invoked to extract facial features from the reference video to obtain the second action feature representation.

[0182] This application provides an animation generation apparatus. Using this apparatus, global motion features and facial motion features are extracted using different network layers. By learning these global pose and facial motion features, richer motion feature representations can be learned, thereby improving the motion accuracy in simulated animations.

[0183] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0184] The processing module 202 is used to perform facial cropping on each video frame in the reference video to obtain multiple facial images;

[0185] A convolutional network is invoked to extract features from the multiple facial images to obtain facial action feature representations;

[0186] The facial action feature representation is added to the intensity number to obtain the second input feature representation of the intensity-aware action extraction network. The intensity number is used to indicate the intensity feature representation corresponding to the facial key point.

[0187] The intensity-aware action extraction network is invoked to extract features from the second input feature representation to obtain the second action feature representation.

[0188] This application provides an animation generation apparatus. Using this apparatus, an intensity-aware motion extraction network analyzes facial expressions and head postures in a reference video to generate implicit representations that control animation details. This allows it to learn richer motion feature representations, thereby improving the accuracy of motion in simulated animation. Simultaneously considering the intensity and complexity of motion, it can more accurately capture and reproduce the dynamic features of characters, thus improving the naturalness and realism of the animation.

[0189] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0190] The processing module 202 is used to call the first normalization layer in the intensity-aware action extraction network to perform feature processing on the second input feature to obtain a first intermediate feature representation;

[0191] The multi-head attention network in the intensity-aware action extraction network is invoked to process the first intermediate feature representation to obtain the second intermediate feature representation;

[0192] The second normalization layer in the intensity-aware action extraction network is invoked to process the second intermediate feature representation to obtain the third intermediate feature representation.

[0193] The multilayer perceptual network in the intensity-aware action extraction network is invoked to process the third intermediate feature representation to obtain the second action feature representation.

[0194] This application provides an animation generation apparatus. Using this apparatus, an intensity-aware motion extraction network analyzes facial expressions and head postures in a reference video to generate implicit representations that control animation details. This allows it to learn richer motion feature representations, thereby improving the accuracy of motion in simulated animation. Simultaneously considering the intensity and complexity of motion, it can more accurately capture and reproduce the dynamic features of characters, thus improving the naturalness and realism of the animation.

[0195] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to obtain a learnable action feature representation, which is a feature representation already stored in the action memory bank;

[0196] The multi-head attention network in the intensity-aware action extraction network is invoked to perform feature processing on the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation.

[0197] This application provides an animation generation apparatus. Using this apparatus, during the motion feature learning process, the implicitly represented motion features are enhanced, thereby supplementing the temporal information of the motion based on contextual information. Specifically, by storing and retrieving contextual information between different video frames, the continuity and complexity of corresponding motions within video frames are captured and understood, thereby achieving smoother and more accurate dynamic changes in simulated animation.

[0198] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to extract identity features from the reference image to obtain an identity code representation;

[0199] Global feature extraction is performed on the reference image to obtain a global feature representation;

[0200] Based on the identity encoding representation and the global feature representation, the character identity feature representation of the parameter image is determined, and the character identity feature representation is used to generate the simulated animation.

[0201] This application provides an animation generation apparatus. By extracting and processing fine-grained features from a reference image, the generated simulated animation accurately reproduces background details while maintaining the character's identity, thereby improving the overall visual realism of the simulated animation.

[0202] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0203] The processing module 202 is used to crop the face of the reference image to obtain a first face image;

[0204] The convolutional network is invoked to encode the identity of the first facial image, so as to obtain the identity encoding representation of the corresponding human image in the reference image.

[0205] This application provides an animation generation apparatus. By extracting and processing fine-grained features from a reference image, the generated simulated animation accurately reproduces background details while maintaining the character's identity, thereby improving the overall visual realism of the simulated animation.

[0206] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0207] The processing module 202 is used to call a multi-layer convolutional network to perform multi-scale feature extraction and concatenation on the global feature representation to obtain a fourth intermediate feature representation;

[0208] A multi-head attention network is invoked to process the fourth intermediate feature representation and the identity encoding representation to obtain the identity feature representation of the person.

[0209] This application provides an animation generation apparatus. Using this apparatus, multi-layer convolutional layers and a multi-head attention network are employed to extract and fuse multi-scale features from reference images, thereby maintaining consistency in character identity during the generation of simulated animation and improving the ability to capture facial details, thus achieving precise control over character identity and background in the simulated animation.

[0210] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0211] The processing module 202 is used to call multiple convolutional modules of the denoising network to learn features from the person's identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer.

[0212] The decoder of the third variational network is invoked to decode the denoised feature representation to obtain the simulated animation.

[0213] This application provides an animation generation apparatus. Using this apparatus, a denoising network is employed to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the simulated animation generation process, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0214] Optionally, in the above Figure 9 Based on the corresponding embodiments, in another embodiment of the animation generation apparatus 20 provided in this application,

[0215] The processing module 202 is used to call the spatial convolutional layer in the denoising network to perform feature learning on the first feature representation and the first action feature representation to obtain the fifth intermediate feature representation;

[0216] The spatiotemporal convolutional layer in the denoising network is invoked to learn features from the fifth intermediate feature representation in order to obtain the sixth intermediate feature representation.

[0217] The spatiotemporal attention layer in the denoising network is invoked to learn features from the sixth intermediate feature representation in order to obtain the seventh intermediate feature representation.

[0218] The appearance attention layer in the denoising network is invoked to learn features from the seventh intermediate feature representation and the character identity feature representation to obtain the eighth intermediate feature representation;

[0219] The action attention layer in the denoising network is invoked to learn features from the eighth intermediate feature representation and the second action feature representation to obtain the ninth intermediate feature representation.

[0220] The remaining convolutional modules then learn the features to obtain the denoised feature representation.

[0221] This application provides an animation generation apparatus. Using this apparatus, a denoising network is employed to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the simulated animation generation process, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0222] The animation generation device provided in this application can be used on a server; please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0223] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0224] The steps performed by the server in the above embodiments can be based on this Figure 10 The server structure shown.

[0225] The animation generation device provided in this application can be used in terminal devices; please refer to [link / reference]. Figure 11 For ease of explanation, only the parts relevant to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0226] Figure 11 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 11 The smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that... Figure 11 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0227] The following is combined with Figure 11 A detailed introduction to the various components of a smartphone:

[0228] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0229] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0230] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 480, and can also receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0231] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 11 In this embodiment, the touch panel 431 and the display panel 441 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 431 and the display panel 441 can be integrated to realize the input and output functions of the smartphone.

[0232] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0233] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0234] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi modules (470), can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 11 WiFi module 470 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0235] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0236] The smartphone also includes a power supply 490 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0237] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0238] The steps performed by the terminal device in the above embodiments can be based on this Figure 11 The terminal device structure is shown.

[0239] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0240] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0241] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0243] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0245] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0246] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An animation generation method, characterized in that, include: Acquire reference videos and reference images, wherein the reference videos are used to indicate the behavioral actions corresponding to the target object in the simulated animation, and the reference images are used to indicate the portrait information corresponding to the target object in the simulated animation; Feature extraction is performed on the pixels in the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; Motion features are extracted from the reference video to obtain a first motion feature representation and a second motion feature representation of the reference video. The first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video. The simulation animation is obtained by decoding based on the first feature representation, the first action feature representation, and the second action feature representation.

2. The method according to claim 1, characterized in that, The step of extracting features from pixels in the reference video and the reference image to obtain a first feature representation includes: The first variational network is invoked to extract features from the pixels of each video frame in the reference video to obtain the first pixel signal; The second variational network is invoked to extract features from the pixels in the reference image to obtain the second pixel signal; The first pixel signal and the second pixel signal are latent space modeled and feature concatenated to obtain the first feature representation.

3. The method according to claim 1, characterized in that, Extracting motion features from the reference video to obtain a first motion feature representation and a second motion feature representation of the reference video includes: The pose extraction network is invoked to extract global motion features from the reference video to obtain the first motion feature representation; The intensity-aware action extraction network is invoked to extract facial features from the reference video to obtain the second action feature representation.

4. The method according to claim 3, characterized in that, The step of invoking the intensity-aware action extraction network to extract facial features from the reference video to obtain the second action feature representation includes: Facial images are obtained by cropping the faces from each video frame in the reference video. A convolutional network is invoked to extract features from the multiple facial images to obtain facial action feature representations; The facial action feature representation is added to the intensity number to obtain the second input feature representation of the intensity-aware action extraction network, whereby the intensity number is used to indicate the intensity feature representation corresponding to the facial key point. The intensity-aware action extraction network is invoked to extract features from the second input feature representation to obtain the second action feature representation.

5. The method according to claim 4, characterized in that, The step of invoking the intensity-aware action extraction network to extract features from the second input feature representation to obtain the second action feature representation includes: The first normalization layer in the intensity-aware action extraction network is invoked to process the second input features to obtain a first intermediate feature representation; The multi-head attention network in the intensity-aware action extraction network is invoked to perform feature processing on the first intermediate feature representation to obtain the second intermediate feature representation; The second normalization layer in the intensity-aware action extraction network is invoked to perform feature processing on the second intermediate feature representation to obtain the third intermediate feature representation; The multilayer perceptual network in the intensity-aware action extraction network is invoked to perform feature processing on the third intermediate feature representation to obtain the second action feature representation.

6. The method according to claim 5, characterized in that, The step of invoking the multi-head attention network in the intensity-aware action extraction network to perform feature processing on the first intermediate feature representation to obtain the second intermediate feature representation includes: Obtain a learnable action feature representation, wherein the learnable action feature representation is a feature representation already stored in an action memory bank; The multi-head attention network in the intensity-aware action extraction network is invoked to perform feature processing on the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The reference image is subjected to identity feature extraction to obtain an identity encoding representation; Global feature extraction is performed on the reference image to obtain a global feature representation; The character identity feature representation of the parameter image is determined based on the identity encoding representation and the global feature representation, and the character identity feature representation is used to generate the simulated animation.

8. The method according to claim 7, characterized in that, The step of extracting identity features from the reference image to obtain an identity encoding representation includes: The reference image is cropped to obtain a first facial image; A convolutional network is invoked to encode the identity of the first facial image, so as to obtain the identity encoding representation of the corresponding human image in the reference image.

9. The method according to claim 7, characterized in that, The process of determining the person identity feature representation of the parameter image based on the identity encoding representation and the global feature representation includes: A multi-layer convolutional network is invoked to perform multi-scale feature extraction and concatenation on the global feature representation to obtain a fourth intermediate feature representation; A multi-head attention network is invoked to perform feature processing on the fourth intermediate feature representation and the identity encoding representation to obtain the person's identity feature representation.

10. The method according to claim 7, characterized in that, The decoding process based on the first feature representation, the first action feature representation, and the second action feature representation to obtain the simulated animation includes: Multiple convolutional modules of the denoising network are invoked to learn features from the person's identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer. The denoised feature representation is decoded by calling the decoder of the variational network to obtain the simulated animation.

11. The method according to claim 10, characterized in that, The process of calling multiple convolutional modules of the denoising network to learn features from the person's identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain the denoised feature representation includes: The spatial convolutional layer in the denoising network is invoked to perform feature learning on the first feature representation and the first action feature representation to obtain the fifth intermediate feature representation; The spatiotemporal convolutional layer in the denoising network is invoked to perform feature learning on the fifth intermediate feature representation to obtain the sixth intermediate feature representation; The spatiotemporal attention layer in the denoising network is invoked to perform feature learning on the sixth intermediate feature representation to obtain the seventh intermediate feature representation; The appearance attention layer in the denoising network is invoked to learn features from the seventh intermediate feature representation and the character identity feature representation to obtain the eighth intermediate feature representation; The action attention layer in the denoising network is invoked to perform feature learning on the eighth intermediate feature representation and the second action feature representation to obtain the ninth intermediate feature representation; The denoised feature representation is obtained by learning the features of the remaining convolutional modules.

12. An animation generation device, characterized in that, include: The acquisition module is used to acquire reference videos and reference images, wherein the reference videos are used to indicate the behavioral actions corresponding to the target object in the simulated animation, and the reference images are used to indicate the portrait information corresponding to the target object in the simulated animation; The processing module is used to extract features from the pixels in the reference video and the reference image to obtain a first feature representation, wherein the first feature representation is the latent feature of the reference video and the reference image; Motion features are extracted from the reference video to obtain a first motion feature representation and a second motion feature representation of the reference video. The first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video. Decoding is performed based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

13. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method of any one of claims 1 to 11 according to instructions in the program code; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

14. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the method as described in any one of claims 1 to 11.

Images