Multi-pose portrait animation generation method and device, equipment and storage medium

Abstract

The present application relates to a kind of multi-pose portrait animation generation methods, by obtaining portrait image and converting it into three-dimensional morphable model;Acquire audio data, and input audio-driven facial motion parameter extraction network, calculate facial motion parameter according to facial expression parameter;Using the pose generation of variational autoencoder, head pose parameter, hand pose parameter and audio data are calculated to obtain head pose motion parameter and hand pose motion parameter;Extract the appearance feature of portrait image, and standard key point estimation is carried out to portrait image, generates standard key point;Head pose parameter, hand pose parameter, facial expression parameter, facial motion parameter, head pose motion parameter and hand pose motion parameter are mapped into parameter representation under 2D view;Finally, the parameter representation after mapping, standard key point and appearance feature are input into image generator, and multi-pose portrait animation containing hand pose and dynamic expression is generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a multi-pose portrait animation generation method, device, equipment and storage medium. BACKGROUND

[0002] In recent years, with the rapid development of computer graphics and artificial intelligence technology, generating realistic animation portraits based on image and audio data has become an important research direction. In the fields of entertainment, film production, virtual reality (VR), augmented reality (AR), etc., generating realistic and natural animation portraits has wide application prospects.

[0003] Traditional portrait animation generation methods usually rely on pre-recorded materials or manual adjustment to generate animations. Although these methods can generate high-quality animation effects, they have the following limitations:

[0004] High labor cost: manually adjusting facial expressions, postures and motion parameters requires a lot of human intervention, resulting in long production cycles and high costs.

[0005] Lack of flexibility: pre-recorded materials are often limited to specific scenes and actions, and cannot adapt to different user needs, especially when generating personalized or customized animations, which often appear to be inadequate.

[0006] Complex data processing: as the complexity of input data increases, such as multi-angle, multi-pose portrait images and synchronous audio data, existing technologies are difficult to process and generate dynamic animations that meet the requirements in real time while maintaining high quality.

[0007] In order to overcome the limitations of traditional methods, portrait animation generation technology based on three-dimensional deformation model has received widespread attention. This technology extracts three-dimensional information from two-dimensional portrait images, combines facial expression parameters, posture parameters, etc., to generate adjustable three-dimensional models. This method can effectively improve the flexibility and quality of animation generation. However, existing three-dimensional model generation technology also has some shortcomings:

[0008] Limited model generation accuracy: in the process of generating three-dimensional models from two-dimensional images, the accuracy of the model may be insufficient due to factors such as image quality, lighting conditions, etc., which affects the naturalness of the final animation.

[0009] Complexity of motion parameter generation: dynamic changes in facial expressions and postures require complex algorithms to generate, especially when dealing with multi-modal data (such as the combination of audio and image), existing methods are difficult to generate accurate motion parameters while ensuring real-time performance. SUMMARY

[0010] The main purpose of the present application is to provide a multi-pose portrait animation generation method, device, equipment and storage medium, aiming at solving the technical problems that the prior art is difficult to effectively combine multi-modal data when generating multi-pose portrait animation, resulting in the lack of realism and synchronization of the generated animation.

[0011] To achieve the above-mentioned purpose, the present application provides a multi-pose portrait animation generation method, comprising:

[0012] Obtaining a portrait image, converting the portrait image into a three-dimensional morphable model, the three-dimensional morphable model comprising facial expression parameters, head pose parameters and hand pose parameters, obtaining audio data, inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters;

[0013] Using a pose-generated variational autoencoder to calculate head pose motion parameters and hand pose motion parameters according to the head pose parameters, hand pose parameters and audio data;

[0014] Extracting appearance features of the portrait image, and generating standard key points by performing standard key point estimation on the portrait image;

[0015] Inputting the mapped parameter representation, standard key points and appearance features into an image generator to generate a multi-pose portrait animation containing hand pose and dynamic expression.

[0016] In one embodiment, the pose-generated variational autoencoder calculates head pose motion parameters and hand pose motion parameters according to the head pose parameters, hand pose parameters and audio data, comprising:

[0017] Selecting one or more motion style parameters from a pre-defined motion style library, the motion style parameters being used to determine the style characteristics of head and hand motion;

[0018] Inputting the head pose parameters, hand pose parameters, motion style parameters and audio data into the pose-generated variational autoencoder model;

[0019] The pose-generated variational autoencoder model generates corresponding head pose motion parameters and hand pose motion parameters according to the input data through an encoding and decoding process.

[0020] In one embodiment, before the pose-generated variational autoencoder calculates head pose motion parameters and hand pose motion parameters according to the head pose parameters, hand pose parameters, motion style parameters and audio data, it further comprises:

[0021] collecting a standard dataset comprising head pose parameters, hand pose parameters, motion style parameters, and audio data;

[0022] defining a weighted loss function composed of a mean square error, a KL divergence, and an adversarial loss adjusted by weights;

[0023] The pose generation variational autoencoder is a PoseVAE model, and the PoseVAE model is trained using the weighted loss function and the standard dataset, and the parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

[0024] In one embodiment, the head pose motion parameters and the hand pose motion parameters are calculated from the head pose parameters, the hand pose parameters, the motion style parameters, and the audio data using a pose generation variational autoencoder, comprising:

[0025] The head pose parameters, the hand pose parameters, the motion style parameters, and the audio data are input into the improved PoseVAE model to generate head pose change amounts and hand pose change amounts.

[0026] Two VAE encoders that do not share parameters are used to respectively encode the head pose change amounts and the hand pose change amounts, and the head pose change amounts and the hand pose change amounts are mapped into two different latent space distributions;

[0027] The two latent space distributions are fused and resampled through Gaussian mixture to obtain a new latent space distribution, and the new latent space distribution is decoded using a VAE decoder to generate the head pose motion parameters and the hand pose motion parameters.

[0028] In one embodiment, the portrait image is converted into a three-dimensional morphable model, comprising:

[0029] The portrait image is preprocessed, including image denoising, illumination correction, and contrast adjustment, to improve the quality of the portrait image;

[0030] The preprocessed portrait image is converted into the three-dimensional morphable model using monocular 3D face recognition technology;

[0031] The facial expression parameters, the head pose parameters, and the hand pose parameters in the three-dimensional morphable model are extracted.

[0032] In one embodiment, the audio data is input into an audio-driven facial motion parameter extraction network, and facial motion parameters are calculated from the facial expression parameters, comprising:

[0033] Feature extraction is performed on the audio data to extract audio features including emotion, tone, and rhythm;

[0034] The extracted audio features and facial expression parameters are input into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, which are used to generate dynamic facial expressions.

[0035] In one embodiment, mapping the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view includes:

[0036] The head posture parameters, hand posture parameters, facial expression parameters, and facial movement parameters are mapped onto a 2D plane through perspective projection to generate a preliminary parameter representation in a 2D view.

[0037] Based on facial expression parameters and facial motion parameters, adjust the positions of standard key points in the 2D view to accurately reflect changes in facial expression and posture on the 2D plane.

[0038] Based on the adjusted standard key point positions and the preliminary parameter representation in the 2D view, the head pose motion parameters and hand pose motion parameters are smoothed using an interpolation algorithm to generate the final parameter representation in the 2D view, which is used to drive the generation of multi-pose portrait animation.

[0039] Furthermore, to achieve the above objectives, the present invention also provides a multi-pose portrait animation generation apparatus, comprising:

[0040] The 3D model generation module acquires a portrait image, converts the portrait image into a 3D deformation model, the 3D deformation model includes facial expression parameters, head posture parameters, and hand posture parameters, acquires audio data, inputs the audio data into an audio-driven facial motion parameter extraction network, and calculates facial motion parameters based on the facial expression parameters.

[0041] The posture generation module uses a variational autoencoder to calculate head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, and audio data.

[0042] The 2D mapping module extracts the appearance features of the portrait image, performs standard key point estimation on the portrait image to generate standard key points, and maps the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view.

[0043] The multi-pose animation generation module takes the mapped parameter representation, standard key points, and appearance features as inputs to the image generator to generate multi-pose portrait animations that include hand poses and dynamic expressions.

[0044] Furthermore, to achieve the above objectives, the present invention also provides a multi-pose portrait animation generation device, the multi-pose portrait animation generation device including a memory, a processor, and a multi-pose portrait animation generation program stored in the memory and executable on the processor, wherein when the multi-pose portrait animation generation program is executed by the processor, it implements the steps of the multi-pose portrait animation generation method as described above.

[0045] Furthermore, to achieve the above objectives, the present invention also provides a computer storage medium storing a multi-pose portrait animation generation program, which, when executed by a processor, implements the steps of the multi-pose portrait animation generation method as described above.

[0046] Beneficial Effects: This invention relates to a method for generating multi-pose portrait animations. The method involves acquiring a portrait image and converting it into a three-dimensional deformation model; acquiring audio data and inputting it into an audio-driven facial motion parameter extraction network to calculate facial motion parameters based on facial expression parameters; using a variational autoencoder for pose generation to calculate head pose motion parameters and hand pose motion parameters based on head pose parameters, hand pose parameters, and audio data; extracting appearance features from the portrait image and estimating standard key points to generate standard key points; mapping the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view; and finally, inputting the mapped parameter representations, standard key points, and appearance features into an image generator to generate a multi-pose portrait animation containing hand poses and dynamic expressions. This invention transforms portrait images into 3D deformation models containing parameters of facial expressions, head posture, and hand posture. Combined with an audio data-driven facial motion parameter extraction network, it accurately generates facial expressions and posture changes that match the sound, thus enhancing the realism and naturalness of the animation. By using a variational autoencoder for posture generation to calculate the motion parameters of the head and hands, it achieves synchronization between facial expressions and postures and audio. Furthermore, by estimating standard keypoints of the appearance features of the portrait image and inputting the mapped parameters into an image generator, it achieves a natural combination of dynamic expressions and hand postures, thereby generating richer and more realistic multi-pose portrait animations. Attached Figure Description

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0048] Figure 1This is a flowchart illustrating an embodiment of the multi-pose portrait animation generation method of the present invention;

[0049] Figure 2 This is a schematic diagram of the functional modules of a preferred embodiment of the multi-pose portrait animation generation device of the present invention;

[0050] Figure 3 This is a schematic diagram of the hardware operating environment of the device involved in the embodiment of the multi-pose portrait animation generation device of the present invention. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] It should be noted that in recent years, with the rapid development of computer graphics and artificial intelligence technologies, generating realistic animated portraits based on image and audio data has become an important research direction. In fields such as entertainment, film and television production, virtual reality (VR), and augmented reality (AR), generating realistic and natural animated portraits has broad application prospects.

[0053] Traditional methods of portrait animation generation typically rely on pre-recorded footage or manual adjustments to generate animation. While these methods can produce high-quality animation effects, they have the following limitations:

[0054] High labor costs: Manually adjusting facial expressions, postures, and motion parameters requires a lot of human intervention, resulting in long production cycles and high costs.

[0055] Lack of flexibility: Pre-recorded materials are often limited to specific scenes and actions, and cannot flexibly adapt to different user needs, especially when generating personalized or customized animations, where they often fall short.

[0056] Data processing complexity: As the complexity of input data increases, such as multi-angle, multi-pose portrait images and synchronized audio data, existing technologies struggle to process and generate dynamic animations that meet requirements in real time while maintaining high quality.

[0057] To overcome the limitations of traditional methods, portrait animation generation technology based on 3D deformation models has attracted widespread attention. This technology extracts 3D information from 2D portrait images and combines it with facial expression parameters, pose parameters, etc., to generate an adjustable 3D model. This method can effectively improve the flexibility and quality of animation generation. However, existing 3D model generation technologies also have some shortcomings:

[0058] Limited model generation accuracy: In the process of generating a 3D model from a 2D image, factors such as image quality and lighting conditions may affect the accuracy of the model, thus affecting the naturalness of the final animation.

[0059] Complexity of motion parameter generation: Dynamic changes in facial expressions and postures require complex algorithms to generate, especially when multimodal data (such as the combination of audio and images) are involved. Existing methods struggle to generate accurate motion parameters while ensuring real-time performance.

[0060] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multi-pose portrait animation generation method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0061] like Figure 1 As shown, the multi-pose portrait animation generation method proposed in this invention includes the following steps:

[0062] S10, acquire a portrait image, convert the portrait image into a three-dimensional deformation model, the three-dimensional deformation model includes facial expression parameters, head posture parameters and hand posture parameters, acquire audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters based on the facial expression parameters;

[0063] In this embodiment, a portrait image of the target person is acquired via a camera or other image acquisition device, or a pre-generated portrait image is received directly via a network or other data transmission method. Then, the portrait image is converted into a 3D Morphable Model (3DMM), which uses a fixed average shape. As a foundation, and combined with the identity feature parameter αU id and facial feature parameter βU exp Generate a personalized 3D face model. The spatial expression of the 3D deformation model is:

[0064]

[0065] in, The average shape of a 3D human face serves as the basis for the model; αU id The variable parameters refer to identity features, where α is an 80-dimensional identity weight vector, and U id It is the identity feature basis vector, representing the deviation of an individual from the average shape; βU exp Refers to the variable parameters of facial expressions, where β is a 64-dimensional expression weight vector, and U exp It is the facial feature basis vector, representing the deviation of facial expression from neutral expression.

[0066] Based on the generated three-dimensional deformation model, motion parameters ρ = [r, t] are added, where r represents the rotation of the head [belonging to SO(3)], and t represents the translation of the head (belonging to R). 3 In addition, the model includes key points of the upper body skeleton and their motion parameters λ, describing changes in human posture.

[0067] A 3D deformation model is used to represent the facial structure and posture of a target person. This model includes facial expression parameters, head posture parameters, and hand posture parameters.

[0068] Real-time audio data is acquired via microphone or other audio acquisition devices, or pre-recorded audio data is received directly via network or other data transmission methods. This audio data drives changes in facial expressions and lip movements. The acquired audio data is input into a specific deep learning network to extract facial motion parameters synchronized with the audio. These facial motion parameters control changes in facial expressions within a 3D deformation model. By combining the facial expression parameters from the 3D deformation model with the facial motion parameters extracted from the audio data, the final facial motion parameters used for animation generation are calculated.

[0069] The acquired audio data is input into an audio-driven facial motion parameter extraction network (such as ExpNet). The network analyzes the audio data and extracts facial motion parameters (β) synchronized with the audio. 1...n Then, by combining the facial expression parameter β in the three-dimensional deformation model, the final facial motion parameters used to drive dynamic facial expression changes are calculated.

[0070] In one specific implementation, the data acquisition method based on the local device includes: acquiring portrait images via a local camera to generate a 3D deformation model; converting the acquired portrait images into a 3D deformation model including facial expressions, head posture, and hand posture using 3DMM technology; acquiring audio data in real time via a local microphone; inputting the acquired audio data into an audio-driven facial motion parameter extraction network, and combining it with facial expression parameters from the 3D deformation model to calculate and generate facial motion parameters used to drive dynamic facial expression changes.

[0071] In another specific implementation, the remote data processing method includes: retrieving a pre-stored portrait image from a cloud server via a network; converting the image into a three-dimensional deformation model using 3DMM technology, the model containing facial expression, head pose, and hand pose parameters; transmitting a pre-recorded audio file via the network; inputting the audio file into an audio-driven facial motion parameter extraction network, and combining it with facial expression parameters to calculate the final facial motion parameters for synchronizing the facial expression changes in the animation. Through a high-precision three-dimensional deformation model, a high degree of synchronization between dynamic expressions and audio is achieved. This system offers flexible data input methods and is widely applicable to scenarios such as virtual reality, film and television production, and virtual assistants, significantly improving the efficiency and quality of animation production and ensuring the system's practicality and superiority.

[0072] S20, the variational autoencoder for posture generation calculates the head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters and audio data;

[0073] In this embodiment, the variational autoencoder for pose generation is a deep learning model used to generate and predict pose motion parameters. It can be based on various variational autoencoder (VAE) architectures, including but not limited to PoseVAE. This autoencoder can process input head pose parameters, hand pose parameters, and audio data, and generate pose motion parameters that match these input data.

[0074] Head and hand pose motion parameters represent the dynamic changes of the head and hands in the animation, encompassing posture information such as rotation and translation. Motion parameters generated by a variational autoencoder ensure that pose changes in the animation are synchronized with the input data (such as audio).

[0075] In one specific implementation, the variational autoencoder for pose generation, based on multiple VAE models, involves acquiring head pose parameters, hand pose parameters, and audio data via a local device or network. The input data is then fed into various different variational autoencoder models, including but not limited to PoseVAE, ConditionalVAE (CVAE), or other VAE models suitable for pose generation. The pose motion parameters generated by each model can be compared or fused to obtain the optimal pose motion result. Using the generated pose motion parameters, the head and hand movements of the virtual character are driven, ensuring that the expressions and actions in the animation are highly consistent with the input data.

[0076] In another specific implementation, the variational autoencoder for pose generation, based on an adaptive VAE model, involves acquiring real-time or pre-stored head pose parameters, hand pose parameters, and audio data. An adaptive variational autoencoder model is used, which dynamically adjusts its structure and parameters based on the characteristics of the input data (such as intonation variations in the audio, pose complexity, etc.) to generate optimal pose motion parameters. The model may include adaptive PoseVAE, adaptive CVAE, or other VAE variants. The generated motion parameters are then applied to the animation of the virtual character, achieving pose changes highly synchronized with the input data.

[0077] In other specific implementations, the variational autoencoder for pose generation based on a multimodal VAE model is achieved by combining multiple input sources, such as head pose parameters, hand pose parameters, audio data, and other auxiliary information (such as ambient sound, background information, etc.). Using a multimodal VAE model, which can process various types of data inputs, richer and more complex pose motion parameters can be generated. This model can capture more details and achieve more realistic animation effects. By utilizing the motion parameters generated by multimodal generation, complex dynamic animations with more detail can be generated for virtual characters.

[0078] Through the above steps, precise generation of head and hand posture motion parameters is achieved, solving the problem of insufficient synchronization between posture and audio in traditional methods. It can flexibly adapt to different input conditions, ensuring high accuracy and consistency of posture motion parameters, effectively improving the naturalness and real-time performance of virtual character animation.

[0079] S30, extract the appearance features of the portrait image, and perform standard key point estimation on the portrait image to generate standard key points, and map the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters and hand pose motion parameters to parameter representations in 2D view.

[0080] In this embodiment, appearance features refer to visual features extracted from the portrait image, such as color, texture, and shape. These features are used to ensure that the generated animation retains visual consistency with the original portrait during the 2D and 3D mapping process.

[0081] Standard keypoints refer to a set of important reference points defined in a portrait image, such as the positions of the eyes, nose, and mouth. Standard keypoint estimation generates standardized reference points for subsequent pose and expression mapping.

[0082] The process maps head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters from 3D space onto a 2D view. This involves projecting 3D information onto a 2D plane to accurately represent these pose and expression changes in 2D animation.

[0083] In one specific implementation, the parameter mapping method based on standard keypoint estimation and mapping of 2D images includes: extracting appearance features from the input portrait image, ensuring these features remain consistent throughout subsequent 2D and 3D mapping processes; using deep learning or computer vision algorithms to estimate standard keypoints in the portrait image, generating a set of standardized reference points for subsequent pose and expression mapping; and mapping head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters onto a 2D view. This process may involve perspective projection or other geometric transformations to ensure that 3D information is accurately reflected in the 2D plane.

[0084] In another specific implementation, the parameter mapping method combines a hybrid approach of 3D modeling and 2D mapping. This may include: generating a comprehensive appearance feature vector by combining appearance features from the 3D deformation model and visual features from the portrait image; performing standard keypoint estimation on both the 3D model and the 2D image, and aligning them using a registration algorithm to ensure mapping accuracy; using the registered keypoint information to map the 3D pose and expression parameters onto a 2D plane; and achieving a more accurate parameter representation in the 2D view through multi-view projection technology.

[0085] In other specific implementations, the parameter mapping method based on dynamic mapping of real-time data streams includes: extracting appearance features from a continuous real-time portrait image data stream to support the mapping process in dynamic scenes; estimating key points in real time and dynamically updating them to cope with rapid changes in the scene; and combining the real-time input data to project 3D parameters onto a 2D view and generate a 2D animation synchronized with real-time audio and motion.

[0086] By accurately mapping 3D pose and expression parameters to parameter representations in a 2D view, the technical challenge of representing complex 3D movements and expressions in 2D animation is effectively solved. This ensures the consistency between the visual effects in the animation and the original portrait, improves the accuracy and realism of animation production, and is suitable for applications that require a high degree of realism, such as virtual character generation and animation production.

[0087] S40 inputs the mapped parameter representation, standard key points, and appearance features into the image generator to generate a multi-pose portrait animation that includes hand gestures and dynamic expressions.

[0088] In this embodiment, the mapped parameters are a set of values ​​obtained by projecting three-dimensional pose and expression parameters onto a two-dimensional view. These parameters represent the poses of the head and hands, as well as the dynamic expression information of the face, and are used to generate animation on a two-dimensional plane after mapping.

[0089] Standard key points are a set of reference points extracted from a portrait image, such as the eyes, nose, and mouth, representing important facial features. Standard key points are used to ensure the accuracy of facial expressions in animation.

[0090] An image generator is a model or algorithm that can generate images based on input parameters. It takes mapped parameter representations, standard key points, and appearance features as input, and outputs multi-pose portrait animations that include hand gestures and dynamic expressions.

[0091] In one specific implementation, the image generation method based on a predefined model includes: inputting a two-dimensional parameter representation obtained by mapping pose parameters, expression parameters, and motion parameters, along with extracted standard key points and appearance features, into a predefined image generation model. The image generator generates a dynamic two-dimensional portrait animation based on the input data, including hand pose changes and dynamic expressions. The generator utilizes the input standard key points to ensure the accuracy of facial expressions and uses appearance features to maintain consistency with the original portrait. The output is a dynamic portrait animation containing various poses and expression changes, which can be applied to virtual avatars, animation production, or real-time interactive scenarios.

[0092] In another specific implementation, deep learning-based image generation involves inputting a mapped two-dimensional parameter representation, standard keypoints, and appearance features into a trained deep learning model (such as a GAN or VAE). The deep learning model generates high-quality dynamic portrait animations based on the input data. By learning complex facial expression and pose change patterns, the model can generate more realistic animation effects. The generated animations feature highly realistic facial expressions and hand pose changes, and can be used in real-time applications in film and television production or virtual reality.

[0093] In other specific implementations, image generation based on hybrid technologies includes combining rule-based mapping methods and deep learning algorithms, inputting parameter representations, standard key points, and appearance features into the image generation system. The system uses a combination of technologies to generate multi-pose portrait animations that include hand gestures and dynamic expressions. By comprehensively utilizing rules and deep learning, the system can improve generation speed and quality while maintaining animation realism. The generated animations can be flexibly applied in different scenarios, such as virtual meetings, social media, or animation production.

[0094] By inputting mapped parameter representations, standard key points, and appearance features into the image generator, multi-pose portrait animations incorporating hand gestures and dynamic expressions can be effectively generated, solving the problems of multimodal data fusion and insufficient realism. This improves the accuracy and real-time performance of animation generation, making it suitable for fields such as virtual avatars, film and television production, and virtual reality, significantly enhancing user experience and animation production efficiency.

[0095] This invention transforms portrait images into 3D deformation models containing parameters of facial expressions, head posture, and hand posture. Combined with an audio data-driven facial motion parameter extraction network, it accurately generates facial expressions and posture changes that match the sound, thus enhancing the realism and naturalness of the animation. By using a variational autoencoder for posture generation to calculate the motion parameters of the head and hands, it achieves synchronization between facial expressions and postures and audio. Furthermore, by estimating standard keypoints of the appearance features of the portrait image and inputting the mapped parameters into an image generator, it achieves a natural combination of dynamic expressions and hand postures, thereby generating richer and more realistic multi-pose portrait animations.

[0096] In one embodiment, S20 above includes:

[0097] S201, Select one or more motion style parameters from a predefined motion style library, the motion style parameters being used to determine the style characteristics of head and hand movements;

[0098] S202, the head posture parameters, hand posture parameters, motion style parameters and audio data are input into the variational autoencoder model for posture generation;

[0099] S203, the variational autoencoder model for posture generation generates corresponding head posture motion parameters and hand posture motion parameters based on the input data through encoding and decoding processes.

[0100] In this embodiment, the variational autoencoder for pose generation is a deep learning model used to generate pose motion parameters that match the input data. By encoding and decoding the input head pose parameters, hand pose parameters, motion style parameters, and audio data, the model can generate accurate head and hand pose motion parameters.

[0101] Head pose parameters and hand pose parameters describe the position and orientation of the head and hands in three-dimensional space, including information such as rotation and translation. They are important basic data for generating dynamic poses.

[0102] Motion style parameters describe the stylistic features of head and hand movements, such as speed, amplitude, and smoothness. By introducing motion style parameters, the model can generate poses that conform to a specific style, making the generated animations more vivid and personalized.

[0103] Audio data typically includes speech or other sound signals used for synchronization with posture movements. By combining audio data, the model can generate posture changes consistent with the speech content.

[0104] In one specific implementation, the method of generating motion style parameters based on predefined motion styles for posture includes: the system selecting one or more motion style parameters from a predefined motion style library. These parameters may include speed, intensity, and smoothness. Head posture parameters, hand posture parameters, the selected motion style parameters, and audio data are input into a variational autoencoder model for posture generation. The model generates corresponding head posture motion parameters and hand posture motion parameters based on the input data through encoding and decoding processes. This method is suitable for generating dynamic animation scenes that require a specific style, such as advertisements and game character animations.

[0105] In another specific implementation, the motion style parameters are generated based on real-time adjustments. This involves dynamically adjusting the motion style parameters by collecting user or environmental input (such as gestures and facial expressions) in real time. Head posture parameters, hand posture parameters, dynamically adjusted motion style parameters, and audio data are input into a variational autoencoder model. The model calculates the head and hand posture motion parameters based on the motion style parameters generated from the dynamic input, ensuring that the animation responds to user input in real time. This approach is suitable for real-time interactive applications, such as character animation in virtual reality or virtual avatars in real-time performances.

[0106] In other specific implementations, the motion style parameters generated based on multimodal fusion include: generating comprehensive motion style parameters by combining multimodal inputs (such as visual and haptic feedback). The multimodal-generated motion style parameters, along with head pose parameters, hand pose parameters, and audio data, are then input into a variational autoencoder model. By integrating the multimodal inputs, the model generates more complex and nuanced head and hand pose motion parameters, making the animation more expressive. This approach is suitable for applications requiring high immersion and complex interactions, such as advanced virtual reality experiences or complex film and television special effects production.

[0107] This embodiment effectively solves the problem of monotonous and impersonal motion styles in traditional animation generation by combining motion style parameters and using a variational autoencoder model for posture generation to calculate head and hand posture motion parameters. It can generate dynamic postures that conform to specific styles based on different input data and requirements, making the generated animations more vivid and personalized. This is suitable for fields requiring dynamic posture generation, such as game development, virtual assistants, and film and television production. Simultaneously, by incorporating audio data, it ensures a high degree of synchronization between motion and audio, improving the user's interactive experience and the expressiveness of the animation.

[0108] In one embodiment, prior to S20 above, the following is also included:

[0109] S204, collects a standard dataset containing head pose parameters, hand pose parameters, motion style parameters and audio data;

[0110] S205, Define a weighted loss function, which is composed of mean squared error, KL divergence and adversarial loss adjusted for weights;

[0111] S206, the variational autoencoder for pose generation is a PoseVAE model. The PoseVAE model is trained using the weighted loss function and the standard dataset. The parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

[0112] In this embodiment, PoseVAE is a variational autoencoder (VAE) model specifically designed to generate posture motion parameters. This model can process input posture parameters, motion style parameters, and audio data to generate corresponding motion parameters.

[0113] The weighted loss function is the objective function used to guide model training. Here, the loss function consists of three parts: mean squared error (MSE), Kullback-Leibler divergence, and adversarial loss. The mean squared error is used to minimize the difference between the generated results and the real data, the Kullback-Leibler divergence is used to measure the difference between the generated distribution and the real distribution, and the adversarial loss is used to enhance the realism of the generated results.

[0114] The PoseVAE model is trained using a defined weighted loss function, enabling it to generate pose motion parameters that more closely resemble real-world data. The training process optimizes the model parameters by minimizing the loss function, ultimately resulting in an optimized PoseVAE model.

[0115] In one specific implementation, the model training method based on the standard dataset includes...

[0116] Data collection: Collect a standard dataset containing head pose parameters, hand pose parameters, motion style parameters, and audio data.

[0117] Define the weighted loss function: Set a weighted loss function, where mean squared error is used to minimize the difference between the attitude motion parameters and the real data, KL divergence is used to ensure the stability of the generated distribution, and adversarial loss is used to enhance the realism of the generated results.

[0118] Training the PoseVAE model: Input the standard dataset into the PoseVAE model, train the model using a weighted loss function, optimize the model parameters, and make the generated pose motion parameters highly consistent with the real data.

[0119] Application scenarios: This approach is suitable for applications that require high-precision posture and motion generation, such as animation production or advanced virtual reality experiences.

[0120] In another specific implementation, the model training method based on adaptive data includes:

[0121] Data collection and preprocessing: Collect diverse data from different scenarios and preprocess it to form an adaptive dataset.

[0122] Define an adaptive weighted loss function: adjust the weights of the weighted loss function according to different data characteristics to adapt to different types of data input.

[0123] Training the PoseVAE model: The PoseVAE model is trained using an adaptive dataset and a weighted loss function to generate more accurate and stable pose motion parameters in different application scenarios.

[0124] Application scenarios: This method is suitable for diverse scenarios, such as cross-domain virtual character animation generation or film and television production of various styles.

[0125] In other specific implementations, model optimization based on adversarial training includes:

[0126] Collect and generate adversarial examples: Generate data samples containing unusual or complex motion styles for adversarial training.

[0127] Define an enhanced weighted loss function: Based on the standard weighted loss function, the weights of the adversarial loss are increased to improve the model's generative ability in complex scenarios.

[0128] Training the PoseVAE model: Through adversarial training and optimization of the weighted loss function, the generation effect of the PoseVAE model is further enhanced, especially its performance in complex and abnormal pose motions.

[0129] Application scenarios: Suitable for animation scenarios with high complexity and irregularity, such as special effects production in science fiction movies or high-difficulty motion capture.

[0130] This embodiment improves the accuracy and realism of the generated results by optimizing the weighted loss function before calculating the pose motion parameters using the PoseVAE model, thus solving the error and instability problems existing in traditional pose generation methods. The optimized PoseVAE model can generate pose motion parameters that highly match real data in various complex scenarios, making it suitable for multiple fields such as film and television production, virtual assistants, and game development, greatly improving the quality of animation generation and user experience.

[0131] In one embodiment, S20 above includes:

[0132] S207, The head posture parameters, hand posture parameters, motion style parameters and audio data are input into the improved PoseVAE model to generate head posture change and hand posture change.

[0133] S208, Two VAE encoders with non-shared parameters are used to encode the head pose change and the hand pose change respectively, and the head pose change and the hand pose change are mapped to two different latent spatial distributions.

[0134] S209, the two latent spatial distributions are fused by Gaussian mixing and resampled to obtain a new latent spatial distribution. The new latent spatial distribution is then decoded using a VAE decoder to generate the head posture motion parameters and hand posture motion parameters.

[0135] In this embodiment, the improved PoseVAE model is an optimized variational autoencoder model based on the standard PoseVAE model, which can more accurately generate head and hand pose variations. This model accepts various input data, including head pose parameters, hand pose parameters, motion style parameters, and audio data.

[0136] Head pose variation and hand pose variation refer to the dynamic variations generated by the improved PoseVAE model, representing the degree of change in head and hand pose. These variations will be used in further encoding and decoding processes.

[0137] Two independent VAE encoders are used to encode head pose changes and hand pose changes separately. These two encoders do not share parameters to ensure that head and hand pose changes are processed independently, generating different latent space representations.

[0138] The encoded head and hand pose changes are mapped to two different latent spatial distributions. These two latent spatial distributions are then fused using Gaussian mixture and resampled to generate a new latent spatial distribution that more accurately represents the overall pose changes.

[0139] The VAE decoder is responsible for decoding the fused latent spatial distribution into the final head pose motion parameters and hand pose motion parameters. These parameters will be used to generate the dynamic poses of the virtual character.

[0140] In one specific implementation, the pose generation method based on a standard dataset includes:

[0141] Input data: Head pose parameters, hand pose parameters, motion style parameters, and audio data are extracted from a standard dataset.

[0142] Generate changes: Input these parameters into the improved PoseVAE model to generate head pose changes and hand pose changes.

[0143] Encoding process: Two independent VAE encoders are used to encode the head pose changes and hand pose changes respectively, and map them to their respective latent spatial distributions.

[0144] Fusion and Decoding: The two latent space distributions are fused using Gaussian mixture modeling, and a new latent space distribution is generated by resampling. This latent space is then decoded using a VAE decoder to generate the final head pose motion parameters and hand pose motion parameters.

[0145] Application scenarios: Suitable for animation production or virtual character generation that requires precise control of posture changes.

[0146] In another specific implementation, the pose generation method based on multimodal input includes:

[0147] Input data: Combine multimodal inputs (such as visual, audio, haptic feedback, etc.) to obtain head posture parameters, hand posture parameters, motion style parameters and audio data.

[0148] Generate Variations: Input these multimodal data into the improved PoseVAE model to generate pose variations for the head and hands.

[0149] Encoding process: These changes are encoded independently using two VAE encoders to ensure that the pose changes of the head and hands are accurately represented in the latent space.

[0150] Fusion and Decoding: These independent latent space distributions are fused using Gaussian mixture modeling, and a new latent space is generated by resampling. Subsequently, a VAE decoder is used to decode them into head pose motion parameters and hand pose motion parameters.

[0151] Application scenarios: Suitable for scenarios that require the combination of multiple input data, such as virtual reality and augmented reality.

[0152] In other specific implementations, the attitude generation method based on real-time data streams includes:

[0153] Input data: Head pose parameters, hand pose parameters, motion style parameters, and audio data are obtained from the real-time data stream.

[0154] Generate changes: Input these real-time data into the improved PoseVAE model to generate changes in head and hand poses in real time.

[0155] Encoding process: Two VAE encoders are used to encode the real-time generated pose changes and map them into the latent space.

[0156] Fusion and Decoding: The latent space distribution is fused in real time through Gaussian mixture and a new latent space is generated by resampling, which is used to decode the real-time attitude motion parameters.

[0157] Application scenarios: Suitable for real-time interactive applications, such as real-time facial expressions and gestures generation in virtual meetings.

[0158] This embodiment, through an improved PoseVAE model and an independent VAE encoder, can accurately generate head and hand pose motion parameters, solving the accuracy and consistency problems of pose generation in traditional methods. By employing Gaussian mixture and resampling techniques, this method can generate highly natural and coherent pose motions even in complex scenes and with multimodal inputs. It is suitable for animation production, virtual reality, and real-time interactive applications, effectively improving the quality of generated animations and the user experience.

[0159] In one embodiment, in S10 above, converting the portrait image into a three-dimensional deformation model includes:

[0160] S101, preprocess the portrait image, including image denoising, illumination correction and contrast adjustment, to improve the quality of the portrait image;

[0161] S102, using monocular 3D face recognition technology, the preprocessed portrait image is converted into the three-dimensional deformation model;

[0162] S103, extract facial expression parameters, head posture parameters, and hand posture parameters from the three-dimensional deformation model.

[0163] In this embodiment, the image is preprocessed before being converted into a 3D deformable model. The preprocessing steps include image denoising, illumination correction, and contrast adjustment. These operations aim to improve image quality and ensure more accurate subsequent 3D face recognition and model generation.

[0164] Monocular 3D face recognition technology converts pre-processed two-dimensional portrait images into three-dimensional deformation models. This technology uses two-dimensional images captured by a single camera and employs deep learning and computer vision algorithms to infer the three-dimensional facial structure.

[0165] Facial expression parameters, head pose parameters, and hand pose parameters are extracted from the generated 3D deformed model. These parameters describe the dynamic facial expressions and the poses of the head and hands, which are used for subsequent animation generation or other applications.

[0166] In one specific implementation, the method for generating a 3D deformation model based on a standard image includes: preprocessing the portrait image captured from a camera, including image denoising to reduce noise, illumination correction to eliminate the effects of uneven illumination, and contrast adjustment to enhance image details. Using monocular 3D face recognition technology, the preprocessed portrait image is converted into a 3D deformation model. A deep learning algorithm is used to infer the 3D facial structure in the image. Key facial expression parameters, head pose parameters, and hand pose parameters are extracted from the generated 3D deformation model. These parameters will be used for subsequent animation generation and expression recognition applications.

[0167] In another specific implementation, the method for generating a 3D deformation model based on multi-angle images includes: preprocessing portrait images acquired from multiple angles to ensure that each image undergoes denoising, illumination correction, and contrast adjustment. Combining the preprocessed images from multiple angles, a more accurate 3D deformation model is generated using monocular 3D face recognition technology. The accuracy of the 3D model is improved through comprehensive processing of multiple perspectives. Facial expression parameters, head pose parameters, and hand pose parameters are extracted from the comprehensively generated 3D deformation model for use in more complex animation and expression control.

[0168] In other specific implementations, the method for generating a 3D deformation model based on a real-time image stream includes: rapidly preprocessing each frame of the portrait image in the real-time image stream to ensure that each frame in the image stream maintains high quality; applying monocular 3D face recognition technology in real time to convert each preprocessed image frame into a 3D deformation model and continuously generating 3D dynamic changes in the face; and extracting facial expression parameters, head pose parameters, and hand pose parameters from the 3D deformation model in real time to drive virtual characters or realize other real-time interactive applications.

[0169] This embodiment achieves high-precision conversion from 2D images to 3D structures by converting portrait images into 3D deformation models, solving the model distortion problem caused by poor image quality in traditional methods. The preprocessing step significantly improves the quality of the input image, ensuring a more accurate generated 3D model. Combined with monocular 3D face recognition technology, a complete 3D structure can be inferred from a single camera input, saving the cost of multiple cameras or complex equipment. Finally, the extracted facial expression parameters, head pose parameters, and hand pose parameters provide accurate data support for subsequent animation generation and real-time applications, applicable to various fields such as virtual reality and film production.

[0170] In one embodiment, in S10 above, inputting the audio data into an audio-driven facial motion parameter extraction network and calculating facial motion parameters based on the facial expression parameters includes:

[0171] S104, extract features from the audio data, including audio features such as emotion, tone, and rhythm;

[0172] S105, the extracted audio features and facial expression parameters are input into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, which are used to generate dynamic facial expressions.

[0173] In this embodiment, feature extraction is performed on the audio data before inputting it into the audio-driven facial motion parameter extraction network. These features include key information such as emotion, tone, and rhythm, used to capture emotional changes, speaking rhythm, and intonation in the audio. These features directly affect the generated facial expressions.

[0174] Audio-driven facial motion parameter extraction networks are deep learning networks such as ExpNet, specifically designed to compute facial motion parameters based on input audio features and facial expression parameters. This network combines features from the audio data with static information about facial expressions to generate dynamic facial expressions synchronized with the audio content.

[0175] Facial motion parameters are key data generated by the network to drive dynamic facial expressions. These parameters control the movement of different parts of the face, allowing expressions to update dynamically as the audio changes.

[0176] In one specific implementation, the method for generating facial motion based on standard audio data includes:

[0177] Audio feature extraction: Feature extraction is performed on standard audio data to extract features such as emotion, tone, and rhythm. These features will be used to generate facial expressions that are consistent with the speaker's emotional state and manner of expression.

[0178] Input to the network: The extracted audio features, along with pre-computed facial expression parameters, are input into the audio-driven facial motion parameter extraction network.

[0179] Calculate facial motion parameters: The network generates facial motion parameters based on the input data. These parameters are used to drive the dynamic changes in facial expressions, synchronizing them with the audio content.

[0180] Application scenarios: Suitable for animation scenarios that require a high degree of matching with audio content, such as lip-syncing and facial expression generation for virtual characters.

[0181] In another specific implementation, the facial motion generation method based on multimodal fusion includes:

[0182] Multimodal feature extraction: Feature extraction is performed on multimodal data (such as audio and video), especially extracting emotion, tone and rhythm features in audio, and combining them with facial expression information in visual data.

[0183] Input to the network: Multimodal features and facial expression parameters are input together into an audio-driven facial motion parameter extraction network to generate more accurate and natural facial motion parameters.

[0184] Calculate facial motion parameters: The network generates more complex facial motion parameters based on this rich input data, which are used for more realistic facial expression changes.

[0185] Application scenarios: Suitable for film and television production and advanced virtual reality applications, capable of generating multi-layered facial expression changes.

[0186] In other specific implementations, methods for generating facial motion based on real-time audio data include:

[0187] Real-time feature extraction: Extract emotion, tone, and rhythm features from real-time audio streams and process these features quickly.

[0188] Input to the network: The real-time extracted audio features and the current facial expression parameters are input into the audio-driven facial motion parameter extraction network.

[0189] Calculate facial motion parameters: The network calculates facial motion parameters in real time, enabling facial expressions in virtual characters or applications to change in sync with real-time audio content.

[0190] Application scenarios: Suitable for real-time interactive applications, such as character animation generation in virtual meetings, ensuring synchronization with the user's voice.

[0191] This embodiment effectively solves the problem of asynchronous expression and audio in traditional methods by combining audio data with facial expression parameters into an audio-driven facial motion parameter extraction network. Audio feature extraction ensures that dynamic changes in facial expressions reflect the emotions, tone, and rhythm in the audio. The generated facial motion parameters enable natural synchronization between facial expressions and audio content, making it suitable for scenarios such as virtual character generation, animation production, and virtual meetings, significantly improving the expressiveness and user experience of animation and interactive applications.

[0192] In one embodiment, in S30 above, mapping the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view includes:

[0193] S301, the head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters are mapped to a 2D plane through perspective projection to generate a preliminary parameter representation in a 2D view;

[0194] S302, based on facial expression parameters and facial motion parameters, adjust the position of standard key points in the 2D view to accurately reflect the changes in facial expression and posture on the 2D plane;

[0195] S303, based on the adjusted standard key point positions and the parameter representation in the preliminary 2D view, uses an interpolation algorithm to smooth the head pose motion parameters and hand pose motion parameters, generating the final parameter representation in the 2D view, which is used to drive the generation of multi-pose portrait animation.

[0196] In this embodiment, perspective projection is a technique that maps points in three-dimensional space onto a two-dimensional plane. During this process, head pose parameters, hand pose parameters, facial expression parameters, and facial motion parameters are mapped onto the two-dimensional plane through perspective projection, generating a preliminary parameter representation in a 2D view. This step accurately converts information in three-dimensional space into a two-dimensional representation, preserving a sense of depth and space.

[0197] Standard key points are reference points on the face (such as the eyes, nose, and mouth). The positions of these points on the two-dimensional plane need to be adjusted according to facial expression parameters and facial motion parameters to accurately reflect changes in facial expression and posture on the two-dimensional plane. Adjusting the positions of these key points ensures that the final generated two-dimensional view has correct expression and posture representation.

[0198] Interpolation algorithms are used to smooth head and hand pose motion parameters, ensuring a smooth and natural transition between parameter representations in a 2D view. By calculating intermediate values ​​between different parameter representations, interpolation algorithms make the final animation more coherent and natural in its dynamic performance.

[0199] In one specific implementation, the method for generating a 2D view based on a static image includes:

[0200] Perspective projection mapping: The head pose parameters, hand pose parameters, facial expression parameters, and facial motion parameters in a static image are mapped to a two-dimensional plane through perspective projection to generate a preliminary parameter representation in a 2D view.

[0201] Standard key point adjustment: Adjust the position of standard key points in the 2D view according to facial expression parameters and facial motion parameters to accurately reflect changes in facial expression and posture.

[0202] Interpolation processing: The head pose motion parameters and hand pose motion parameters are smoothed using an interpolation algorithm to generate the final parameter representation in the 2D view.

[0203] Application scenarios: Suitable for portrait animation generation of static images, such as character performances in artworks or animation clips.

[0204] In another specific implementation, the method of generating 2D views based on dynamic video includes:

[0205] Perspective projection mapping: Perspective projection is performed on the head and hand pose parameters, facial expression parameters, and facial motion parameters in dynamic video frames to generate a parameter representation in the preliminary 2D view of each frame.

[0206] Standard key point adjustment: The position of standard key points is dynamically adjusted based on facial expressions and motion parameters in each frame to reflect changes in expression and posture throughout the animation.

[0207] Interpolation processing: Interpolation algorithms are used to smooth the pose motion parameters between consecutive frames to ensure a natural transition between frames and generate the parameter representation in the final 2D view.

[0208] Application scenarios: Suitable for character animation generation in animated films or games, especially for high-quality poses and expressions in consecutive frames.

[0209] In other specific implementations, methods for generating 2D views based on real-time interaction include:

[0210] Perspective projection mapping: In real time, the user's head posture, hand posture, facial expression and motion parameters are mapped to a two-dimensional plane to generate a preliminary parameter representation in a 2D view.

[0211] Standard key point adjustment: Based on real-time captured facial expression and motion data, the position of standard key points is dynamically adjusted to ensure that the real-time generated animation is synchronized with the user's facial expressions and posture.

[0212] Interpolation processing: The real-time generated posture motion parameters are smoothed using an interpolation algorithm to generate the parameter representation in the final 2D view, which is used to drive the dynamic performance of the virtual avatar.

[0213] Application scenarios: Suitable for real-time virtual meetings, virtual assistants, or other application scenarios that require real-time interaction.

[0214] This embodiment effectively solves the problem of converting from three-dimensional space to two-dimensional representation by mapping head pose parameters, hand pose parameters, facial expression parameters, and facial motion parameters to a two-dimensional view. Perspective projection ensures the preservation of depth and spatial sense, while adjustments to standard key points ensure accurate representation of facial expressions and poses in the two-dimensional view. The use of interpolation algorithms further improves the smoothness and naturalness of the animation. The generated two-dimensional view can be used for high-quality portrait animation generation and is suitable for various fields such as animation production, virtual assistants, and real-time interaction.

[0215] The present invention also provides a multi-pose portrait animation generation device, referring to Figure 2 , Figure 2 This is a functional module diagram of a preferred embodiment of the multi-pose portrait animation generation device of the present invention. The multi-pose portrait animation generation device includes:

[0216] The 3D model generation module acquires a portrait image, converts the portrait image into a 3D deformation model, the 3D deformation model includes facial expression parameters, head posture parameters, and hand posture parameters, acquires audio data, inputs the audio data into an audio-driven facial motion parameter extraction network, and calculates facial motion parameters based on the facial expression parameters.

[0217] The posture generation module uses a variational autoencoder to calculate head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, and audio data.

[0218] The 2D mapping module extracts the appearance features of the portrait image, performs standard key point estimation on the portrait image to generate standard key points, and maps the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view.

[0219] The multi-pose animation generation module takes the mapped parameter representation, standard key points, and appearance features as inputs to the image generator to generate multi-pose portrait animations that include hand poses and dynamic expressions.

[0220] The specific implementation of the multi-pose portrait animation generation device of the present invention is basically the same as the embodiments of the multi-pose portrait animation generation method described above, and will not be repeated here.

[0221] This invention also provides a multi-pose portrait animation generation device, such as... Figure 3 As shown, the multi-pose portrait animation generation device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0222] Those skilled in the art will understand that Figure 3 The hardware structure of the multi-pose portrait animation generation device shown in the figure does not constitute a limitation on the multi-pose portrait animation generation device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0223] like Figure 3 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a multi-pose portrait animation generation program. The operating system is a program that manages and controls the multi-pose portrait animation generation device and software resources, supporting the operation of the network communication module, the user interface module, the multi-pose portrait animation generation program, and other programs or software. The network communication module manages and controls the network interface 1004; the user interface module manages and controls the user interface 1003.

[0224] exist Figure 3 In the hardware structure of the multi-pose portrait animation generation device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the multi-pose portrait animation generation program stored in the memory 1005 and perform the same operation as the multi-pose portrait animation generation method.

[0225] The specific implementation of the multi-pose portrait animation generation device of the present invention is basically the same as the embodiments of the multi-pose portrait animation generation method described above, and will not be repeated here.

[0226] Furthermore, this embodiment of the invention also proposes a computer storage medium storing a multi-pose portrait animation generation program, which, when executed by a processor, implements the steps of the multi-pose portrait animation generation method described above.

[0227] The specific implementation of the computer storage medium of the present invention is basically the same as the embodiments of the above-described multi-pose portrait animation generation method, and will not be repeated here.

[0228] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit and scope of the claims. All equivalent structural or procedural transformations made using the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are within the protection scope of the present invention.

[0229] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for generating multi-pose portrait animation, characterized in that, Includes the following steps: A portrait image is acquired and converted into a three-dimensional deformation model, which includes facial expression parameters, head posture parameters, and hand posture parameters. Audio data is acquired and input into an audio-driven facial motion parameter extraction network. Facial motion parameters are calculated based on the facial expression parameters. The variational autoencoder, which generates the posture, calculates the head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, and audio data. Extract the appearance features of the portrait image, and perform standard key point estimation on the portrait image to generate standard key points. Map the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters, and hand pose motion parameters to parameter representations in a 2D view. The mapped parameter representation, standard key points, and appearance features are input into the image generator to generate multi-pose portrait animations that include hand gestures and dynamic expressions.

2. The multi-pose portrait animation generation method as described in claim 1, characterized in that, The variational autoencoder, using posture generation, calculates head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, and audio data, including: Select one or more motion style parameters from a predefined motion style library, the motion style parameters being used to determine the style characteristics of head and hand movements; The head posture parameters, hand posture parameters, motion style parameters, and audio data are input into the variational autoencoder model for posture generation. The variational autoencoder model for posture generation generates corresponding head posture motion parameters and hand posture motion parameters based on the input data through encoding and decoding processes.

3. The multi-pose portrait animation generation method as described in claim 1, characterized in that, Before using the variational autoencoder generated by posture generation to calculate the head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, motion style parameters, and audio data, the following steps are also included: Collect a standard dataset containing head pose parameters, hand pose parameters, motion style parameters, and audio data; Define a weighted loss function, which is composed of mean squared error, KL divergence and adversarial loss adjusted for weights; The variational autoencoder for pose generation is the PoseVAE model. The PoseVAE model is trained using the weighted loss function and the standard dataset. The parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

4. The multi-pose portrait animation generation method as described in claim 3, characterized in that, The variational autoencoder, using posture generation, calculates head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters, motion style parameters, and audio data, including: The head posture parameters, hand posture parameters, motion style parameters, and audio data are input into the improved PoseVAE model to generate head posture change and hand posture change. Two VAE encoders with non-shared parameters are used to encode the head pose change and the hand pose change respectively, mapping the head pose change and the hand pose change to two different latent spatial distributions. The two latent spatial distributions are fused using Gaussian mixing and resampled to obtain a new latent spatial distribution. The new latent spatial distribution is then decoded using a VAE decoder to generate the head posture motion parameters and hand posture motion parameters.

5. The multi-pose portrait animation generation method as described in claim 1, characterized in that, Converting the portrait image into a three-dimensional deformation model includes: The portrait image is preprocessed, including image denoising, illumination correction and contrast adjustment, to improve the quality of the portrait image; Using monocular 3D face recognition technology, the preprocessed portrait image is converted into the three-dimensional deformation model; The facial expression parameters, head posture parameters, and hand posture parameters are extracted from the three-dimensional deformation model.

6. The multi-pose portrait animation generation method as described in claim 1, characterized in that, The audio data is input into an audio-driven facial motion parameter extraction network, and facial motion parameters are calculated based on the facial expression parameters, including: Feature extraction is performed on the audio data to extract audio features including emotion, tone, and rhythm; The extracted audio features and facial expression parameters are input into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, which are used to generate dynamic facial expressions.

7. The multi-pose portrait animation generation method as described in claim 1, characterized in that, Mapping the head pose parameters, hand pose parameters, facial expression parameters, facial movement parameters, head pose movement parameters, and hand pose movement parameters to parameter representations in a 2D view includes: The head posture parameters, hand posture parameters, facial expression parameters, and facial movement parameters are mapped onto a 2D plane through perspective projection to generate a preliminary parameter representation in a 2D view. Based on facial expression parameters and facial motion parameters, the positions of standard key points in the 2D view are adjusted to accurately reflect the changes in facial expression and posture on the 2D plane. Based on the adjusted standard key point positions and the preliminary parameter representation in the 2D view, the head pose motion parameters and hand pose motion parameters are smoothed using an interpolation algorithm to generate the final parameter representation in the 2D view, which is used to drive the generation of multi-pose portrait animation.

8. A multi-pose portrait animation generation device, characterized in that, The multi-pose portrait animation generation device includes: A 3D model generation module is used to acquire portrait images, convert the portrait images into 3D deformation models, the 3D deformation models include facial expression parameters, head posture parameters, and hand posture parameters, acquire audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters based on the facial expression parameters. The posture generation module is used to calculate head posture motion parameters and hand posture motion parameters based on the head posture parameters, hand posture parameters and audio data using a variational autoencoder generated by posture generation. The 2D mapping module is used to extract the appearance features of the portrait image, perform standard key point estimation on the portrait image to generate standard key points, and map the head pose parameters, hand pose parameters, facial expression parameters, facial motion parameters, head pose motion parameters and hand pose motion parameters into parameter representations in a 2D view. The multi-pose animation generation module is used to input the mapped parameter representation, standard key points and appearance features into the image generator to generate multi-pose portrait animations that include hand poses and dynamic expressions.

9. A multi-pose portrait animation generation device, characterized in that, The multi-pose portrait animation generation device includes a memory, a processor, and a multi-pose portrait animation generation program stored in the memory and executable on the processor. When the multi-pose portrait animation generation program is executed by the processor, it implements the steps of the multi-pose portrait animation generation method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The storage medium stores a multi-pose portrait animation generation program, which, when executed by a processor, implements the steps of the multi-pose portrait animation generation method as described in any one of claims 1-7.

Images