Multi-posture portrait animation generation method and device, equipment and storage medium

Abstract

The invention relates to a multi-posture portrait animation generation method. The method comprises the following steps: acquiring a portrait image and converting the portrait image into a three-dimensional deformation model; acquiring audio data, inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters; calculating a head posture motion parameter and a hand posture motion parameter according to the head posture parameter, the hand posture parameter and the audio data by using a posture generation variational auto-encoder; extracting appearance features of the portrait image, and performing standard key point estimation on the portrait image to generate standard key points; mapping the head posture parameter, the hand posture parameter, the facial expression parameter, the facial motion parameter, the head posture motion parameter and the hand posture motion parameter into parameter representation under a 2D view; and finally, inputting the mapped parameter representation, the standard key points and the appearance features into an image generator, and generating a multi-pose portrait animation containing hand postures and dynamic expressions.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method, device, equipment and storage medium for generating a multi-pose portrait animation. Background Art

[0002] In recent years, with the rapid development of computer graphics and artificial intelligence technology, generating realistic animated portraits based on image and audio data has become an important research direction. Generating realistic and natural animated portraits has broad application prospects in the fields of entertainment, film and television production, virtual reality (VR), augmented reality (AR), etc.

[0003] Traditional portrait animation generation methods usually rely on pre-recorded footage or manual adjustments to generate animations. Although these methods can generate high-quality animation effects, they have the following limitations:

[0004] High labor costs: Manually adjusting facial expressions, postures, and motion parameters requires a lot of human intervention, which results in a long production cycle and high costs.

[0005] Lack of flexibility: Pre-recorded materials are often limited to specific scenes and actions and cannot flexibly adapt to different user needs, especially when generating personalized or customized animations.

[0006] Complex data processing: As the complexity of input data increases, such as portrait images from multiple angles and poses and synchronized audio data, existing technologies find it difficult to process and generate dynamic animations that meet the requirements in real time while maintaining high quality.

[0007] In order to overcome the limitations of traditional methods, portrait animation generation technology based on 3D morphable models has received widespread attention. This technology extracts 3D information from 2D portrait images and combines facial expression parameters, posture parameters, etc. to generate an adjustable 3D model. This method can effectively improve the flexibility and quality of animation generation. However, the existing 3D model generation technology also has some shortcomings:

[0008] Limited model generation accuracy: In the process of generating a 3D model from a 2D image, factors such as image quality and lighting conditions may cause the model to be inaccurate, thus affecting the naturalness of the final animation.

[0009] Complexity of motion parameter generation: The dynamic changes of facial expressions and postures require complex algorithms to generate, especially when multimodal data (such as the combination of audio and images) is involved. Existing methods find it difficult to generate accurate motion parameters while ensuring real-time performance. Summary of the invention

[0010] The main purpose of the present invention is to provide a multi-pose portrait animation generation method, device, equipment and storage medium, aiming to solve the technical problem that the prior art is difficult to effectively combine multimodal data when generating multi-pose portrait animation, resulting in the lack of realism and synchronization of the generated animation.

[0011] To achieve the above object, the present invention provides a method for generating a multi-pose portrait animation, comprising:

[0012] Acquire a portrait image, convert the portrait image into a three-dimensional deformable model, the three-dimensional deformable model including facial expression parameters, head posture parameters and hand posture parameters, acquire audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters according to the facial expression parameters;

[0013] Using a posture-generated variational autoencoder to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data;

[0014] Extracting appearance features of the portrait image, estimating standard key points of the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial motion parameters, head posture motion parameters, and hand posture motion parameters into parameter representations under a 2D view;

[0015] The mapped parameter representation, standard key points and appearance features are input into the image generator to generate multi-pose portrait animations including hand poses and dynamic expressions.

[0016] In one embodiment, the head posture motion parameters and the hand posture motion parameters are calculated using a posture-generated variational autoencoder according to the head posture parameters, the hand posture parameters, and the audio data, including:

[0017] Selecting one or more motion style parameters from a predefined motion style library, wherein the motion style parameters are used to determine style features of head and hand movements;

[0018] Inputting the head posture parameters, hand posture parameters, motion style parameters and audio data into the variational autoencoder model for posture generation;

[0019] The variational autoencoder model for posture generation generates corresponding head posture motion parameters and hand posture motion parameters according to input data through encoding and decoding processes.

[0020] In one embodiment, before using the variational autoencoder generated by the posture to calculate the head posture motion parameters and the hand posture motion parameters according to the head posture parameters, the hand posture parameters, the motion style parameters and the audio data, the method further includes:

[0021] Collect standard datasets containing head pose parameters, hand pose parameters, motion style parameters, and audio data;

[0022] A weighted loss function is defined, wherein the weighted loss function is composed of the mean square error, KL divergence, and adversarial loss after adjusting the weights;

[0023] The variational autoencoder for posture generation is a PoseVAE model. The PoseVAE model is trained using the weighted loss function and the standard data set, and the parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

[0024] In one embodiment, the head posture motion parameters and the hand posture motion parameters are calculated using a posture-generated variational autoencoder according to the head posture parameters, the hand posture parameters, the motion style parameters, and the audio data, including:

[0025] Inputting the head posture parameters, hand posture parameters, motion style parameters and audio data into the improved PoseVAE model to generate head posture change and hand posture change;

[0026] Using two VAE encoders that do not share parameters to encode the head posture change and the hand posture change respectively, and mapping the head posture change and the hand posture change to two different latent space distributions;

[0027] The two latent space distributions are fused and resampled through Gaussian mixture to obtain a new latent space distribution, and the new latent space distribution is decoded using a VAE decoder to generate the head posture motion parameters and the hand posture motion parameters.

[0028] In one embodiment, converting the portrait image into a three-dimensional deformable model includes:

[0029] Preprocessing the portrait image, including image denoising, illumination correction and contrast adjustment, to improve the quality of the portrait image;

[0030] Using monocular 3D face recognition technology, converting the preprocessed portrait image into the three-dimensional deformable model;

[0031] Facial expression parameters, head posture parameters and hand posture parameters in the three-dimensional deformable model are extracted.

[0032] In one embodiment, the audio data is input into an audio-driven facial motion parameter extraction network, and facial motion parameters are calculated based on the facial expression parameters, including:

[0033] Performing feature extraction on the audio data to extract audio features including emotion, intonation and rhythm;

[0034] The extracted audio features and the facial expression parameters are input into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, which are used to generate facial dynamic expressions.

[0035] In one embodiment, mapping the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters, and hand posture movement parameters into parameter representations under a 2D view includes:

[0036] Mapping the head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters to a 2D plane through perspective projection to generate a preliminary parameter representation under a 2D view;

[0037] According to the facial expression parameters and facial movement parameters, the standard key point positions in the 2D view are adjusted to accurately reflect the changes of facial expressions and postures on the 2D plane;

[0038] Based on the adjusted standard key point positions and the preliminary parameter representation under 2D view, the head posture motion parameters and hand posture motion parameters are smoothed using an interpolation algorithm to generate the final parameter representation under 2D view, which is used to drive the generation of multi-pose portrait animation.

[0039] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a multi-pose portrait animation generating device, comprising:

[0040] A three-dimensional model generation module is configured to obtain a portrait image, convert the portrait image into a three-dimensional deformable model, wherein the three-dimensional deformable model includes facial expression parameters, head posture parameters, and hand posture parameters, obtain audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters according to the facial expression parameters;

[0041] A posture generation module, using a posture generation variational autoencoder to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data;

[0042] a 2D mapping module, extracting appearance features of the portrait image, estimating standard key points of the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters into parameter representations under a 2D view;

[0043] The multi-pose animation generation module inputs the mapped parameter representation, standard key points and appearance features into the image generator to generate multi-pose portrait animation including hand poses and dynamic expressions.

[0044] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a multi-pose portrait animation generating device, which includes a memory, a processor, and a multi-pose portrait animation generating program stored in the memory and executable on the processor, and the multi-pose portrait animation generating program, when executed by the processor, implements the steps of the multi-pose portrait animation generating method as described above.

[0045] Furthermore, to achieve the above objectives, the present invention also provides a computer storage medium, on which a multi-pose portrait animation generation program is stored, and when the multi-pose portrait animation generation program is executed by a processor, the steps of the multi-pose portrait animation generation method as described above are implemented.

[0046] Beneficial effects: The present invention relates to a multi-pose portrait animation generation method, which obtains a portrait image and converts it into a three-dimensional deformation model; obtains audio data and inputs it into an audio-driven facial motion parameter extraction network to calculate facial motion parameters according to facial expression parameters; uses a variational autoencoder for posture generation to calculate head posture motion parameters and hand posture motion parameters according to head posture parameters, hand posture parameters and audio data; extracts appearance features of the portrait image, and estimates standard key points of the portrait image to generate standard key points; maps head posture parameters, hand posture parameters, facial expression parameters, facial motion parameters, head posture motion parameters and hand posture motion parameters to parameter representations under a 2D view; finally, inputs the mapped parameter representations, standard key points and appearance features into an image generator to generate a multi-pose portrait animation including hand postures and dynamic expressions. The present invention converts a portrait image into a three-dimensional deformation model including parameters of facial expressions, head postures and hand postures, and combines it with a facial motion parameter extraction network driven by audio data to accurately generate facial expressions and posture changes that match the sound, thereby improving the realism and naturalness of the animation; by using a variational autoencoder for posture generation to calculate the motion parameters of the head and hands, the synchronization of facial expressions and posture movements with the audio is achieved; in addition, by performing standard key point estimation on the appearance features of the portrait image and inputting the mapped parameters into an image generator, a natural combination of dynamic expressions and hand postures is achieved, thereby generating richer and more realistic multi-posture portrait animations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0048] Figure 1It is a flowchart of an embodiment of a method for generating a multi-pose portrait animation according to the present invention;

[0049] Figure 2 It is a functional module schematic diagram of a preferred embodiment of the multi-pose portrait animation generating device of the present invention;

[0050] Figure 3 It is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the multi-pose portrait animation generation device of the present invention. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] It should be noted that in recent years, with the rapid development of computer graphics and artificial intelligence technology, generating realistic animated portraits based on image and audio data has become an important research direction. Generating realistic and natural animated portraits has broad application prospects in the fields of entertainment, film and television production, virtual reality (VR), augmented reality (AR), etc.

[0053] Traditional portrait animation generation methods usually rely on pre-recorded footage or manual adjustments to generate animations. Although these methods can generate high-quality animation effects, they have the following limitations:

[0054] High labor costs: Manually adjusting facial expressions, postures, and motion parameters requires a lot of human intervention, which results in a long production cycle and high costs.

[0055] Lack of flexibility: Pre-recorded materials are often limited to specific scenes and actions and cannot flexibly adapt to different user needs, especially when generating personalized or customized animations.

[0056] Complex data processing: As the complexity of input data increases, such as portrait images from multiple angles and poses and synchronized audio data, existing technologies find it difficult to process and generate dynamic animations that meet the requirements in real time while maintaining high quality.

[0057] In order to overcome the limitations of traditional methods, portrait animation generation technology based on 3D morphable models has received widespread attention. This technology extracts 3D information from 2D portrait images and combines facial expression parameters, posture parameters, etc. to generate an adjustable 3D model. This method can effectively improve the flexibility and quality of animation generation. However, the existing 3D model generation technology also has some shortcomings:

[0058] Limited model generation accuracy: In the process of generating a 3D model from a 2D image, factors such as image quality and lighting conditions may cause the model to be inaccurate, thus affecting the naturalness of the final animation.

[0059] Complexity of motion parameter generation: The dynamic changes of facial expressions and postures require complex algorithms to generate, especially when multimodal data (such as the combination of audio and images) is involved. Existing methods find it difficult to generate accurate motion parameters while ensuring real-time performance.

[0060] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for generating a multi-pose portrait animation provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0061] like Figure 1 As shown, the multi-pose portrait animation generation method proposed by the present invention comprises the following steps:

[0062] S10, acquiring a portrait image, converting the portrait image into a three-dimensional deformable model, wherein the three-dimensional deformable model includes facial expression parameters, head posture parameters, and hand posture parameters, acquiring audio data, inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters;

[0063] In this embodiment, a portrait image of the target person is obtained through a camera or other image acquisition device, or a pre-generated portrait image is directly received through a network or other data transmission method. Then, the portrait image is converted into a 3D Morphable Model (3DMM), which uses a fixed average shape As a basis, combined with the identity feature parameter αU id and expression feature parameter βU exp Generate a personalized 3D face model. The spatial expression of the 3D deformation model is:

[0064]

[0065] in, Refers to the average shape of a 3D face, which serves as the basis of the model; αU id refers to the variable parameter of the identity feature, α is the 80-dimensional identity weight vector, U id is the identity feature basis vector, representing the deviation of the individual from the average shape; βU exp refers to the variable parameter of facial expression, β is the 64-dimensional expression weight vector, U exp is the expression feature basis vector, which represents the deviation of facial expression from the neutral expression.

[0066] Based on the generated 3D deformable model, we add motion parameters ρ = [r, t], where r represents the rotation of the head (belongs to SO(3)) and t represents the translation of the head (belongs to R 3 ). In addition, the model also includes the key points of the upper body skeleton and its motion parameters λ, which describe the posture changes of the human body.

[0067] The 3D deformable model is used to represent the facial structure and posture of the target person. This model includes facial expression parameters, head posture parameters, and hand posture parameters.

[0068] Real-time audio data is obtained through a microphone or other audio acquisition device, or pre-recorded audio data is directly received through the network or other data transmission methods. Audio data is used to drive changes in facial expressions and lip shape. The acquired audio data is input into a specific deep learning network to extract facial motion parameters synchronized with the audio. Facial motion parameters are used to control the changes in facial expressions in the 3D deformable model. The facial expression parameters in the 3D deformable model are combined with the facial motion parameters extracted from the audio data to calculate the final facial motion parameters for animation generation.

[0069] The acquired audio data is input into an audio-driven facial motion parameter extraction network (such as ExpNet). The network analyzes the audio data and extracts facial motion parameters (β 1...n ). Then, the facial expression parameters β in the 3D deformation model are combined to calculate the final facial motion parameters used to drive dynamic expression changes.

[0070] In one specific implementation, the data collection method based on the local device includes: collecting a portrait image through a local camera to generate a three-dimensional deformation model. Using 3DMM technology, the collected portrait image is converted into a three-dimensional deformation model including facial expressions, head postures and hand postures. Audio data is collected in real time through a local microphone. The collected audio data is input into an audio-driven facial motion parameter extraction network, and combined with the facial expression parameters in the three-dimensional deformation model, facial motion parameters for driving dynamic expression changes are calculated and generated.

[0071] In another specific embodiment, the remote data-based processing method includes: obtaining a pre-stored portrait image from a cloud server via a network. Using 3DMM technology, the image is converted into a three-dimensional deformation model, which includes facial expressions, head postures, and hand posture parameters. A pre-recorded audio file is transmitted via the network. The audio file is input into an audio-driven facial motion parameter extraction network, and the final facial motion parameters are calculated in combination with the facial expression parameters to synchronize the expression changes of the animation. Through a high-precision three-dimensional deformation model, a high degree of synchronization between dynamic expressions and audio is achieved, with a flexible data input method, which is widely applicable to scenes such as virtual reality, film and television production, and virtual assistants, significantly improving the efficiency and quality of animation production, and ensuring the practicality and superiority of the system.

[0072] S20, using a variational autoencoder for posture generation to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data;

[0073] In this embodiment, the variational autoencoder for posture generation is a deep learning model for generating and predicting posture motion parameters. It can be based on a variety of variational autoencoder (VAE) architectures, including but not limited to PoseVAE. The autoencoder can process the input head posture parameters, hand posture parameters, and audio data, and generate posture motion parameters that match these input data.

[0074] The head and hand posture motion parameters represent the dynamic changes of the head and hands in the animation, covering posture information such as rotation and translation. The motion parameters generated by the variational autoencoder can ensure that the posture changes in the animation are synchronized with the input data (such as audio).

[0075] In one specific embodiment, the variational autoencoder for posture generation is implemented based on multiple VAE models, including: obtaining head posture parameters, hand posture parameters and audio data through a local device or network. The input data is input into a variety of different variational autoencoder models, including but not limited to PoseVAE, ConditionalVAE (CVAE) or other VAE models suitable for posture generation. The posture motion parameters generated by each model can be compared or fused to obtain the optimal posture motion result. The generated posture motion parameters are used to drive the head and hand movements of the virtual character so that the expressions and actions in the animation are highly consistent with the input data.

[0076] In another specific embodiment, the variational autoencoder for posture generation is implemented based on the adaptive VAE model, including: obtaining real-time or pre-stored head posture parameters, hand posture parameters, and audio data. An adaptive variational autoencoder model is used, which can dynamically adjust its structure and parameters according to the characteristics of the input data (such as intonation changes in audio, complexity of posture, etc.) to generate optimal posture motion parameters. The model may include an adaptive PoseVAE, an adaptive CVAE, or other VAE variants. The generated motion parameters are applied to the animation production of the virtual character to achieve posture changes that are highly synchronized with the input data.

[0077] In other specific embodiments, the variational autoencoder for posture generation is implemented based on the multimodal VAE model in the following ways: combining multiple input sources, such as head posture parameters, hand posture parameters, audio data and other auxiliary information (such as ambient sound, background information, etc.). Using a multimodal variational autoencoder model (Multimodal VAE), the model can process a variety of different types of data inputs to generate richer and more complex posture motion parameters. This model can capture more details and achieve more realistic animation effects. Using the motion parameters generated by multimodality, complex dynamic animations with more details are generated for virtual characters.

[0078] Through the above steps, the accurate generation of head and hand posture motion parameters is achieved, solving the problem of insufficient synchronization between posture and audio in traditional methods. It can flexibly adapt to different input conditions, ensure high accuracy and consistency of posture motion parameters, and effectively improve the naturalness and real-time performance of virtual character animation.

[0079] S30, extracting appearance features of the portrait image, and performing standard key point estimation on the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial motion parameters, head posture motion parameters, and hand posture motion parameters into parameter representations under a 2D view;

[0080] In this embodiment, appearance features refer to visual features extracted from the portrait image, such as color, texture, shape, etc. These features are used to ensure that the generated animation retains the visual consistency of the original portrait during the 2D and 3D mapping process.

[0081] Standard key points refer to a set of important reference points defined in a portrait image, such as the positions of eyes, nose, mouth, etc. Through standard key point estimation, standardized reference points are generated for subsequent posture and expression mapping.

[0082] The head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters in the three-dimensional space are mapped to the two-dimensional view. This process involves projecting the three-dimensional information onto the two-dimensional plane so that these posture and expression changes can be accurately expressed in the 2D animation.

[0083] In one specific embodiment, the parameter mapping method is based on the standard key point estimation and mapping method of the 2D image, including: extracting appearance features from the input portrait image to ensure that these features remain consistent in the subsequent 2D and 3D mapping processes. Use deep learning or computer vision algorithms to estimate standard key points of the portrait image and generate a set of standardized reference points for subsequent posture and expression mapping. Map the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters to the 2D view. This process may involve perspective projection or other geometric transformations to ensure that the three-dimensional information is accurately reflected in the two-dimensional plane.

[0084] In another specific embodiment, the parameter mapping method is a hybrid method combining 3D model and 2D mapping, which may include: combining the appearance features in the three-dimensional deformation model and the visual features in the portrait image to generate a comprehensive appearance feature vector. Standard key point estimation is performed on the three-dimensional model and the 2D image respectively, and the two are aligned using a registration algorithm to ensure the accuracy of the mapping. Using the key point information after registration, the three-dimensional posture and expression parameters are mapped to a two-dimensional plane. Through multi-view projection technology, a more accurate parameter representation in a 2D view is achieved.

[0085] In other specific embodiments, the parameter mapping method is based on the dynamic mapping method of the real-time data stream, including: extracting appearance features from the continuous real-time portrait image data stream to support the mapping process in dynamic scenes. Real-time estimation of key points and dynamic updates to cope with rapid changes in the scene. Combined with the real-time input data, the three-dimensional parameters are projected into a 2D view, and a 2D animation is generated that is synchronized with real-time audio and motion.

[0086] By accurately mapping 3D posture and expression parameters into parameter representations under a 2D view, the technical difficulty of expressing complex 3D movements and expressions in 2D animations is effectively solved, ensuring the consistency of the visual effects in the animation with the original portraits, and improving the accuracy and authenticity of animation production. It is suitable for applications such as virtual image generation and animation production that require a high degree of restoration of real scenes.

[0087] S40, inputting the mapped parameter representation, standard key points and appearance features into an image generator to generate a multi-pose portrait animation including hand poses and dynamic expressions.

[0088] In this embodiment, the mapped parameter representation is a set of values ​​obtained by projecting the three-dimensional posture and expression parameters onto a two-dimensional view. These parameters represent the posture of the head and hands, as well as the dynamic expression information of the face, and are used to generate animation on a two-dimensional plane after mapping.

[0089] Standard key points refer to a set of reference points extracted from portrait images, such as eyes, nose, mouth, etc., representing important parts of the face. Standard key points are used to ensure the accuracy of facial expressions in animations.

[0090] An image generator is a model or algorithm that can generate images based on input parameters. It receives mapped parameter representations, standard key points, and appearance features as input, and outputs multi-pose portrait animations containing hand poses and dynamic expressions.

[0091] In one specific embodiment, the method of image generation based on the predefined model includes: inputting the two-dimensional parameter representation obtained by mapping the posture parameters, expression parameters and motion parameters into the predefined image generation model together with the extracted standard key points and appearance features. The image generator generates a dynamic two-dimensional portrait animation based on the input data, and the animation includes hand posture changes and dynamic expressions. The generator uses the input standard key points to ensure the accuracy of facial expressions and uses appearance features to maintain consistency with the original portrait. The output result is a dynamic portrait animation containing multiple postures and expression changes, which can be applied to virtual images, animation production or real-time interactive scenes.

[0092] In another specific embodiment, the method of image generation based on deep learning includes: inputting the mapped two-dimensional parameter representation, standard key points and appearance features into a trained deep learning model (such as GAN or VAE). The deep learning model generates high-quality dynamic portrait animation based on the input data. The model can generate more realistic animation effects by learning complex expression and posture change patterns. The generated animation has highly realistic facial expressions and hand posture changes, which can be used for real-time applications in film and television production or virtual reality.

[0093] In other specific embodiments, the image generation method based on hybrid technology includes: combining a rule-based mapping method and a deep learning algorithm to input parameter representations, standard key points and appearance features into an image generation system. The system uses a combination of multiple technologies to generate multi-pose portrait animations including hand gestures and dynamic expressions. By comprehensively utilizing rules and deep learning, the system can improve the generation speed and quality while maintaining the authenticity of the animation. The generated animation can be flexibly applied in different scenarios, such as virtual meetings, social media or animation production.

[0094] By inputting the mapped parameter representation, standard key points and appearance features into the image generator, multi-pose portrait animations containing hand gestures and dynamic expressions can be effectively generated, solving the problems of multimodal data fusion and lack of realism. The accuracy and real-time performance of animation generation are improved, which is suitable for virtual images, film and television production, virtual reality and other fields, and significantly enhances user experience and animation production efficiency.

[0095] The present invention converts a portrait image into a three-dimensional deformation model including parameters of facial expressions, head postures and hand postures, and combines it with a facial motion parameter extraction network driven by audio data to accurately generate facial expressions and posture changes that match the sound, thereby improving the realism and naturalness of the animation; by using a variational autoencoder for posture generation to calculate the motion parameters of the head and hands, the synchronization of facial expressions and posture movements with the audio is achieved; in addition, by performing standard key point estimation on the appearance features of the portrait image and inputting the mapped parameters into an image generator, a natural combination of dynamic expressions and hand postures is achieved, thereby generating richer and more realistic multi-posture portrait animations.

[0096] In one embodiment, the above S20 includes:

[0097] S201, selecting one or more motion style parameters from a predefined motion style library, wherein the motion style parameters are used to determine style features of head and hand movements;

[0098] S202, inputting the head posture parameter, hand posture parameter, motion style parameter and audio data into the variational autoencoder model for posture generation;

[0099] S203, the variational autoencoder model for posture generation generates corresponding head posture motion parameters and hand posture motion parameters according to the input data through encoding and decoding processes.

[0100] In this embodiment, the variational autoencoder for gesture generation is a deep learning model for generating gesture motion parameters that match the input data. By encoding and decoding the input head gesture parameters, hand gesture parameters, motion style parameters, and audio data, the model can generate accurate head and hand gesture motion parameters.

[0101] The head posture parameters and hand posture parameters describe the position and posture of the head and hand in three-dimensional space, including information such as rotation and translation. They are important basic data for generating dynamic postures.

[0102] The motion style parameters are used to describe the style characteristics of head and hand movements, such as the speed, amplitude, and smoothness of the movements. By introducing the motion style parameters, the model can generate gesture movements that conform to a specific style, making the generated animation more vivid and personalized.

[0103] Audio data usually includes speech or other sound signals, which are used to synchronize with gesture movements. Combined with audio data, the model can generate gesture changes consistent with the speech content.

[0104] In one specific embodiment, the method of generating motion style parameters based on the posture of a predefined motion style includes: the system selects one or more motion style parameters from a predefined motion style library. These parameters may include speed, intensity, softness, etc. The head posture parameters, hand posture parameters, selected motion style parameters and audio data are input into the variational autoencoder model for posture generation. The model generates corresponding head posture motion parameters and hand posture motion parameters according to the input data through the encoding and decoding process. This method is suitable for the need to generate dynamic animation scenes with a specific style, such as advertisements, game character animations, etc.

[0105] In another specific embodiment, the motion style parameters are generated based on the motion style adjusted in real time, including: real-time collection of user or environment input (such as gestures, expression changes) to dynamically adjust the motion style parameters. The head posture parameters, hand posture parameters, dynamically adjusted motion style parameters and audio data are input into the variational autoencoder model. The model calculates the head and hand posture motion parameters based on the motion style parameters generated by the dynamic input to ensure that the animation responds to the user's input in real time. This method is suitable for real-time interactive applications, such as character animation in virtual reality or virtual images in real-time performances.

[0106] In other specific embodiments, the motion style parameters are generated based on the motion style of multimodal fusion, including: combining multimodal input (such as visual, tactile feedback) to generate comprehensive motion style parameters. The multimodally generated motion style parameters are input into the variational autoencoder model together with the head posture parameters, hand posture parameters and audio data. By integrating the multimodal input, the model generates more complex and delicate head and hand posture motion parameters, making the animation more expressive. This method is suitable for application scenarios that require a high degree of immersion and complex interaction, such as advanced virtual reality experience or complex film and television special effects production.

[0107] This embodiment uses the variational autoencoder model of posture generation to calculate the head and hand posture motion parameters by combining motion style parameters, effectively solving the problem of single motion style and lack of personality in traditional animation generation. It can generate dynamic postures that meet specific styles according to different input data and requirements, making the generated animation more vivid and personalized, and is suitable for game development, virtual assistants, film and television production and other fields that require dynamic posture generation. At the same time, combined with audio data, it ensures a high degree of synchronization between action and audio, improving the user's interactive experience and the expressiveness of animation.

[0108] In one embodiment, before the above S20, the method further includes:

[0109] S204, collecting a standard data set including head posture parameters, hand posture parameters, motion style parameters and audio data;

[0110] S205, defining a weighted loss function, wherein the weighted loss function is composed of a mean square error, a KL divergence, and an adversarial loss with weights adjusted;

[0111] S206, the variational autoencoder for posture generation is a PoseVAE model, the weighted loss function and the standard data set are used to train the PoseVAE model, and the parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

[0112] In this embodiment, PoseVAE is a variational autoencoder model (VAE) specifically used to generate posture motion parameters. The model can process input posture parameters, motion style parameters and audio data to generate corresponding motion parameters.

[0113] The weighted loss function is the objective function used to guide model training. Here, the loss function consists of three parts: mean square error (MSE), KL divergence (Kullback-Leibler Divergence), and adversarial loss. The mean square error is used to minimize the difference between the generated results and the real data, the KL divergence is used to measure the difference between the generated distribution and the real distribution, and the adversarial loss is used to enhance the authenticity of the generated results.

[0114] The PoseVAE model is trained through the defined weighted loss function, so that the model can generate posture motion parameters that are closer to the real data. The training process optimizes the model parameters by minimizing the value of the loss function, and finally obtains the optimized PoseVAE model.

[0115] In one specific implementation, the model training method based on the standard data set includes:

[0116] Collect data: Collect standard datasets containing head pose parameters, hand pose parameters, motion style parameters, and audio data.

[0117] Define weighted loss function: Set a weighted loss function, where the mean square error is used to minimize the difference between the posture motion parameters and the real data, the KL divergence is used to ensure the stability of the generated distribution, and the adversarial loss is used to enhance the authenticity of the generated results.

[0118] Training the PoseVAE model: Input the standard data set into the PoseVAE model, train the model through the weighted loss function, optimize the model parameters, and make the generated posture motion parameters highly consistent with the real data.

[0119] Application scenarios: This method is suitable for applications that require high-precision gesture motion generation, such as animation production or advanced virtual reality experience.

[0120] In another specific implementation, the method of model training based on adaptive data includes:

[0121] Collect and preprocess data: Collect diverse data from different scenarios and preprocess them to form an adaptive dataset.

[0122] Define an adaptive weighted loss function: adjust the weights of the weighted loss function according to different data characteristics to adapt to different types of data input.

[0123] Training the PoseVAE model: Use an adaptive dataset and a weighted loss function to train the PoseVAE model so that it can generate more accurate and stable posture motion parameters in different application scenarios.

[0124] Application scenarios: This method is suitable for a variety of scenarios, such as cross-domain virtual character animation generation or film and television production in various styles.

[0125] In other specific embodiments, the method of model optimization based on adversarial training includes:

[0126] Collect and generate adversarial examples: Generate data samples containing unusual or complex motion styles for adversarial training.

[0127] Define an enhanced weighted loss function: Based on the standard weighted loss function, the weight of the adversarial loss is enhanced to improve the model's generation ability in complex scenarios.

[0128] Training the PoseVAE model: Through adversarial training and optimization of the weighted loss function, the generation effect of the PoseVAE model is further enhanced, especially in complex and abnormal posture movements.

[0129] Application scenarios: Suitable for animation scenes with high complexity and irregularity, such as special effects production in science fiction movies or difficult motion capture.

[0130] This embodiment effectively improves the accuracy and authenticity of the generated results by performing weighted loss function optimization training before using the PoseVAE model to calculate the pose motion parameters, and solves the error and instability problems existing in traditional pose generation methods. The optimized PoseVAE model can generate pose motion parameters that are highly matched with real data in various complex scenarios, and is suitable for film and television production, virtual assistants, game development and other fields, greatly improving the quality of animation generation and user experience.

[0131] In one embodiment, the above S20 includes:

[0132] S207, inputting the head posture parameter, hand posture parameter, motion style parameter and audio data into the improved PoseVAE model to generate head posture change amount and hand posture change amount;

[0133] S208, using two VAE encoders that do not share parameters to respectively encode the head posture change amount and the hand posture change amount, and map the head posture change amount and the hand posture change amount to two different latent space distributions;

[0134] S209, fusing and resampling the two latent space distributions through Gaussian mixture to obtain a new latent space distribution, decoding the new latent space distribution using a VAE decoder, and generating the head posture motion parameters and the hand posture motion parameters.

[0135] In this embodiment, the improved PoseVAE model is an optimized variational autoencoder model based on the standard PoseVAE model, which can more accurately generate the head and hand posture changes. The model accepts multiple input data, including head posture parameters, hand posture parameters, motion style parameters and audio data.

[0136] The head posture change and hand posture change refer to the dynamic changes generated by the improved PoseVAE model, indicating the degree of change in the head and hand postures. These changes will be used in the further encoding and decoding process.

[0137] Two independent VAE encoders are used to encode the head pose variation and the hand pose variation respectively. The two encoders do not share parameters to ensure that the head and hand pose variations are processed independently and generate different latent space representations.

[0138] The encoded head and hand pose variations are mapped to two different latent space distributions, which are then fused via Gaussian mixture and resampled to generate a new latent space distribution that more accurately represents the overall pose variation.

[0139] The VAE decoder is responsible for decoding the fused latent space distribution into the final head pose motion parameters and hand pose motion parameters. These parameters will be used to generate the dynamic pose of the virtual character.

[0140] In one specific implementation, the method of generating a posture based on a standard data set includes:

[0141] Input data: Head pose parameters, hand pose parameters, motion style parameters, and audio data are extracted from standard datasets.

[0142] Generate variance: These parameters are fed into the improved PoseVAE model to generate head pose variance and hand pose variance.

[0143] Encoding process: Use two independent VAE encoders to encode the head pose changes and hand pose changes respectively, mapping them to their respective latent space distributions.

[0144] Fusion and decoding: The two latent space distributions are fused through Gaussian mixture and resampled to generate a new latent space distribution. The latent space is decoded using the VAE decoder to generate the final head posture motion parameters and hand posture motion parameters.

[0145] Application scenarios: Suitable for animation production or virtual character generation that requires precise control of posture changes.

[0146] In another specific implementation, the method of gesture generation based on multimodal input includes:

[0147] Input data: Combine multimodal input (such as vision, audio, tactile feedback, etc.) to obtain head posture parameters, hand posture parameters, motion style parameters and audio data.

[0148] Generate variation: These multimodal data are input into the improved PoseVAE model to generate the variation of head and hand postures.

[0149] Encoding process: Two VAE encoders are used to independently encode these changes to ensure that the posture changes of the head and hands are accurately represented in the latent space.

[0150] Fusion and decoding: These independent latent space distributions are fused through Gaussian mixture and resampled to generate a new latent space, which is then decoded into head pose motion parameters and hand pose motion parameters using a VAE decoder.

[0151] Application scenarios: Suitable for scenarios such as virtual reality and augmented reality that require combining multiple input data.

[0152] In other specific implementations, the gesture generation method based on the real-time data stream includes:

[0153] Input data: Obtain head posture parameters, hand posture parameters, motion style parameters and audio data from real-time data streams.

[0154] Generate changes: These real-time data are input into the improved PoseVAE model to generate real-time changes in the posture of the head and hands.

[0155] Encoding process: Two VAE encoders are used to encode the pose changes generated in real time and map them to the latent space.

[0156] Fusion and decoding: The latent space distribution is fused in real time through Gaussian mixture, and a new latent space is generated by resampling to decode the real-time posture motion parameters.

[0157] Application scenarios: Suitable for real-time interactive applications, such as real-time expression and gesture generation in virtual meetings.

[0158] This embodiment can accurately generate the posture motion parameters of the head and hands through the improved PoseVAE model and independent VAE encoder, solving the accuracy and consistency problems of posture generation in traditional methods. Through Gaussian mixture and resampling technology, this method can generate highly natural and coherent posture motion in complex scenes and multimodal input. It is suitable for animation production, virtual reality and real-time interactive applications, and effectively improves the quality of generated animation and user experience.

[0159] In one embodiment, in the above S10, converting the portrait image into a three-dimensional deformable model includes:

[0160] S101, preprocessing the portrait image, including image denoising, illumination correction and contrast adjustment, to improve the quality of the portrait image;

[0161] S102, using monocular 3D face recognition technology, converting the preprocessed portrait image into the three-dimensional deformable model;

[0162] S103, extracting facial expression parameters, head posture parameters and hand posture parameters from the three-dimensional deformation model.

[0163] In this embodiment, the portrait image is preprocessed before being converted into a three-dimensional deformable model. The preprocessing steps include image denoising, illumination correction, and contrast adjustment. These operations are intended to improve image quality and ensure that subsequent 3D face recognition and model generation are more accurate.

[0164] The pre-processed 2D portrait image is converted into a 3D deformable model using monocular 3D face recognition technology. Monocular 3D face recognition technology uses 2D images captured by a single camera to infer the 3D facial structure through deep learning and computer vision algorithms.

[0165] Facial expression parameters, head posture parameters and hand posture parameters are extracted from the generated 3D deformable model. These parameters describe the dynamic facial expressions, head and hand postures, and are used for subsequent animation generation or other applications.

[0166] In one specific embodiment, the method of generating a three-dimensional deformable model based on a standard image includes: preprocessing the portrait image collected from the camera, including image denoising to reduce noise, illumination correction to eliminate the influence of uneven illumination, and contrast adjustment to enhance image details. Using monocular 3D face recognition technology, the preprocessed portrait image is converted into a three-dimensional deformable model. Through a deep learning algorithm, the three-dimensional facial structure in the image is inferred. Key facial expression parameters, head posture parameters, and hand posture parameters are extracted from the generated three-dimensional deformable model. These parameters will be used for subsequent animation generation and expression recognition applications.

[0167] In another specific embodiment, the method for generating a three-dimensional deformable model based on multi-angle images includes: pre-processing the portrait images collected from multiple angles respectively to ensure that the images at each angle are denoised, light corrected and contrast adjusted. Combining the pre-processed images from multiple angles, a more accurate three-dimensional deformable model is generated using monocular 3D face recognition technology. The accuracy of the three-dimensional model is improved by comprehensive processing of multiple perspectives. Facial expression parameters, head posture parameters and hand posture parameters are extracted from the comprehensively generated three-dimensional deformable model for more complex animation and expression control.

[0168] In other specific embodiments, the method of generating a three-dimensional deformable model based on a real-time image stream includes: in the real-time image stream, quickly preprocessing each frame of the portrait image to ensure that each frame in the image stream maintains high quality. Real-time application of monocular 3D face recognition technology to convert each preprocessed image frame into a three-dimensional deformable model, and continuously generate three-dimensional dynamic changes of the face. Real-time extraction of facial expression parameters, head posture parameters, and hand posture parameters in the three-dimensional deformable model for driving virtual characters or realizing other real-time interactive applications.

[0169] This embodiment achieves high-precision conversion from two-dimensional images to three-dimensional structures by converting portrait images into three-dimensional deformation models, solving the model distortion problem caused by poor image quality in traditional methods. The preprocessing step significantly improves the quality of the input image, ensuring that the generated three-dimensional model is more accurate. Combined with monocular 3D face recognition technology, the complete three-dimensional structure can be inferred from a single camera input, thereby saving the cost of multiple cameras or complex equipment. Ultimately, the extracted facial expression parameters, head posture parameters, and hand posture parameters provide accurate data support for subsequent animation generation and real-time applications, and are suitable for a variety of fields such as virtual reality and film and television production.

[0170] In one embodiment, in the above S10, the audio data is input into an audio-driven facial motion parameter extraction network, and facial motion parameters are calculated according to the facial expression parameters, including:

[0171] S104, performing feature extraction on the audio data to extract audio features including emotion, intonation and rhythm;

[0172] S105, inputting the extracted audio features and the facial expression parameters into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, and the facial motion parameters are used to generate facial dynamic expressions.

[0173] In this embodiment, the audio data needs to be feature extracted before being input into the audio-driven facial motion parameter extraction network. These features include key information such as emotion, intonation, and rhythm, which are used to capture emotional changes in the audio, the rhythm and intonation of speech, and these features will directly affect the generated facial expressions.

[0174] The audio-driven facial motion parameter extraction network is a deep learning network such as ExpNet, which is specifically designed to calculate facial motion parameters based on input audio features and facial expression parameters. The network combines the features of audio data and static information of facial expressions to generate dynamic facial expressions synchronized with the audio content.

[0175] Facial motion parameters are key data generated by the network to drive facial dynamic expressions. These parameters control the movement of different parts of the face, allowing the expression to be dynamically updated as the audio changes.

[0176] In one specific implementation, the facial motion generation method based on standard audio data includes:

[0177] Audio feature extraction: Extract features from standard audio data to extract features such as emotion, intonation, and rhythm. These features will be used to generate facial expressions that are consistent with the speaker's emotional state and expression.

[0178] Input to the network: The extracted audio features are input into the audio-driven facial motion parameter extraction network together with the pre-computed facial expression parameters.

[0179] Calculate facial motion parameters: The network generates facial motion parameters based on the input data. These parameters are used to drive the dynamic changes of facial expressions to synchronize them with the audio content.

[0180] Application scenarios: Suitable for animation scenarios that need to be highly matched with audio content, such as lip synchronization and expression generation of virtual characters.

[0181] In another specific implementation, the facial motion generation method based on multimodal fusion includes:

[0182] Multimodal feature extraction: Extract features from multimodal data (such as audio and video), especially extracting emotion, intonation and rhythm features from audio, and combining them with facial expression information from visual data.

[0183] Input to the network: The multimodal features and facial expression parameters are input into the audio-driven facial motion parameter extraction network to generate more accurate and natural facial motion parameters.

[0184] Calculate facial motion parameters: The network generates more complex facial motion parameters based on these rich input data for more realistic expression changes.

[0185] Application scenarios: Suitable for film and television production and advanced virtual reality applications, capable of generating multi-level facial expression changes.

[0186] In other specific embodiments, the facial motion generation based on real-time audio data includes:

[0187] Real-time feature extraction: Extract sentiment, intonation, and rhythm features from real-time audio streams and process them quickly.

[0188] Input to the network: The real-time extracted audio features and current facial expression parameters are input to the audio-driven facial motion parameter extraction network.

[0189] Calculate facial motion parameters: The network calculates facial motion parameters in real time, allowing facial expressions in virtual characters or applications to change in sync with real-time audio content.

[0190] Application scenarios: Suitable for real-time interactive applications, such as character animation generation in virtual meetings to ensure synchronization with the user's voice.

[0191] This embodiment effectively solves the problem of facial expression and audio being out of sync in traditional methods by combining audio data with facial expression parameters and inputting them into an audio-driven facial motion parameter extraction network. Audio feature extraction ensures that the dynamic changes in facial expressions can reflect the emotions, intonation, and rhythm in the audio. The generated facial motion parameters enable facial expressions to be naturally synchronized with the audio content, which is suitable for scenarios such as virtual character generation, animation production, and virtual meetings, significantly improving the expressiveness and user experience of animation and interactive applications.

[0192] In one embodiment, in the above S30, mapping the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters into parameter representations under a 2D view includes:

[0193] S301, mapping the head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters to a 2D plane through perspective projection to generate a preliminary parameter representation under a 2D view;

[0194] S302, adjusting the positions of the standard key points in the 2D view according to the facial expression parameters and the facial movement parameters to accurately reflect the changes of the facial expression and posture on the 2D plane;

[0195] S303, based on the adjusted standard key point positions and the preliminary parameter representation under the 2D view, the head posture motion parameters and the hand posture motion parameters are smoothed using an interpolation algorithm to generate a final parameter representation under the 2D view for driving the generation of a multi-pose portrait animation.

[0196] In this embodiment, perspective projection is a technique for mapping points in a three-dimensional space onto a two-dimensional plane. In this process, head posture parameters, hand posture parameters, facial expression parameters, and facial movement parameters are mapped onto a two-dimensional plane through perspective projection to generate a preliminary parameter representation under a 2D view. This step accurately converts the information in the three-dimensional space into a two-dimensional representation, maintaining the sense of depth and space.

[0197] Standard key points are the reference points of the face (such as eyes, nose, mouth, etc.). The positions of these points on the two-dimensional plane need to be adjusted according to the facial expression parameters and facial movement parameters to accurately reflect the changes in facial expressions and postures on the two-dimensional plane. Adjusting the positions of these key points can ensure that the final generated two-dimensional view has the correct expression and posture performance.

[0198] The interpolation algorithm is used to smooth the head posture motion parameters and hand posture motion parameters to ensure that the transition between parameter representations in the two-dimensional view is smooth and natural. The interpolation algorithm calculates the intermediate values ​​between different parameter representations, making the final generated animation more coherent and natural in dynamic performance.

[0199] In one specific implementation, the method of generating a 2D view based on a static image includes:

[0200] Perspective projection mapping: The head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters in the static image are mapped to a two-dimensional plane through perspective projection to generate a preliminary parameter representation under a 2D view.

[0201] Standard key point adjustment: According to facial expression parameters and facial movement parameters, the standard key point positions in the 2D view are adjusted to accurately reflect facial expressions and posture changes.

[0202] Interpolation processing: Use the interpolation algorithm to smooth the head posture motion parameters and hand posture motion parameters to generate the final parameter representation under the 2D view.

[0203] Application scenarios: Suitable for portrait animation generation of static images, such as character performance in artwork or animation clips.

[0204] In another specific implementation, the method of generating a 2D view based on a dynamic video includes:

[0205] Perspective projection mapping: Perspective projection is performed on the head and hand posture parameters, facial expression parameters, and facial movement parameters in dynamic video frames to generate parameter representations under the preliminary 2D view of each frame.

[0206] Standard key point adjustment: Dynamically adjust the position of standard key points based on facial expressions and motion parameters in each frame to reflect changes in expression and posture throughout the animation.

[0207] Interpolation processing: Use interpolation algorithms between consecutive frames to smooth the posture motion parameters to ensure a natural transition between frames and generate the final parameter representation under the 2D view.

[0208] Application scenarios: Suitable for character animation generation in animated films or games, especially when high-quality posture and expression performance is required in consecutive frames.

[0209] In other specific implementations, the method of generating a 2D view based on real-time interaction includes:

[0210] Perspective projection mapping: Map the user's head posture, hand posture, facial expression and motion parameters to a two-dimensional plane in real time to generate a preliminary parameter representation under a 2D view.

[0211] Standard key point adjustment: Dynamically adjust the positions of standard key points based on the expression and motion data captured in real time to ensure that the animation generated in real time is synchronized with the user's facial expressions and postures.

[0212] Interpolation processing: Use the interpolation algorithm to smooth the posture motion parameters generated in real time and generate the final parameter representation under the 2D view to drive the dynamic performance of the virtual image.

[0213] Application scenarios: Suitable for real-time virtual meetings, virtual assistants, or other application scenarios that require real-time interaction.

[0214] This embodiment effectively solves the problem of conversion from three-dimensional space to two-dimensional expression by mapping head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters to two-dimensional views. Perspective projection ensures the retention of depth and spatial sense, the adjustment of standard key points enables facial expressions and postures to be accurately expressed in two-dimensional views, and the use of interpolation algorithms further improves the smoothness and naturalness of animation. The generated two-dimensional views can be used for high-quality portrait animation generation, and are suitable for animation production, virtual assistants, real-time interaction and other fields.

[0215] The present invention also provides a multi-pose portrait animation generating device, referring to Figure 2 , Figure 2 Schematic diagram of functional modules of a preferred embodiment of a multi-pose portrait animation generation device of the present invention. The multi-pose portrait animation generation device comprises:

[0216] A three-dimensional model generation module is configured to obtain a portrait image, convert the portrait image into a three-dimensional deformable model, wherein the three-dimensional deformable model includes facial expression parameters, head posture parameters, and hand posture parameters, obtain audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters according to the facial expression parameters;

[0217] A posture generation module, using a posture generation variational autoencoder to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data;

[0218] a 2D mapping module, extracting appearance features of the portrait image, estimating standard key points of the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters into parameter representations under a 2D view;

[0219] The multi-pose animation generation module inputs the mapped parameter representation, standard key points and appearance features into the image generator to generate multi-pose portrait animation including hand poses and dynamic expressions.

[0220] The specific implementation of the multi-pose portrait animation generation device of the present invention is basically the same as the various embodiments of the multi-pose portrait animation generation method described above, and will not be repeated here.

[0221] The present invention also provides a multi-pose portrait animation generating device, such as Figure 3 As shown, the multi-pose portrait animation generation device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0222] Those skilled in the art will understand that Figure 3 The hardware structure of the multi-pose portrait animation generating device shown in the figure does not constitute a limitation of the multi-pose portrait animation generating device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0223] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a multi-pose portrait animation generation program. The operating system is a program for managing and controlling the multi-pose portrait animation generation device and software resources, and supports the operation of the network communication module, the user interface module, the multi-pose portrait animation generation program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0224] exist Figure 3 In the hardware structure of the multi-pose portrait animation generation device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the multi-pose portrait animation generation program stored in the memory 1005 and perform the same operations as the multi-pose portrait animation generation method.

[0225] The specific implementation of the multi-pose portrait animation generation device of the present invention is basically the same as the various embodiments of the multi-pose portrait animation generation method described above, and will not be repeated here.

[0226] In addition, an embodiment of the present invention further provides a computer storage medium on which a multi-pose portrait animation generation program is stored. When the multi-pose portrait animation generation program is executed by a processor, the steps of the multi-pose portrait animation generation method described above are implemented.

[0227] The specific implementation of the computer storage medium of the present invention is basically the same as the above-mentioned embodiments of the multi-pose portrait animation generation method, and will not be repeated here.

[0228] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0229] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A method for generating a multi-pose portrait animation, characterized in that: The following steps are involved: Acquire a portrait image, convert the portrait image into a three-dimensional deformable model, the three-dimensional deformable model including facial expression parameters, head posture parameters and hand posture parameters, acquire audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters according to the facial expression parameters; Using a posture-generated variational autoencoder to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data; Extracting appearance features of the portrait image, estimating standard key points of the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial motion parameters, head posture motion parameters, and hand posture motion parameters into parameter representations under a 2D view; The mapped parameter representation, standard key points and appearance features are input into the image generator to generate multi-pose portrait animations including hand poses and dynamic expressions.

2. The method for generating a multi-pose portrait animation according to claim 1, wherein: The variational autoencoder generated by the posture calculates the head posture motion parameters and the hand posture motion parameters according to the head posture parameters, the hand posture parameters and the audio data, including: Selecting one or more motion style parameters from a predefined motion style library, wherein the motion style parameters are used to determine style features of head and hand movements; Inputting the head posture parameters, hand posture parameters, motion style parameters and audio data into the variational autoencoder model for posture generation; The variational autoencoder model for posture generation generates corresponding head posture motion parameters and hand posture motion parameters according to input data through encoding and decoding processes.

3. The method for generating a multi-pose portrait animation according to claim 1, wherein: Before using the variational autoencoder generated by the posture to calculate the head posture motion parameters and the hand posture motion parameters according to the head posture parameters, the hand posture parameters, the motion style parameters and the audio data, the method further includes: Collect standard datasets containing head pose parameters, hand pose parameters, motion style parameters, and audio data; A weighted loss function is defined, wherein the weighted loss function is composed of the mean square error, KL divergence, and adversarial loss after adjusting the weights; The variational autoencoder for posture generation is a PoseVAE model. The PoseVAE model is trained using the weighted loss function and the standard data set, and the parameters of the PoseVAE model are iteratively adjusted to obtain an improved PoseVAE model.

4. The method for generating a multi-pose portrait animation according to claim 3, wherein: The head posture motion parameters and the hand posture motion parameters are calculated using the posture-generated variational autoencoder according to the head posture parameters, the hand posture parameters, the motion style parameters and the audio data, including: Inputting the head posture parameters, hand posture parameters, motion style parameters and audio data into the improved PoseVAE model to generate head posture change and hand posture change; Using two VAE encoders that do not share parameters to encode the head posture change and the hand posture change respectively, and mapping the head posture change and the hand posture change to two different latent space distributions; The two latent space distributions are fused and resampled through Gaussian mixture to obtain a new latent space distribution, and the new latent space distribution is decoded using a VAE decoder to generate the head posture motion parameters and the hand posture motion parameters.

5. The method for generating a multi-pose portrait animation according to claim 1, wherein: Converting the portrait image into a three-dimensional deformable model comprises: Preprocessing the portrait image, including image denoising, illumination correction and contrast adjustment, to improve the quality of the portrait image; Using monocular 3D face recognition technology, converting the preprocessed portrait image into the three-dimensional deformable model; Facial expression parameters, head posture parameters and hand posture parameters in the three-dimensional deformable model are extracted.

6. The method for generating a multi-pose portrait animation according to claim 1, wherein: Inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters, including: Performing feature extraction on the audio data to extract audio features including emotion, intonation and rhythm; The extracted audio features and the facial expression parameters are input into an audio-driven facial motion parameter extraction network to calculate the facial motion parameters, which are used to generate facial dynamic expressions.

7. The method for generating a multi-pose portrait animation as claimed in claim 1, characterized in that: Mapping the head posture parameters, hand posture parameters, facial expression parameters, facial movement parameters, head posture movement parameters and hand posture movement parameters into parameter representations under a 2D view includes: Mapping the head posture parameters, hand posture parameters, facial expression parameters and facial movement parameters to a 2D plane through perspective projection to generate a preliminary parameter representation under a 2D view; According to the facial expression parameters and facial movement parameters, the standard key point positions in the 2D view are adjusted to accurately reflect the changes of facial expressions and postures on the 2D plane; Based on the adjusted standard key point positions and the preliminary parameter representation under 2D view, the head posture motion parameters and hand posture motion parameters are smoothed using an interpolation algorithm to generate the final parameter representation under 2D view, which is used to drive the generation of multi-pose portrait animation.

8. A multi-pose portrait animation generating device, characterized in that: The multi-posture portrait animation generating device comprises: A three-dimensional model generation module is used to obtain a portrait image, convert the portrait image into a three-dimensional deformable model, the three-dimensional deformable model includes facial expression parameters, head posture parameters and hand posture parameters, obtain audio data, input the audio data into an audio-driven facial motion parameter extraction network, and calculate facial motion parameters according to the facial expression parameters; A posture generation module, configured to calculate head posture motion parameters and hand posture motion parameters according to the head posture parameters, hand posture parameters and audio data using a variational autoencoder for posture generation; a 2D mapping module, for extracting appearance features of the portrait image, estimating standard key points of the portrait image to generate standard key points, and mapping the head posture parameters, hand posture parameters, facial expression parameters, facial motion parameters, head posture motion parameters and hand posture motion parameters into parameter representations under a 2D view; The multi-pose animation generation module is used to input the mapped parameter representation, standard key points and appearance features into the image generator to generate a multi-pose portrait animation including hand poses and dynamic expressions.

9. A multi-pose portrait animation generating device, characterized in that: The multi-pose portrait animation generation device includes a memory, a processor, and a multi-pose portrait animation generation program stored in the memory and executable on the processor. When the multi-pose portrait animation generation program is executed by the processor, the steps of the multi-pose portrait animation generation method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a multi-pose portrait animation generation program, which, when executed by a processor, implements the steps of the multi-pose portrait animation generation method according to any one of claims 1 to 7.

Images