Video generation method and device, electronic equipment and storage medium

By obtaining the pose heatmap sequence and hand features in the driver video, and combining the appearance characteristics of the target object for feature fusion, the problem of insufficient performance of human-driven videos in the prior art in fine-grained areas is solved, and the detailed expression of the video is improved .

CN119031203APending Publication Date: 2024-11-26BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411266270.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing human-driven video generation technology has insufficient fine-grained areas such as hand performance, resulting in poor details of the generated human-driven video.

Method used

By obtaining the pose heatmap sequence in the driver video, pose encoding and adding noise, combining the appearance characteristics of the target object and the hand characteristics to generate the target video, and enhancing the display of fine-grained areas such as the hands.

Benefits of technology

Improve the performance quality of generated videos in fine-grained areas such as hands, and enhance the detailed expression of videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031203A_ABST
    Figure CN119031203A_ABST
Patent Text Reader

Abstract

The invention discloses a video generation method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC, digital people, intelligent e-commerce and the like. According to the implementation scheme, an attitude heat map sequence of a driving object in a driving video is obtained; performing attitude coding on the attitude heat map sequence to obtain a first attitude feature of the driving object, and adding noise into the first attitude feature to obtain a second attitude feature; obtaining a target image displaying a target object, and performing appearance coding on the target image to obtain a first appearance feature of the target object; acquiring hand key point information of a driving object in the driving video, and acquiring hand features of the driving object according to the hand key point information; performing feature fusion according to the second posture feature, the first appearance feature and the hand feature to obtain a first fusion feature; and decoding the first fusion feature to generate a target video displaying the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as AIGC (Artificial Intelligence Generated Content), digital humans, and smart e-commerce. Specifically, it relates to a video generation method, device, electronic device and storage medium. Background Art

[0002] Human body driving technology refers to the technology of driving the generation of a target object's limb movement video based on any video of human body movement. The target person in the generated human body driving video has the same limb movement, gestures, facial expressions and lip movements as the driving person. Summary of the Invention

[0003] The present application provides a video generation method, device, electronic device and storage medium.

[0004] According to one aspect of the present application, a video generation method is provided, comprising:

[0005] Obtain a sequence of posture heat maps of the driving object in the driving video;

[0006] Performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driven object, and adding noise to the first posture feature to obtain a second posture feature;

[0007] Acquire a target image showing a target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object;

[0008] Acquire key point information of a hand of a driving object in the driving video, and acquire hand features of the driving object based on the key point information of the hand;

[0009] performing feature fusion based on the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature;

[0010] The first fusion feature is decoded to generate a target video showing the target object.

[0011] According to another aspect of the present application, a video generating apparatus is provided, comprising:

[0012] A first acquisition module is used to acquire a posture heat map sequence of a driving object in a driving video;

[0013] a posture encoding module, configured to perform posture encoding on the posture heat map sequence to obtain a first posture feature of the driven object, and to add noise to the first posture feature to obtain a second posture feature;

[0014] an appearance coding module, configured to acquire a target image showing a target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object;

[0015] a second acquisition module, configured to acquire key point information of a hand of a driving object in the driving video, and acquire hand features of the driving object based on the key point information of the hand;

[0016] a fusion module, configured to perform feature fusion based on the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature;

[0017] A generation module is used to decode the first fusion feature and generate a target video showing the target object.

[0018] According to another aspect of the present application, an electronic device is provided, including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiment.

[0022] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.

[0023] According to another aspect of the present application, a computer program product is provided, including a computer program, which implements the steps of the method described in the above embodiment when executed by a processor.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0026] Figure 1A flowchart of a video generation method provided in one embodiment of the present application;

[0027] Figure 2 A flowchart of a video generation method provided in another embodiment of the present application;

[0028] Figure 3 A flowchart of a video generation method provided in another embodiment of the present application;

[0029] Figure 4 A schematic diagram of a video generation process provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of a video generation device provided in one embodiment of the present application;

[0031] Figure 6 It is a block diagram of an electronic device used to implement the video generation method of an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The following describes the video generation method, device, electronic device, and storage medium according to the embodiments of the present application with reference to the accompanying drawings.

[0034] Figure 1 A flowchart of a video generation method provided in one embodiment of the present application.

[0035] In some embodiments, a 3D human body mesh can be used as a driving motion signal to generate a human-driven video. However, this approach focuses more on generating human limb movements, resulting in poor performance in fine-grained areas such as hands.

[0036] Based on this, an embodiment of the present application provides a video generation method. The video generation method of the embodiment of the present application can be executed by a video generation device of the embodiment of the present application, and the device can be configured in an electronic device.

[0037] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, mobile terminal, server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0038] like Figure 1 As shown, the video generation method includes:

[0039] Step 101: Acquire a posture heat map sequence of a driving object in a driving video.

[0040] In this application, the driving video can be any given video used to drive the action of a target object, where the target object can be a digital human, a virtual human, or a character, etc.

[0041] For example, human key point detection can be performed on each frame image in the driving video in turn to obtain the human key point information of the driving object in each frame image, and the human key point information of the driving object in each frame image can be drawn to obtain the posture heat map of the driving object in each frame image. Then, the posture heat map corresponding to each frame image is sorted according to the time sequence of each frame image in the driving video to form a posture heat map sequence.

[0042] Pose heatmap is a method of converting the coordinates of key points of the human body into image representation. Pose heatmap can be achieved by generating a heatmap on the image for each key point.

[0043] For example, the shape of the pose heatmap sequence can be F*H*W*3, where F, H, and W represent the length, height, and width of the pose heatmap sequence, respectively. The length F of the pose heatmap sequence can be understood as the number of frames in the time dimension. For example, if the length F is 50, it means that the pose heatmap sequence is a sequence of pose heatmaps corresponding to 50 frames of images in the driving video.

[0044] Exemplarily, the acquired human body key point information of the driving object may be 2D key point information, that is, the 2D key points may be used as human body driving signals. 2D key points are easy to acquire and relatively robust.

[0045] Step 102 : performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driving object, and adding noise to the first posture feature to obtain a second posture feature.

[0046] For example, a posture encoder can be used to sequentially perform posture encoding on the posture heat maps in the posture heat map sequence to obtain a first posture feature containing whole-body posture information of the driven object at different time points. The first posture feature can include posture information about various parts of the driven object, such as the head, torso, and limbs.

[0047] Exemplarily, the pose encoder can be a convolutional neural network with 8 convolutional layers.

[0048] For example, the first posture feature can be added to the noise to obtain a second posture feature, and then the second posture feature can be input into the denoising network. The noise used can be Gaussian noise, which is not limited to this. For example, the denoising network can be a U-net network, which can include a convolution layer, a maximum pooling layer (downsampling), an attention layer, a deconvolution layer (upsampling), and a ReLU nonlinear activation function.

[0049] Step 103 : Acquire a target image showing the target object, and perform appearance coding on the target image to obtain appearance features of the target object.

[0050] VAE (Variational AutoEncoder) is a generative model that combines the concepts of autoencoders and variational Bayesian inference. VAEs can learn the latent representation of input data and generate new, similar data samples based on the latent representation. VAEs consist of an encoder and a decoder. The encoder maps the input data to a point in a latent space, also known as the latent representation, while the decoder generates new, similar data samples based on the latent representation.

[0051] In this application, the target image can be input into the encoder of a pre-trained VAE (Variational AutoEncoder), and the encoder in the VAE is used to encode the target image to obtain the appearance features of the latent space. The appearance features of the latent space are then input into the appearance encoder, and the appearance encoder is used to further encode the appearance features of the latent space to obtain the first appearance features of the target object. For example, the shape of the first appearance feature can be h*w*c, where h, w, and c represent the height, width, and number of channels, respectively.

[0052] Exemplarily, the appearance encoder can have the same network structure as the denoising network, for example, both the appearance encoder and the denoising network can employ a U-net network. Exemplarily, the appearance encoder can include multiple appearance attention layers, each of which can perform self-attention calculations on input features. Multiple appearance attention layers can be used to obtain first appearance features at different scales.

[0053] In this application, the encoder and appearance encoder in the VAE can be used to encode the appearance information of the target object to obtain the first appearance feature of the target object. The appearance information may include but is not limited to texture information, facial information, body shape, clothing, etc.

[0054] Step 104 : obtaining the key point information of the hand of the driving object in the driving video, and obtaining the hand features of the driving object based on the key point information of the hand.

[0055] Exemplarily, based on the above-mentioned human body key point detection of each frame image in the driving video, the human body key points of the driving object in each frame image are obtained. The hand key point information of the driving object in each frame image can be obtained from the human body key points of the driving object in each frame image, and then the hand features of the driving object can be obtained based on the hand key point information of the driving object in each frame image.

[0056] For example, hand key point detection can be performed on each frame image in the driving video in sequence to obtain the hand key point information of the driving object in each frame image, and then the hand features of the driving object can be obtained based on the hand key point information of the driving object in each frame image.

[0057] For example, a pre-trained hand encoder can be used to encode the hand key point information of the driving object in each frame image to obtain the hand features of the driving object in each frame image, and then the hand features corresponding to each frame image are sorted according to the time sequence of each frame image to obtain the final hand features of the target object.

[0058] Step 105 , performing feature fusion based on the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature.

[0059] In the present application, the denoising function can be achieved by performing feature fusion on the second posture feature, the first appearance feature and the hand feature.

[0060] Since the first appearance feature is obtained by appearance encoding a single image, while the second posture and hand features involve multiple frames of images in the driving video, to ensure the order of feature fusion, the second appearance feature can be obtained based on multiple first appearance features. For example, the first appearance feature can be repeated multiple times to obtain the second appearance feature. Assuming that the second posture and hand features involve F frames of images and the shape of the first appearance feature is h*w*c, then the shape of the second appearance feature obtained by repeating the first appearance feature F times is F*h*w*c.

[0061] Afterwards, a first fused feature can be obtained by performing feature fusion on the second posture feature, the second appearance feature, and the hand feature. Exemplarily, the denoising network can include an appearance attention layer and a hand attention layer. The second posture feature and the second appearance feature can be input into the appearance attention layer for feature fusion, and then the features output by the appearance attention layer and the hand features can be input into the hand attention layer for feature fusion to obtain the first fused feature. Exemplarily, the appearance attention layer can use cross-attention calculation to capture the association and interaction between the second posture feature and the appearance feature, and the hand attention layer can use cross-attention calculation to capture the association and interaction between the features output by the appearance attention layer and the hand feature.

[0062] Step 106: decode the first fusion feature to generate a target video showing the target object.

[0063] For example, the decoder in the VAE can be used to decode the first fused feature to generate a target video. The target video displays a target object, and the target object's motion is consistent with the motion of the driving object in the driving video. Thus, a video of the target object's motion is generated based on a given driving video of a human body's motion.

[0064] In an embodiment of the present application, the hand features of the driving object are obtained based on the key hand information of the driving object in the driving video, and feature fusion is performed based on the appearance features and posture features containing noise of the driving object combined with the hand features to achieve denoising and generate a video of the driving target object, thereby enhancing the display of the generated video in fine-grained areas such as the hand, thereby improving the quality of the generated video.

[0065] Figure 2 A flowchart of a video generation method provided in another embodiment of the present application.

[0066] like Figure 2 As shown, the video generation method includes:

[0067] Step 201: Acquire a posture heat map sequence of a driving object in a driving video.

[0068] In the present application, step 201 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0069] Step 202 : performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driving object, and adding noise to the first posture feature to obtain a second posture feature.

[0070] In the present application, step 202 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0071] Step 203 : Acquire a target image showing the target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object.

[0072] In the present application, step 203 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0073] Step 204: Acquire key point information of the hand of the driving object in the driving video.

[0074] In this application, step 204 can be implemented in any of the embodiments of this application, so it will not be repeated here.

[0075] Step 205: Position encoding is performed on the key point information of the hand to obtain high-dimensional spatial features of the hand.

[0076] Since the hand area accounts for a limited proportion in the driving signal, the driving information of the hand becomes weaker after passing through the posture encoder. Therefore, in this embodiment, attention can be paid to the hand area to enhance the driving signal of the hand.

[0077] For example, the hand key point information can be transformed into a high-dimensional space through Fourier encoding to perform position encoding on the hand key point information to obtain the hand high-dimensional space features.

[0078] Since the hand key point information can be the hand key point information of the driving object in each frame image in the driving video, the hand key point information corresponding to each frame image can be position-encoded in sequence to obtain the high-dimensional spatial features of the hand.

[0079] For example, by encoding the position of the hand key point information corresponding to F frame images in sequence, the shape of the obtained high-dimensional spatial feature of the hand can be F*2*2m(L+1), where the first 2 represents both hands, the second 2 represents the x and y coordinates of a hand key point, m represents the number of key points for each hand, and L represents the number of Fourier code expansions.

[0080] In step 206, linear transformation and self-attention calculation are performed on the high-dimensional spatial features of the hand to obtain a coefficient matrix sequence.

[0081] For example, the hand spatial features can be linearly transformed through multiple linear layers and self-attention calculated through a self-attention layer to obtain a coefficient matrix sequence. The coefficient matrix sequence includes multiple coefficient matrices, the number of elements in each coefficient matrix is ​​the same as the number of hand key points, and one element in each coefficient matrix corresponds to one hand key point of the driving object.

[0082] For example, if the shape of the high-dimensional spatial feature of the hand can be F*2*2m(L+1), the shape of the coefficient matrix sequence can be F*2*m, that is, F coefficient matrices of size 2*m. It can be understood that each frame image in the F frame image corresponds to a coefficient matrix of shape 2*m, where 2 refers to both hands. Each row element in the 2*m coefficient matrix represents the coefficient corresponding to the key point on one hand, and each element represents the coefficient corresponding to a key point of the hand of the driving object.

[0083] Step 207: Based on the coefficient matrix sequence, weight the basis vectors corresponding to the hand key points to obtain hand features.

[0084] In order to represent the texture information of the hand, a key point-based hand texture codebook can be pre-constructed in the present application, wherein the hand texture codebook contains basis vectors corresponding to each hand key point, and the basis vectors corresponding to each hand key point are mutually orthogonal, which can fully express the texture information corresponding to each key point.

[0085] For example, the basis vectors corresponding to the hand key points may be weighted in sequence according to the coefficient matrices in the coefficient matrix sequence to obtain the hand features.

[0086] For example, the shape of the coefficient matrix sequence can be F*2*m, with m basis vectors. Based on the coefficient matrix sequence, the m basis vectors are linearly weighted to obtain hand features. The shape of the hand features is F*2*d1, where d1 represents the dimension of the basis vector.

[0087] Step 208: performing feature fusion based on the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature.

[0088] In the present application, step 208 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0089] Step 209 : Decode the first fusion feature to generate a target video showing the target object.

[0090] In the present application, step 209 can be implemented in any of the embodiments of the present application, so it will not be described in detail here.

[0091] In the embodiment of the present application, the position encoding of the key point information of the hand is performed to enhance the key point's ability to represent the hand structure. At the same time, the discretized basis vectors can be used to robustly represent the texture information of the hand. By weighting the basis vectors corresponding to each hand key point, the hand features are obtained, the hand driving signal is enhanced, and the details of the hand area in the generated video are enhanced, so that the hand area in the generated video is more accurate.

[0092] Figure 3A flowchart of a video generation method provided in another embodiment of the present application.

[0093] like Figure 3 As shown, the video generation method includes:

[0094] Step 301: Acquire a posture heat map sequence of a driving object in a driving video.

[0095] In this application, step 301 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0096] Step 302 : performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driving object, and adding noise to the first posture feature to obtain a second posture feature.

[0097] In this application, step 302 can be implemented in any of the embodiments of this application, so it will not be described here in detail.

[0098] Step 303: Acquire a target image showing the target object, and perform appearance coding on the target image to obtain appearance features of the target object.

[0099] In this application, step 303 can be implemented in any of the embodiments of this application, so it will not be described here in detail.

[0100] Step 304 : Acquire the key point information of the hand of the driving object in the driving video, and acquire the hand features of the driving object based on the key point information of the hand.

[0101] In this application, step 304 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0102] Step 305: Acquire a face image aligned with the target object from the target image.

[0103] The aligned face image may refer to a face image in which the face of the target object has a standardized posture.

[0104] In this application, an initial face image of the target object can be extracted from the target image, and then the initial face image can be subjected to posture normalization processing to obtain an aligned face image of the target object.

[0105] Exemplarily, face recognition can be performed on the target image to determine the face area of ​​the target object, and based on the face area, the initial face image of the target object can be extracted from the target image. The initial face image of the target object is then subjected to posture standardization processing to adjust the face posture to the standard posture, thereby obtaining an aligned face image of the target object.

[0106] Exemplarily, posture standardization processing may include adjusting the eyes, nose, mouth, etc. to the same horizontal line, eliminating possible tilt or rotation of the face in the initial face image, making the face present a frontal perspective, reducing side or slanted face situations, etc.

[0107] Therefore, through face extraction and posture standardization processing, a face image with standard posture can be obtained, and a video of driving the target object can be generated based on the face image with standard posture, which can enhance the display of details in the face area.

[0108] Step 306 : performing feature extraction on the aligned facial images to obtain a first global facial feature and a first spatial facial texture feature of the target object.

[0109] Exemplarily, a pre-trained face recognition model can be used to extract features from the aligned face images to obtain a first global face feature, and the encoder in the pre-trained VAE can be used to encode the aligned face images to obtain a first spatial face texture feature.

[0110] Step 307: Obtain facial features based on the first global facial features and the first spatial facial texture features.

[0111] As a possible implementation method, the first global facial feature and the first spatial facial texture feature may be fused to obtain the facial feature.

[0112] As another possible implementation method, the facial posture features of the target object can also be obtained, the first global facial features and the facial posture features are cross-attention calculated to obtain the second global facial features, the first spatial facial texture features and the facial posture features are cross-attention calculated to obtain the second spatial facial texture features, and then the second global facial features and the second spatial facial texture features are added to obtain the facial features.

[0113] Exemplarily, a face mask matrix can be obtained, and based on the face mask matrix, the first posture feature is masked to obtain a facial posture feature. The size of the face mask matrix is ​​the same as that of the first posture feature. For example, in the face mask matrix, the element value corresponding to the facial area can be 1, and the element value corresponding to other human body parts other than the face can be 0. Then, the face mask matrix is ​​multiplied by the first posture feature. Except for the eigenvalue of the facial area, which remains unchanged, the eigenvalues ​​of all other areas will be set to 0. The facial posture feature thus obtained only contains the facial posture information and removes the interference of other non-facial areas.

[0114] Therefore, the first posture feature can be masked using the face mask matrix to retain the facial posture features in the first posture feature. The method is simple and convenient.

[0115] Exemplarily, an initial facial image of the target object may be extracted from the target image, and then gesture encoding is performed on the initial facial image of the target object to obtain facial gesture features of the target object.

[0116] In this embodiment, by performing cross-attention calculation on the first global facial feature and the first spatial facial texture feature extracted from the facial image aligned with the target object and the facial posture feature respectively, and then adding the two calculated features together, a comprehensive facial feature can be obtained, so that the facial feature can better represent the overall information, texture information, etc. of the face, and can enhance the details of the facial area in the generated video.

[0117] Step 308 , performing feature fusion on the second posture feature, the first appearance feature, the hand feature, and the face feature to obtain a first fused feature.

[0118] In this embodiment, the second appearance feature can be obtained based on the length of the posture heat map sequence and the first appearance feature. For example, the first appearance feature of the same length as the sequence is repeated to obtain the second appearance feature. For example, if the length of the posture heat map sequence is F and the first appearance feature is h*w*c, then the shape of the second appearance feature is F*h*w*c. Subsequently, feature fusion is performed on the second posture feature, the second appearance feature, the hand feature, and the facial feature to obtain the first fused feature.

[0119] For example, feature fusion can be performed based on the second posture feature and the second appearance feature to obtain a second fused feature, and feature fusion can be performed based on the second fused feature and the facial feature to obtain a third fused feature. Feature fusion can be performed based on the third fused feature and the hand feature to obtain a fourth fused feature, and then temporal attention calculation can be performed on the fourth fused feature to obtain the first fused feature. For example, the temporal attention calculation can be a self-attention calculation, which can capture the internal dependencies of the fourth fused feature.

[0120] Exemplarily, the second posture feature and the second appearance feature can be input into the appearance attention layer in the denoising network for feature fusion. For example, the shapes of the second posture feature and the second appearance feature are both F*h*w*c. When fusing, the second posture feature and the second appearance feature can be first spliced ​​in the w dimension, and then self-attention calculation can be performed in the spatial dimension, that is, the h dimension and the w dimension. The features of the first half of the spatial dimension of the finally output feature can be used as the result of feature fusion.

[0121] Exemplarily, temporal attention calculation can be a self-attention calculation based on the time dimension and channel dimension in the fourth fusion feature, which can capture the internal dependency of the fourth fusion feature.

[0122] Therefore, the first fusion feature includes facial features and hand features, enriching the feature representation information, and performing temporal attention calculation on the fourth fusion feature can capture the internal dependency of the fourth fusion feature, thereby generating a video based on the obtained first fusion feature, ensuring the temporal consistency of the generated video and the driving video, and reducing the blur and jitter of the generated video.

[0123] For example, to focus more on the face area, a face mask matrix can be obtained, and a cross-attention calculation can be performed on the second fused feature and the face feature to obtain a first intermediate feature. The first intermediate feature can then be masked based on the face mask matrix to obtain a second intermediate feature. The second intermediate feature can then be added to the second fused feature to obtain a third fused feature. The second fused feature and the face feature can be input into the face attention layer for a cross-attention calculation to obtain the first intermediate feature.

[0124] For example, the third fusion feature can be obtained by using the following formula (1):

[0125] latents f = Att(latents r1 ,Gf) *M f + latents r1 (1)

[0126] Among them, latents f Represents the third fusion feature, Att represents the cross attention calculation, latents r1 represents the second fusion feature, Gf represents the face feature, M f Represents the face mask matrix, through Att(latents r1 ,Gf) calculates the first intermediate feature, which is obtained by Att(latents r1 ,Gf)*M f The second intermediate feature is calculated.

[0127] Therefore, when fusing facial features, the face mask matrix can be applied to the features obtained by fusing the second fused features with the facial features, so that the obtained third fused features pay more attention to the facial area.

[0128] For example, to pay more attention to the hand area, a hand mask matrix can be obtained, and a cross-attention calculation can be performed on the third fused feature and the hand feature to obtain a third intermediate feature. Based on the hand mask matrix, the third intermediate feature is masked to obtain a fourth intermediate feature, and then the fourth intermediate feature is added to the third fused feature to obtain a fourth fused feature. The third fused feature and the hand feature can be input into the hand attention layer for cross-attention calculation to obtain the third intermediate feature.

[0129] For example, the fourth fusion feature can be obtained by using the following formula (2):

[0130] latents h = Att(latents r2 , Gh) * M h + latents r2 (2)

[0131] Among them, latents h Represents the fourth fusion feature, Att represents cross attention calculation, latents r2 represents the third fusion feature, Gh represents the hand feature, M h Represents the hand mask matrix, through Att(latents r2 ,Gh) calculates the third intermediate feature, which is obtained by Att(latents r2 ,Gh)*M h What is calculated is the fourth intermediate feature.

[0132] Therefore, when fusing the hand features, the hand mask matrix can be applied to the features obtained by fusing the third fused feature with the hand feature, so that the obtained fourth fused feature pays more attention to the hand area.

[0133] Step 309: decode the first fusion feature to generate a target video showing the target object.

[0134] In this application, step 309 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0135] In an embodiment of the present application, by extracting the first global facial features and the first spatial facial texture features from the aligned facial images, facial features are obtained based on the first global facial features and the first spatial facial texture features, and feature fusion is performed in combination with the facial features according to the second posture features, the first appearance features, the hand features, etc., which can not only make the generated video better display fine-grained areas such as the hands, but also enhance the details of the facial area in the generated video, thereby improving the quality of the facial area in the generated video.

[0136] In order to facilitate understanding of the video generation method of this application, the following Figure 4 To explain, Figure 4 A schematic diagram of a video generation process provided in an embodiment of the present application.

[0137] like Figure 4As shown in the figure, the generation of the video includes four processes: (a) posture information encoding and denoising process; (b) appearance information encoding; (c) key point-based fine-grained hand modeling; (d) face enhancement.

[0138] In the embodiment of the present application, a more robust and easily accessible 2D key point can be used as a human body driving signal, and a posture heat map sequence is obtained by drawing the 2D key point, the shape of which is F*H*W*3, where F, H, and W represent the sequence length, height, and width of the posture heat map, respectively. Figure 4 As shown, a pose encoder can be used to perform pose encoding on the pose heatmap sequence. The encoded pose features are added to Gaussian noise and then fed into a denoising network for noise prediction. For example, a convolutional neural network with 8 convolutional layers can be used as the pose encoder.

[0139] To extract the appearance information of the target object, a network with the same structure as the denoising network can be used as an appearance encoder. First, the target object image is fed into a pre-trained VAE encoder to obtain an appearance feature map in the latent space. This map is then fed into the appearance encoder to obtain appearance feature maps at different scales. This process is repeated F times in the temporal dimension to obtain the final appearance feature map Zr, with a shape of F*h*w*c. Finally, it is fused with the features Zt from the denoising network at the corresponding scale, with a shape of F*h*w*c. This fusion can be performed by concatenating Zr and Zt in the w dimension, then performing self-attention calculations in the spatial dimensions, i.e., the h and w dimensions. The features of the first half of the spatial dimension of the final output can be used as the result of feature fusion.

[0140] Since the hand area accounts for a limited proportion in the driving signal, the driving information of the hand becomes weaker after passing through the posture encoder. In order to enhance the driving signal of the hand and generate a more accurate hand area, an embodiment of the present application provides a fine-grained hand modeling module based on key points. First, the key points of the hand (including the key points of the left hand and the key points of the right hand) are transformed into a high-dimensional space through Fourier coding to obtain the high-dimensional space feature Fh of the hand, which has a shape of F*2*2m(L+1), where the first 2 represents both hands, the second 2 represents the x and y coordinates, m represents the number of key points of each hand, and L represents the number of Fourier code expansions. In order to represent the texture information of the hand, a hand texture codebook based on key points can be pre-constructed, where the codebook contains m basis vectors γ1, γ2, γ3, ...γ m-1 , γ m, corresponding to each key point of the hand, each basis vector in the codebook is orthogonal to each other and can fully express the texture information corresponding to each key point. The high-dimensional spatial feature Fh of the hand obtains a set of coefficients A (that is, the coefficient matrix series in the above embodiment) through multiple linear layers and self-attention layers, with a shape of F*2*m. Then, the hand feature Gh is obtained by linearly weighting the m basis vectors, with a shape of F*2*d1, where d1 represents the dimension of the basis vector. Finally, the hand feature Gh is sent to the hand attention layer in the denoising network for feature fusion. Figure 4 middle, Represents the coefficient matrix corresponding to a frame of image, such as the first row element The coefficients of the m key points of the left hand can be represented. The second row of elements The coefficients of the m key points on the right hand can be expressed, where one key point corresponds to one coefficient.

[0141] In order to improve the quality of the face area in the generated video, this embodiment provides a face enhancement module. First, a face extraction and alignment algorithm is used to obtain an aligned face photo from the target image. Then, a pre-trained face recognition model is used to extract the first global face feature. A pre-trained VAE encoder is used to obtain the first spatial face texture feature. Then, cross-attention is performed with the facial posture feature to obtain the second global face feature Gid and the second spatial face texture feature Gtex. Finally, Gid and Gtex are added to obtain a comprehensive face feature Gf, which has the shape of F*h*w*df. Figure 4 The face enhancement module shown in

[15] can output facial features Gf. The facial features Gh are fed into the face attention layer in the denoising network for feature fusion.

[0142] When performing feature fusion on the hand feature Gf and the face feature Gh, the face mask matrix Mf and the face mask matrix Mh can be applied to the fused features respectively to focus more on specific areas.

[0143] Figure 4 The denoising network shown in the figure includes a downsampling layer (the left area network in the denoising network), an intermediate layer (the middle area network in the denoising network), and an upsampling layer (the right area network in the denoising network), where the downsampling layer includes three groups of attention layers ( Figure 4 A set of attention layers is shown in the figure), each set of attention layers includes appearance attention layer, face attention layer, and temporal attention layer in sequence, the middle layer includes appearance attention layer, face attention layer, hand attention layer, and temporal attention layer in sequence, and the upsampling layer can include three sets of attention layers ( Figure 4A group of attention layers is shown in the figure), each group of attention layers includes an appearance attention layer, a face attention layer, a hand attention layer and a temporal attention layer in sequence. The appearance features of different scales output by the appearance encoder can be input into the appearance attention layer in the corresponding denoising network to perform feature fusion of the appearance features with the features output by the previous layer. The face features can be input into each face attention layer in the denoising network. The face attention layer can perform feature fusion of the face features with the features output by the previous layer. The hand features can be input into each hand attention layer in the denoising network to fuse the hand features with the features output by the previous layer. The features output by the hand attention layer can be input into the temporal attention layer for temporal attention calculation. The features output by the denoising network can be input into the VAE decoder for decoding to generate a video.

[0144] In order to implement the above embodiment, the embodiment of the present application also proposes a video generating device. Figure 5 A schematic diagram of the structure of a video generation device provided in one embodiment of the present application.

[0145] like Figure 5 As shown, the video generating device 500 includes:

[0146] A first acquisition module 510 is configured to acquire a posture heat map sequence of a driving object in a driving video;

[0147] a posture encoding module 520 for performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driven object, and adding noise to the first posture feature to obtain a second posture feature;

[0148] An appearance coding module 530 is configured to obtain a target image showing a target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object;

[0149] A second acquisition module 540 is configured to acquire key point information of a hand of a driving object in the driving video, and acquire hand features of the driving object based on the key point information of the hand;

[0150] a fusion module 550 for performing feature fusion based on the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature;

[0151] The generating module 560 is configured to decode the first fusion feature and generate a target video showing the target object.

[0152] Optionally, the second acquisition module 540 is configured to:

[0153] Position encoding is performed on the key point information of the hand to obtain a high-dimensional spatial feature of the hand;

[0154] Performing linear transformation and self-attention calculation on the high-dimensional spatial features of the hand to obtain a coefficient matrix sequence;

[0155] Based on the coefficient matrix sequence, basis vectors corresponding to each hand key point are weighted to obtain the hand features; wherein the basis vectors corresponding to each hand key point are orthogonal to each other.

[0156] Optionally, the fusion module 550 is configured to:

[0157] Acquire a face image aligned with the target object from the target image;

[0158] Performing feature extraction on the aligned facial images to obtain a first global facial feature and a first spatial facial texture feature of the target object;

[0159] Obtaining a facial feature based on the first global facial feature and the first spatial facial texture feature;

[0160] The second posture feature, the first appearance feature, the hand feature and the face feature are fused to obtain a first fused feature.

[0161] Optionally, the fusion module 550 is configured to:

[0162] Acquire facial posture features of the target object;

[0163] Performing cross-attention calculation on the first global face feature and the facial posture feature to obtain a second global face feature;

[0164] Performing cross-attention calculation on the first spatial facial texture feature and the facial posture feature to obtain a second spatial facial texture feature;

[0165] The second global facial feature and the second spatial facial texture feature are added to obtain a facial feature.

[0166] Optionally, the fusion module 550 is configured to:

[0167] Get the face mask matrix;

[0168] Based on the face mask matrix, mask processing is performed on the first posture feature to obtain the facial posture feature.

[0169] Optionally, the fusion module 550 is configured to:

[0170] Acquire a second appearance feature according to the length of the posture heat map sequence and the first appearance feature;

[0171] Performing feature fusion based on the second posture feature and the second appearance feature to obtain a second fused feature;

[0172] Performing feature fusion based on the second fusion feature and the facial feature to obtain a third fusion feature;

[0173] Performing feature fusion based on the third fusion feature and the hand feature to obtain a fourth fusion feature;

[0174] Perform temporal attention calculation on the fourth fusion feature to obtain the first fusion feature.

[0175] Optionally, the fusion module 550 is configured to:

[0176] Get the face mask matrix;

[0177] Performing a cross attention calculation on the second fusion feature and the facial feature to obtain a first intermediate feature;

[0178] Based on the face mask matrix, masking the first intermediate feature to obtain a second intermediate feature;

[0179] The second intermediate feature and the second fused feature are added to obtain the third fused feature.

[0180] Optionally, the fusion module 550 is configured to:

[0181] Get the hand mask matrix;

[0182] Performing cross attention calculation on the third fusion feature and the hand feature to obtain a third intermediate feature;

[0183] performing mask processing on the third intermediate feature based on the hand mask matrix to obtain a fourth intermediate feature;

[0184] The fourth intermediate feature is added to the third fused feature to obtain the fourth fused feature.

[0185] Optionally, the fusion module 550 is configured to:

[0186] Performing face recognition on the target image to determine the face area of ​​the target object;

[0187] Extracting an initial face image of a target object from the target image according to the face area;

[0188] Performing posture normalization processing on the initial face image of the target object to obtain an aligned face image of the target object.

[0189] It should be noted that the explanation of the aforementioned video generation method embodiment is also applicable to the video generation device of this embodiment, and therefore will not be repeated here.

[0190] In an embodiment of the present application, the hand features of the driving object are obtained based on the key hand information of the driving object in the driving video, and feature fusion is performed based on the appearance features and posture features containing noise of the driving object combined with the hand features to achieve denoising and generate a video of the driving target object, thereby enhancing the display of the generated video in fine-grained areas such as the hand, thereby improving the quality of the generated video.

[0191] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0192] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0193] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 602 or a computer program loaded from a storage unit 608 into a RAM (Random Access Memory) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0194] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0195] The computing unit 601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the video generation method. For example, in some embodiments, the video generation method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the video generation method in any other appropriate manner (eg, by means of firmware).

[0196] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0197] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0198] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0200] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0201] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0202] According to an embodiment of the present application, the present application also provides a computer program product, which, when an instruction processor in the computer program product executes, executes the video generation method proposed in the above embodiment of the present application.

[0203] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0204] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A video generation method, comprising: Obtain a posture heat map sequence of the driving object in the driving video; Performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driven object, and adding noise to the first posture feature to obtain a second posture feature; Acquire a target image showing a target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object; Acquire the key point information of the hand of the driving object in the driving video, and acquire the hand features of the driving object according to the key point information of the hand; Performing feature fusion according to the second posture feature, the first appearance feature and the hand feature to obtain a first fused feature; The first fusion feature is decoded to generate a target video showing the target object.

2. The method of claim 1, wherein: The step of acquiring the hand features of the driving object according to the hand key point information includes: Position encoding is performed on the key point information of the hand to obtain a high-dimensional spatial feature of the hand; Performing linear transformation and self-attention calculation on the high-dimensional spatial features of the hand to obtain a coefficient matrix sequence; Based on the coefficient matrix sequence, basis vectors corresponding to each hand key point are weighted to obtain the hand feature; wherein the basis vectors corresponding to each hand key point are orthogonal to each other.

3. The method of claim 1, wherein: The step of fusing the second posture feature, the first appearance feature, and the hand feature to obtain a first fused feature includes: Acquire a face image aligned with the target object from the target image; Performing feature extraction on the aligned face images to obtain a first global face feature and a first spatial face texture feature of the target object; Obtaining a facial feature according to the first global facial feature and the first spatial facial texture feature; The second posture feature, the first appearance feature, the hand feature and the face feature are fused to obtain a first fused feature.

4. The method of claim 3, wherein: The obtaining of facial features according to the first global facial features and the first spatial facial texture features includes: Acquire facial posture features of the target object; Performing cross-attention calculation on the first global face feature and the facial posture feature to obtain a second global face feature; Performing cross-attention calculation on the first spatial face texture feature and the facial posture feature to obtain a second spatial face texture feature; The second global facial feature and the second spatial facial texture feature are added to obtain a facial feature.

5. The method of claim 4, wherein: The step of acquiring the facial posture feature of the target object comprises: Get the face mask matrix; Based on the face mask matrix, mask processing is performed on the first posture feature to obtain the facial posture feature.

6. The method of claim 3, wherein: The step of fusing the second posture feature, the first appearance feature, the hand feature, and the face feature to obtain a first fused feature includes: Acquire a second appearance feature according to the length of the posture heat map sequence and the first appearance feature; Performing feature fusion based on the second posture feature and the second appearance feature to obtain a second fused feature; Performing feature fusion based on the second fusion feature and the face feature to obtain a third fusion feature; Performing feature fusion based on the third fusion feature and the hand feature to obtain a fourth fusion feature; Perform temporal attention calculation on the fourth fusion feature to obtain the first fusion feature.

7. The method of claim 6, wherein: The step of fusing the second fused feature with the face feature to obtain a third fused feature includes: Get the face mask matrix; Performing cross attention calculation on the second fusion feature and the face feature to obtain a first intermediate feature; Based on the face mask matrix, masking the first intermediate feature to obtain a second intermediate feature; The second intermediate feature and the second fused feature are added to obtain the third fused feature.

8. The method of claim 6, wherein: The step of performing feature fusion based on the third fusion feature and the hand feature to obtain a fourth fusion feature includes: Get the hand mask matrix; Performing cross attention calculation on the third fusion feature and the hand feature to obtain a third intermediate feature; Based on the hand mask matrix, masking the third intermediate feature to obtain a fourth intermediate feature; The fourth intermediate feature is added to the third fused feature to obtain the fourth fused feature.

9. The method of claim 3, wherein: The step of acquiring the face image aligned with the target object from the target image comprises: Performing face recognition on the target image to determine the face area of ​​the target object; Extracting an initial face image of a target object from the target image according to the face area; Performing posture normalization processing on the initial face image of the target object to obtain an aligned face image of the target object.

10. A video generating device, comprising: A first acquisition module is used to acquire a posture heat map sequence of a driving object in a driving video; A posture encoding module, used for performing posture encoding on the posture heat map sequence to obtain a first posture feature of the driven object, and adding noise to the first posture feature to obtain a second posture feature; An appearance coding module, used to acquire a target image showing a target object, and perform appearance coding on the target image to obtain a first appearance feature of the target object; A second acquisition module is used to acquire the key point information of the hand of the driving object in the driving video, and acquire the hand features of the driving object according to the key point information of the hand; A fusion module, used for performing feature fusion according to the second posture feature, the first appearance feature and the hand feature to obtain a first fusion feature; A generation module is used to decode the first fusion feature and generate a target video showing the target object.

11. The device of claim 10, wherein: The second acquisition module is used to: Position encoding is performed on the key point information of the hand to obtain a high-dimensional spatial feature of the hand; Performing linear transformation and self-attention calculation on the high-dimensional spatial features of the hand to obtain a coefficient matrix sequence; Based on the coefficient matrix sequence, basis vectors corresponding to each hand key point are weighted to obtain the hand feature; wherein the basis vectors corresponding to each hand key point are orthogonal to each other.

12. The device of claim 10, wherein: The fusion module is used to: Acquire a face image aligned with the target object from the target image; Performing feature extraction on the aligned face images to obtain a first global face feature and a first spatial face texture feature of the target object; Obtaining a facial feature according to the first global facial feature and the first spatial facial texture feature; The second posture feature, the first appearance feature, the hand feature and the face feature are fused to obtain a first fused feature.

13. The device of claim 12, wherein: The fusion module is used to: Acquire facial posture features of the target object; Performing cross-attention calculation on the first global face feature and the facial posture feature to obtain a second global face feature; Performing cross-attention calculation on the first spatial face texture feature and the facial posture feature to obtain a second spatial face texture feature; The second global facial feature and the second spatial facial texture feature are added to obtain a facial feature.

14. The device of claim 13, wherein: The fusion module is used to: Get the face mask matrix; Based on the face mask matrix, mask processing is performed on the first posture feature to obtain the facial posture feature.

15. The device of claim 12, wherein: The fusion module is used to: Acquire a second appearance feature according to the length of the posture heat map sequence and the first appearance feature; Performing feature fusion based on the second posture feature and the second appearance feature to obtain a second fused feature; Performing feature fusion based on the second fusion feature and the face feature to obtain a third fusion feature; Performing feature fusion based on the third fusion feature and the hand feature to obtain a fourth fusion feature; Perform temporal attention calculation on the fourth fusion feature to obtain the first fusion feature.

16. The device of claim 15, wherein: The fusion module is used to: Get the face mask matrix; Performing cross attention calculation on the second fusion feature and the face feature to obtain a first intermediate feature; Based on the face mask matrix, masking the first intermediate feature to obtain a second intermediate feature; The second intermediate feature and the second fused feature are added to obtain the third fused feature.

17. The device of claim 15, wherein: The fusion module is used to: Get the hand mask matrix; Performing cross attention calculation on the third fusion feature and the hand feature to obtain a third intermediate feature; Based on the hand mask matrix, masking the third intermediate feature to obtain a fourth intermediate feature; The fourth intermediate feature is added to the third fused feature to obtain the fourth fused feature.

18. The device of claim 12, wherein: The fusion module is used to: Performing face recognition on the target image to determine the face area of ​​the target object; Extracting an initial face image of a target object from the target image according to the face area; Performing posture normalization processing on the initial face image of the target object to obtain an aligned face image of the target object.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Gesture recognition and posture estimation combined modeling method, application method and device

    CN120071437A

  • Digital human video generation method and device, equipment and storage medium

    CN121217996A

  • Animation generation method and apparatus, and device, medium and program product

    WO2026157936A1