Motion video generation method, device, computer equipment and storage medium

Abstract

This application provides a motion video generation method, apparatus, computer device, and storage medium. Based on the motion video generation method of this embodiment, a target person image, target background, and action-driven video are first acquired. Two-dimensional and three-dimensional posture features are then extracted from the action-driven video. These two features are then fused to generate fused posture features. Finally, the target person image, target background, and fused posture features are input into a video generation model to generate the target motion video. This solution utilizes both three-dimensional and two-dimensional motion information to guide action, reducing the probability of ambiguous action generation and significantly improving video generation quality.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a motion video generation method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the advancement of artificial intelligence and image processing technologies, it is now possible to generate videos by driving a person's image through a specified target action sequence. This technology is relatively mature, such as the Latent Image Animator (an image generator based on shallow encoding). With the development of related technologies, diffusion models have become widely used, demonstrating powerful generation and editing capabilities for both image and video generation. Some researchers have also applied them to human pose transfer. For example, DreamPose, MagicDance, MagicAnimate, AnimateAnyone, and MimicMotion are all algorithms for human pose transfer based on diffusion models. These methods all use the pose information of the driving video as a guide to drive the target image to generate the resulting video. However, videos produced by current mainstream frameworks are prone to discontinuity and jitter. Summary of the Invention

[0003] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the defects that motion videos generated in the prior art are prone to picture discontinuity and jitter.

[0004] In a first aspect, the present application provides a motion video generation method, comprising:

[0005] Obtain target person image, target background and action-driven video;

[0006] Extract the two-dimensional posture features and three-dimensional posture features corresponding to each frame according to the action-driven video;

[0007] Fusing the two-dimensional posture feature and the three-dimensional posture feature to obtain a fused posture feature;

[0008] The target person image, target background and fused posture features are input into the video generation model to obtain the target motion video.

[0009] In one embodiment, the three-dimensional posture feature includes rotation angles corresponding to a first number of joint points, and the two-dimensional posture feature and the three-dimensional posture feature are fused to obtain a fused posture feature, including:

[0010] Expanding the initial number of channels of the two-dimensional posture feature to a first number of times, dividing the two-dimensional posture feature into sub-feature graphs corresponding to each joint point in the channel dimension, and determining a first mean and a first standard deviation corresponding to each channel of each sub-feature graph;

[0011] Transforming and mapping the three-dimensional posture feature to obtain a three-dimensional mapping posture feature, and determining a second mean and a second standard deviation corresponding to each joint point in the three-dimensional mapping posture feature; each row of the three-dimensional mapping posture feature corresponds to a joint point, and each second number of columns corresponds to a rotation angle, where the second number is a ratio between the number of initial channels and the number of types of rotation angles;

[0012] For any sub-feature map, a fused sub-feature map is obtained according to the corresponding first mean, first standard deviation, second mean and second standard deviation;

[0013] The fused posture features are obtained according to each fused sub-feature map.

[0014] In one embodiment, obtaining a fused sub-feature map according to the corresponding first mean, first standard deviation, second mean, and second standard deviation includes:

[0015] The intermediate sub-feature map is obtained according to the first expression; the first expression is:

[0016]

[0017] in, is the intermediate sub-feature map, is the i-th sub-feature map, is the first mean, is the first standard deviation, is the second mean, is the second standard deviation;

[0018] The intermediate sub-feature map is weightedly summed with the corresponding sub-feature map to obtain a fused sub-feature map.

[0019] In one embodiment, the process of acquiring the target background includes:

[0020] Perform human body detection on the original background to obtain the human body area mask;

[0021] The target background is obtained based on the human body region mask and the original background.

[0022] In one embodiment, before obtaining the target background according to the human body region mask and the original background, the method further includes:

[0023] The human body area mask is sequentially subjected to closing and dilation operations.

[0024] In one embodiment, a video generation model includes an encoder, a feature extraction module, a UNET network, and a decoder. The target person image, target background, and fused posture features are input into the video generation model to obtain a target motion video, including:

[0025] Input the fused posture features into the downsampling layer of the UNET network;

[0026] Input the target person image into the encoder;

[0027] Add potential noise encoding to the target background and sum it with the output of the encoder; the summation result is input to the downsampling layer of the UNET network;

[0028] The target person image is input into the feature extraction module; the result of the feature extraction module is input into the downsampling layer and upsampling layer of the UNET network, and the output of the upsampling layer of the UNET network is connected to the decoder, which is used to generate the target motion video based on the output of the upsampling layer of the UNET network.

[0029] In one embodiment, the training process of the video generation model includes:

[0030] The training 3D posture features are obtained by driving the video based on the training action;

[0031] The predicted three-dimensional posture features are obtained by using the video generation model to obtain the predicted motion video from the training action-driven video;

[0032] Obtain a target loss value based on the predicted 3D posture features and the trained 3D posture features;

[0033] Tune the video generation model based on the target loss value.

[0034] In one embodiment, obtaining a target loss value based on the predicted 3D pose features and the trained 3D pose features includes:

[0035] The target loss value is obtained according to the second expression, the predicted three-dimensional posture features and the training three-dimensional posture features; the second expression is:

[0036]

[0037] in, is the target loss value, n is the number of joint points in the 3D posture feature, is the training 3D posture feature of the i-th joint point, is the predicted 3D pose feature of the i-th joint point.

[0038] In a second aspect, the present application provides a motion video generation device, comprising:

[0039] Data acquisition module, used to obtain the target person image, target background and action-driven video;

[0040] A posture feature extraction module is used to extract the two-dimensional posture features and three-dimensional posture features corresponding to each frame of the action-driven video;

[0041] A posture feature fusion module is used to fuse two-dimensional posture features and three-dimensional posture features to obtain fused posture features;

[0042] The video generation module is used to input the target person image, target background and fusion posture features into the video generation model to obtain the target motion video.

[0043] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the steps of the motion video generation method in any of the above embodiments are executed.

[0044] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the motion video generation method in any of the above embodiments.

[0045] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0046] The motion video generation method in this embodiment first obtains the target person image, target background, and action-driven video. It then extracts 2D and 3D posture features from the action-driven video. These features are then fused to create a fused posture feature. Finally, the target person image, target background, and fused posture feature are input into a video generation model to generate the target motion video. This solution utilizes both 3D and 2D motion information to guide action, reducing the probability of ambiguous action generation and significantly improving video generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a flow chart of a motion video generation method provided in one embodiment of the present application;

[0049] Figure 2This is a schematic diagram of a process for obtaining fused posture features in one embodiment of the present application;

[0050] Figure 3 This is a schematic diagram of the process of obtaining fused posture features in one embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of a process for obtaining a target background in one embodiment of the present application;

[0052] Figure 5 A schematic diagram of a process for obtaining a target background in one embodiment of the present application;

[0053] Figure 6 This is a structural diagram of a video generation model in one embodiment of the present application;

[0054] Figure 7 This is a schematic diagram of the working process of the video generation model in one embodiment of the present application;

[0055] Figure 8 This is a flowchart of training a video generation model in one embodiment of the present application;

[0056] Figure 9 A diagram of the internal structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] This application provides a motion video generation method, see Figure 1 , including steps S102 to S108.

[0059] S102, obtaining a target person image, a target background, and an action-driven video.

[0060] The target character image refers to the appearance of the person the user desires to be represented in the generated video. This can be a character image captured from user-uploaded images or videos. The character image contains information such as the person's appearance, body shape, and clothing, which will be used to shape the character's image when the video is subsequently generated. The target background refers to the scene in which the character is located after the video is generated. Its sources vary, such as user-provided images of specific scenes or images of natural scenery or indoor environments selected from a library of resources. Action-driven videos are videos containing specific action sequences and are the key basis for guiding the target character to perform corresponding actions.

[0061] The principle of this step is to provide the foundational material for subsequent video generation. By acquiring these three types of data, the human motion transfer algorithm based on the diffusion model is fed with input conditions to integrate the target person image into the target background and enable it to perform the movements in the motion-driven video. Throughout the video generation process, the target person image, target background, and motion-driven video work together. The target person image determines the main features of the person in the video, the target background provides the context for the person's movements, and the motion-driven video imparts dynamic behavior. Together, these three elements constitute the complete video generation elements. These three types of data can be acquired by building a user interface. For example, in a web or mobile application, users can select a locally stored full-body photo as the target person image, select a local image or a target background from the application's built-in background library, and simultaneously upload a locally stored motion-driven video. If the user needs to use a person from a video or image they uploaded as the target person image, person detection and cropping can be used to crop out only the person image, decoupling it from the original background. This allows the model to better generate videos based solely on the person image.

[0062] S104 , extracting two-dimensional posture features and three-dimensional posture features corresponding to each frame according to the action-driven video.

[0063] As you can understand, 2D pose features describe the positions of human joints and the relative relationships of limbs on a two-dimensional plane, providing a visual representation of the shape of human motion on a two-dimensional plane. 3D pose features describe the positions of human joints, limb rotation angles, and spatial relationships in three-dimensional space, more accurately conveying the depth and three-dimensionality of human motion. Specifically, 2D pose features can be extracted from human body images using mediapipe. It detects multiple key human joints, such as the head, shoulders, elbows, wrists, hips, knees, and ankles, and converts their positional information into feature data. 3D pose features can be extracted from human body images using PyMAF-X. It accurately estimates 3D human pose and shape information based on monocular images, presenting them as pose parameters of the SMPL model, including the rotation angles of 24 key points. The relevant functions and interfaces of mediapipe and PyMAF-X can be directly called. For example, in Python, after installing the corresponding libraries, use mediapipe's human pose detection module and input each frame of a motion-driven video to obtain 2D pose feature data, such as the coordinates of each joint. For three-dimensional posture features, the frame image is input into the PyMAF-X model, and after calculation and processing by the model, the posture parameters of the SMPL model are output.

[0064] This step provides multi-dimensional posture information for subsequent action guidance. Two-dimensional posture features can quickly capture the general outline and basic posture changes of human movements, while three-dimensional posture features supplement depth and spatial information. The combination of the two provides a more comprehensive description of human movements. Throughout the video generation process, these features work closely with other steps. The extracted posture features provide the data foundation for feature fusion in step S106, providing accurate action guidance for the video generation model, ensuring that the character movements in the generated video are more consistent with expectations.

[0065] S106: Fusing the two-dimensional posture feature and the three-dimensional posture feature to obtain a fused posture feature.

[0066] It can be understood that the fusion of two-dimensional posture features and three-dimensional posture features is to combine the advantages of both and provide more accurate and comprehensive action guidance information. In the present invention, the fusion process is based on specific algorithms and operations. First, the two-dimensional posture feature map is mapped to a higher-dimensional space, and the number of channels is increased to align it with the number of joint points in the three-dimensional posture feature. Then, the mean and standard deviation of the two-dimensional posture feature map and the three-dimensional posture feature vector are calculated respectively. These statistics are used to remove the original features in the two-dimensional posture feature map, and then the three-dimensional posture features are integrated into it. Finally, a fused posture feature map is obtained through weighted fusion.

[0067] This step aims to address the shortcomings of using 2D and 3D pose features separately. While 2D pose features can capture the planar nature of motion, they lack depth information. While 3D pose features can accurately describe spatial motion, they can be less intuitive in some situations. By combining the strengths of both, the video generation model can be provided with richer and more accurate motion guidance, reducing ambiguity in motion results and making the generated character movements more natural and plausible.

[0068] S108, inputting the target person image, target background and fused posture features into a video generation model to obtain a target motion video.

[0069] As you can understand, the video generation model is a system built on the diffusion model, with core components including the UNet, encoder, and decoder. After the target person image, target background, and fused posture features are input into the video review model, the diffusion model uses an iterative denoising process. Guided by the fused posture features, it gradually adjusts the generated image to produce a target action video in which the target person image performs similar actions in the action-driven video against the target background.

[0070] The motion video generation method in this embodiment first obtains the target person image, target background, and action-driven video. It then extracts 2D and 3D posture features from the action-driven video. These features are then fused to create a fused posture feature. Finally, the target person image, target background, and fused posture feature are input into a video generation model to generate the target motion video. This solution utilizes both 3D and 2D motion information to guide action, reducing the probability of ambiguous action generation and significantly improving video generation quality.

[0071] In one embodiment, the three-dimensional posture feature includes the rotation angles corresponding to the first number of joints. If the three-dimensional posture feature is a Smpl model, the number of joints is 24, and includes three different rotation angles. The two-dimensional posture feature and the three-dimensional posture feature are fused to obtain a fused posture feature. Figure 2 , including steps S202 to S208.

[0072] S202: Expand the initial number of channels of the two-dimensional posture feature to a first number of times, divide the two-dimensional posture feature into sub-feature graphs corresponding to each joint point in the channel dimension, and determine a first mean and a first standard deviation corresponding to each channel of each sub-feature graph.

[0073] It can be understood that the two-dimensional posture feature is feature data that describes the posture of the human body on a two-dimensional plane, usually in the form of a feature map, which contains the position information of the human body joints on the plane. The initial number of channels is the initial number of channels of the two-dimensional posture feature map, which reflects the dimensional information of the feature map. The first number refers to the number of joints in the three-dimensional posture feature. The sub-feature map is obtained by dividing the two-dimensional posture feature after the number of channels is expanded on the channel dimension. Each sub-feature map corresponds to a joint point, which is used for the subsequent fusion of the three-dimensional posture features corresponding to the joint point. The first mean and the first standard deviation are the mean and standard deviation of the data on each channel of the sub-feature map, respectively, and are used for subsequent feature fusion operations. They can reflect the central tendency and discreteness of the sub-feature map data.

[0074] This step is to make the two-dimensional posture features match the number of joint points of the three-dimensional posture features in the channel dimension, so as to facilitate effective feature fusion. By expanding the initial number of channels of the two-dimensional posture features to the first number of times, a corresponding feature space can be allocated to each joint point in the channel dimension. After dividing the sub-feature graphs corresponding to each joint point, the first mean and first standard deviation of each channel of each sub-feature graph are calculated. These statistics will be used to adjust the distribution of the sub-feature graphs in the subsequent fusion process, so that they can be better integrated with the three-dimensional posture features, thereby fully combining the advantages of two-dimensional and three-dimensional posture features and laying the foundation for the final generation of more accurate fused posture features.

[0075] Figure 3The upper part shows the operation of this step. Taking the number of joints as an example, the two-dimensional posture features are represented by green blocks, and the initial number of channels is c. The first number is 24 (the figure shows that the feature map is divided into 24 parts according to the joint points). After the number of channels is expanded to 24 times, it is divided into 24 sub-feature maps in the channel dimension (different color blocks represent different sub-feature maps). By kpi ) calculation to obtain the first mean value corresponding to each channel μ (map kpi ) and the first standard deviation σ(map kpi ) (i represents different sub-feature maps, from 1 to 24). The purpose of this operation is to prepare for the subsequent fusion with the three-dimensional posture features, so that the two-dimensional posture features are adapted to the three-dimensional posture features in terms of structure and statistical characteristics.

[0076] S204: Transform and map the three-dimensional posture features to obtain three-dimensional mapped posture features, and determine a second mean and a second standard deviation corresponding to each joint point in the three-dimensional mapped posture features. Each row of the three-dimensional mapped posture features corresponds to a joint point, and each second number of columns corresponds to a rotation angle, where the second number is the ratio of the initial number of channels to the number of types of rotation angles.

[0077] It can be understood that the three-dimensional posture feature is the feature data that describes the human body posture in three-dimensional space, and includes information such as the rotation angle of the human body joints. Transformation and mapping are operations performed on the three-dimensional posture feature, the purpose of which is to convert it into a form suitable for fusion with the two-dimensional posture feature. The result is the three-dimensional mapping posture feature. The second quantity is calculated based on the initial number of channels and the number of types of rotation angles, and is used to determine how many columns in the three-dimensional mapping posture feature correspond to a rotation angle. The second mean and the second standard deviation are the mean and standard deviation of the data corresponding to each joint point in the three-dimensional mapping posture feature, respectively, and are used for subsequent feature fusion operations, reflecting the central tendency and discreteness of the rotation angle data of each joint point.

[0078] The principle of this step is to pre-process the three-dimensional posture features so that they match the sub-feature map after the two-dimensional posture features are divided in terms of structure, which is convenient for subsequent fusion operations. Through transformation and mapping, the three-dimensional posture features are converted into a specific matrix form, that is, each row corresponds to a joint point, and every second number of columns corresponds to a rotation angle. Such a structure enables the rotation angle information of each joint point to clearly correspond to the sub-feature map in the two-dimensional posture feature. The second mean and second standard deviation corresponding to each joint point are calculated in order to standardize the three-dimensional mapped posture features in the subsequent fusion process, so that they are more consistent in distribution with the sub-feature map of the two-dimensional posture features, thereby achieving more effective feature fusion. In the entire video generation process, this step is one of the key links in feature fusion, providing appropriate three-dimensional posture feature representation and statistical information for subsequent steps.

[0079] Figure 3 The lower part shows the operation of this step. The 3D posture feature is initially in the form of a first vector with 72 data points. After transformation and mapping, it is converted into a second vector form with 24 rows (each row corresponds to a joint point), and each c / 3 column (assuming there are 3 rotation angles) corresponds to a rotation angle. Then, the second mean corresponding to each joint point (vector) is calculated. μ (vector) and the second standard deviation σ(vector). Through such processing, the three-dimensional posture feature has the structure and statistical parameters that are fused with the two-dimensional posture feature sub-feature map.

[0080] S206: For any sub-feature map, obtain a fused sub-feature map according to the corresponding first mean, first standard deviation, second mean, and second standard deviation.

[0081] It can be understood that the fused sub-feature graph is the result of fusing the sub-feature graph of the two-dimensional posture feature with the feature graph obtained after processing the three-dimensional posture feature. It integrates the information of the two-dimensional and three-dimensional posture features for the subsequent construction of the fused posture feature. This step adjusts the sub-feature graph of the two-dimensional posture feature based on statistical information so that it can be better integrated with the three-dimensional posture feature. Through the first mean, the first standard deviation, the second mean and the second standard deviation, each sub-feature graph can be standardized and adjusted to remove the distribution influence of the original features in the sub-feature graph, and then integrate the information of the three-dimensional posture feature into it. The purpose of this is to make the fused sub-feature graph retain the information about the position of the joint points in the two-dimensional posture feature, and integrate the information about the rotation angle of the joint points in the three-dimensional posture feature, thereby realizing the complementary advantages of the two-dimensional and three-dimensional posture features.

[0082] S208: Obtain fused posture features according to each fused sub-feature graph.

[0083] As can be understood, the fused pose feature is the final feature obtained by combining all fused sub-feature maps. It integrates information from 2D and 3D pose features and is used in subsequent video generation models to provide more accurate and comprehensive motion guidance information. The fused sub-feature maps are then integrated to form a complete fused pose feature. Since each fused sub-feature map corresponds to a joint, combining them together can comprehensively describe the human body's pose information, including the joint's position and rotation angle. This fused pose feature can provide the video generation model with richer and more accurate motion guidance, making the character movements in the generated video more natural and reasonable. This step is the final step in feature fusion in the entire video generation process, integrating the local fusion results obtained in the previous steps into a comprehensive feature, providing key input for subsequent video generation. Specifically, all sub-feature maps are summed and averaged.

[0084] In one embodiment, obtaining a fused sub-feature map according to the corresponding first mean, first standard deviation, second mean, and second standard deviation includes:

[0085] The intermediate sub-feature map is obtained according to the first expression. The first expression is:

[0086]

[0087] in, is the intermediate sub-feature map, is the i-th sub-feature map, is the first mean, is the first standard deviation, is the second mean, The second standard deviation is the intermediate sub-feature map. The intermediate sub-feature map is an intermediate product in the fusion process of the 2D pose feature sub-feature map and the 3D pose feature. It is obtained by processing the i-th sub-feature map using the first expression. It initially integrates the statistical information of the 3D pose feature, preparing for the final fused sub-feature map. In the first expression, the mean of each data point in the i-th sub-feature map is subtracted to shift the center of the data to around 0, and then divided by its standard deviation to scale the data distribution to a standard deviation of 1. This completes the normalization of the 2D pose feature sub-feature map, removing the influence of the original data distribution, making different sub-feature maps comparable, and facilitating subsequent fusion with the 3D pose feature. At this point, multiplying by the second standard deviation of the 3D pose feature scales the normalized data based on the degree of dispersion of the 3D pose feature. Because the range of variation of the 3D pose feature may vary in different actions or scenarios, multiplying by the second standard deviation can align the data range of the 2D sub-feature map with the 3D pose feature. Finally, add the second mean corresponding to the i-th joint point of the three-dimensional posture feature. This step is to move the center of the processed data to the mean position of the i-th joint point in the three-dimensional posture feature. Through this series of operations, the statistical information of the three-dimensional posture feature (the distribution characteristics represented by the mean and standard deviation) is integrated into the standardized two-dimensional sub-feature map, and finally the intermediate sub-feature map is obtained. The intermediate sub-feature map obtained in this way not only retains some basic information of the two-dimensional posture feature, but also has the distribution characteristics of the three-dimensional posture feature, laying the foundation for further weighted summation with the original two-dimensional sub-feature map to generate a fused sub-feature map. After obtaining the intermediate sub-feature map, perform weighted summation on it and the corresponding sub-feature map to obtain a fused sub-feature map. That is and The information of the two is combined by weighted addition according to the set ratio, and finally the fused sub-feature map is obtained.

[0088] In one embodiment, see Figure 4 The target background acquisition process includes step S402 and step S404.

[0089] S402: Perform human body detection on the original background to obtain a human body region mask.

[0090] It can be understood that the original background refers to the complete image information containing scenes and characters initially obtained in the motion video generation process. Human body detection is a technology in the field of computer vision that uses algorithms and models to identify and locate human bodies in images. The human body region mask is a special image representation that has the same size as the image cropped from the human body detection result. It marks the area where the human body is located in the original background in a binary form (usually represented by 0 and 1), where the part with a value of 1 corresponds to the human body area and the part with a value of 0 corresponds to the non-human area. Figure 5 As shown in the figure, the leftmost set of images shows multiple frames of the original background image, which includes a dancer. The human detection phase uses algorithms to identify and locate the human body within the original background based on features such as shape and texture. The dancer's body region is determined from the original background image containing the dancer. Next, a human matting process is used to generate a human region mask. Furthermore, because each person's appearance is unique, the mask sequence must be processed to prevent the mask generated by the matting process from inheriting features from the original background sequence. First, the mask is closed to smooth the edges and remove any features of the original background. The closed mask is then dilated to ensure that the generated human body has sufficient space to fit within the specified scene. The original background can be obtained from an action-driven video, allowing the target human image to move within the same background as the action-driven video, or it can be any desired background. Furthermore, background images or videos without a human can be directly input into the model as the target background.

[0091] S404: Obtain the target background according to the human body region mask and the original background.

[0092] The target background is an image that has been processed to remove the human figure from the original background. This image serves as the background for the characters in the subsequent motion video. Specifically, the original background image can be manipulated using the human figure region mask as an index. Specifically, based on the human figure regions marked in the mask, the corresponding human figure regions in the original background image are removed, replaced, or otherwise processed, resulting in a target background that excludes the original human figure. This step in the video generation process provides a suitable background for the target figure, avoiding any conflict between the original background and the target figure, making the resulting motion video more natural and consistent with expectations. As shown in the figure, through the "scaling and positioning" operation, using the mask as a guide, the pixels corresponding to the human figure (dancer) in the original background image are removed or processed, resulting in the "masked background" (target background), which is the dance studio background image without the dancer. This step creates a clean, suitable background for the subsequent integration of the target figure into the motion video, ensuring a proper balance between the background and the figure, and improving the quality and effectiveness of the generated video.

[0093] In one embodiment, see Figure 6 The video generation model includes an encoder (vae encoder), a feature extraction module (dinov2), a UNET network (DownSampling + UpSampling), and a decoder (vae decoder). The encoder input is the target person image, which is connected to the downsampling layer of the UNET network through an adder. The input of the feature extraction module is the target person image, and the output of the feature extraction module is connected to the downsampling layer and upsampling layer of the UNET network respectively. The output of the upsampling layer of the UNET network is connected to the decoder. Please refer to Figure 7 , input the target person image, target background and fusion posture features into the video generation model to obtain the target motion video, including steps S702 to S708.

[0094] S702: Input the fused posture features into the downsampling layer of the UNET network.

[0095] S704: Input the target person image into the encoder.

[0096] S706: Add potential noise encoding to the target background and sum it with the output of the encoder. The summation result is input to the downsampling layer of the UNET network.

[0097] S708: Input the target person's image into a feature extraction module. The results of the feature extraction module are input into the downsampling and upsampling layers of the UNET network. The output of the upsampling layer of the UNET network is connected to a decoder, which is used to generate the target motion video based on the output of the upsampling layer of the UNET network.

[0098] As can be understood, the downsampling layers of the UNET network undertake the key task of feature extraction and abstraction from the input data. Through convolution and pooling operations, the spatial dimension of the data is gradually reduced while the number of feature channels is increased, thereby acquiring high-level features from the data. The input data to the downsampling layers includes fused pose features, the target person image fused with the target background, and the target person image after additional feature extraction. Pose guidance is crucial for generating high-quality human action videos. The fused pose features integrate 2D and 3D human motion information, accurately depicting the state of human motion at different moments. Inputting the fused pose features into the downsampling layers of the UNET network allows the model to play a guiding role in denoising from the outset in the latent space constructed by the UNET. Fusion of the target person image and the target background is achieved through superposition. Before superposition, the target background is encoded with latent noise, which the model then eliminates through diffusion in the UNET space. The target background is also a sequence of video frames with the same number of frames as the driving video sequence, while the target person image can be a single image. The target person image is encoded using an encoder to obtain its representation in the latent space. The latent features of the single target person image are then replicated along the temporal dimension to align with the features of each video frame. The copied latent video frames corresponding to the target person image and the target background are concatenated along the channel dimension and then fed into a U-Net for diffusion. Furthermore, after feature extraction by the feature extraction module, the target person image is fed into the cross-attention of the downsampling and upsampling layers of the U-Net network to control the output. The result of diffusion denoising in the U-Net network is then fed into the decoder for decoding, yielding the target motion video. The encoder and decoder here can be variational autoencoders and variational autodecoders, respectively. The feature extraction module can be based on the DiNOv2 architecture.

[0099] In one embodiment, see Figure 8 , the training process of the video generation model includes steps S802 to S808.

[0100] S802, obtaining training 3D posture features according to the training action driving video.

[0101] It can be understood that the training action-driven video, serving as the fundamental data for training the video generation model, is a video sequence containing a series of human motion and posture variations. It records the human body's movements at different moments, providing rich information for the model to learn human motion patterns. The training 3D posture features are feature information extracted from the training action-driven video that describes the human body's posture in 3D space. Common diffusion model frameworks that drive the human body based on posture information use a loss function that uses pixel-level differences at the latent coding layer as the loss, and then updates the network model parameters through backpropagation. The inference process of the diffusion model is a step-by-step iterative denoising process, and the training mechanism involves randomly performing a denoising step. It has been observed that decoding the latent space features derived from certain steps can produce images with human figures, allowing the determination of the human body's posture in the image. Therefore, this embodiment primarily constructs a loss function based on the difference between the 3D posture predicted during training and the 3D data in the actual training data. This is more intuitive and effective than calculating the loss at the latent coding layer.

[0102] S804: Obtain predicted three-dimensional posture features based on the predicted motion video obtained by the video generation model for the training action-driven video.

[0103] As you can understand, the training motion-driven video is a video sequence generated by the video generation model based on the training motion-driven video. It represents the model's simulation and reproduction of the motion in the input video. The quality and accuracy of the predicted motion video reflects the performance of the video generation model. The predicted 3D pose feature is feature information extracted from the predicted motion video that describes the human body's pose in 3D space. Similar to the training 3D pose feature, it is also obtained through analysis and processing of the predicted motion video and is used to evaluate the difference between the model-generated motion and the real motion.

[0104] S806: Obtain a target loss value based on the predicted 3D posture features and the trained 3D posture features.

[0105] It can be understood that the target loss value is a value obtained by comparing the predicted 3D pose features with the trained 3D pose features. It is used to measure the degree of difference between the movements generated by the video generation model and the real movements. The smaller the target loss value, the closer the movements generated by the model are to the real movements, and the better the model performance. Specifically, the target loss value can be obtained based on the second expression, the predicted 3D pose features, and the trained 3D pose features. The second expression is:

[0106]

[0107] in, is the target loss value, n is the number of joint points in the 3D posture feature, is the training 3D posture feature of the i-th joint point, is the predicted 3D pose feature for the i-th joint. In this expression, the first part of the loss term is the sum of the absolute differences between the trained and predicted 3D pose features for each joint, divided by the number of joints, measuring the average absolute magnitude of the overall difference. The second part of the loss term is the sum of the squares of the differences between the trained and predicted 3D pose features for each joint, divided by the number of joints to emphasize the impact of larger differences. The two parts are added together to obtain the final target loss value.

[0108] S808, adjusting the video generation model according to the target loss value.

[0109] It can be understood that the principle of this step is to use the backpropagation algorithm and the optimizer to adjust the parameters of the video generation model according to the target loss value. In deep learning, the model training process is the process of continuously adjusting the model parameters to reduce the loss function value. After the target loss value is calculated, the gradient of the loss function relative to the model parameters is calculated by the backpropagation algorithm. These gradients represent the rate of change of each parameter to the loss function. Then, the optimizer updates the parameters of the model based on these gradients and parameters such as the preset learning rate. By repeating this process, the parameters of the model will gradually adjust to the state that minimizes the loss function value, thereby improving the performance of the model. In some embodiments, the UNET network part is mainly trained, and the encoder and decoder parts can use frozen pre-trained parameters.

[0110] The present application provides a motion video generation device, which includes a data acquisition module, a posture feature extraction module, a posture feature fusion module and a video generation module.

[0111] The data acquisition module is used to obtain the target person image, target background, and motion-driven video. The posture feature extraction module is used to extract the corresponding 2D and 3D posture features for each frame of the motion-driven video. The posture feature fusion module is used to fuse the 2D and 3D posture features to generate fused posture features. The video generation module is used to input the target person image, target background, and fused posture features into the video generation model to generate the target motion video.

[0112] For the specific limitations of the motion video generation device, please refer to the limitations of the motion video generation method above, which will not be repeated here. The various modules in the above-mentioned motion video generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0113] The present application provides a computer device, including one or more processors and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the steps of the motion video generation method in any of the above embodiments are performed.

[0114] Schematically, as Figure 9 As shown, Figure 9 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. Figure 9 Computer device 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by memory 901 for storing instructions executable by processing component 902, such as an application. The application stored in memory 901 may include one or more modules, each corresponding to a set of instructions. In addition, processing component 902 is configured to execute the instructions to perform the steps of the motion video generation method of any of the above-mentioned embodiments.

[0115] The computer device 900 may further include a power supply component 903 configured to perform power management of the computer device 900 , a wired or wireless module interface 904 configured to connect the computer device 900 to the module, and an input / output (I / O) interface 905 .

[0116] The present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the motion video generation method in any of the above embodiments.

[0117] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0118] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0119] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A motion video generation method, characterized in that: include: Obtain target person image, target background and action-driven video; Extracting two-dimensional posture features and three-dimensional posture features corresponding to each frame of the action-driven video; The three-dimensional posture feature includes rotation angles corresponding to a first number of joint points; fusing the two-dimensional posture feature and the three-dimensional posture feature to obtain a fused posture feature; The fusing of the two-dimensional posture feature and the three-dimensional posture feature to obtain a fused posture feature includes: expanding the initial number of channels of the two-dimensional posture feature to a multiple of the first number, dividing the two-dimensional posture feature into sub-feature graphs corresponding to each of the joint points in the channel dimension, and determining a first mean and a first standard deviation corresponding to each channel of each sub-feature graph; transforming and mapping the three-dimensional posture feature to obtain a three-dimensional mapping posture feature, and determining a second mean and a second standard deviation corresponding to each of the joint points in the three-dimensional mapping posture feature; each row of the three-dimensional mapping posture feature corresponds to one of the joint points, and each second number of columns corresponds to one of the rotation angles, the second number being the ratio between the initial number of channels and the number of types of the rotation angles; for any of the sub-feature graphs, obtaining a fused sub-feature graph according to the corresponding first mean, first standard deviation, second mean and second standard deviation; and obtaining the fused posture feature according to each of the fused sub-feature graphs; The target person image, the target background and the fusion posture features are input into a video generation model to obtain a target motion video.

2. The motion video generation method according to claim 1, wherein: The obtaining a fused sub-feature map according to the corresponding first mean, the first standard deviation, the second mean, and the second standard deviation includes: The intermediate sub-feature map is obtained according to the first expression; the first expression is: ； in, is the intermediate sub-feature map, is the i-th sub-feature map, is the first mean, is the first standard deviation, is the second mean, is the second standard deviation; The intermediate sub-feature map and the corresponding sub-feature map are weightedly summed to obtain the fused sub-feature map.

3. The motion video generation method according to claim 1, wherein: The target background acquisition process includes: Perform human body detection on the original background to obtain the human body area mask; The target background is obtained according to the human body region mask and the original background.

4. The motion video generation method according to claim 3, wherein: Before obtaining the target background according to the human body region mask and the original background, the method further includes: The human body region mask is sequentially subjected to a closing operation and an expansion operation.

5. The motion video generation method according to claim 1, characterized in that: The video generation model includes an encoder, a feature extraction module, a UNET network and a decoder. The target person image, the target background and the fusion posture feature are input into the video generation model to obtain the target motion video, including: Inputting the fused posture features into the downsampling layer of the UNET network; Inputting the target person image into the encoder; Adding potential noise coding to the target background and summing it with the output result of the encoder; the summation result is input into the downsampling layer of the UNET network; The target person image is input into the feature extraction module; the result of the feature extraction module is input into the downsampling layer and upsampling layer of the UNET network, the output of the upsampling layer of the UNET network is connected to the decoder, and the decoder is used to generate the target motion video according to the output of the upsampling layer of the UNET network.

6. The motion video generation method according to claim 5, characterized in that: The training process of the video generation model includes: The training 3D posture features are obtained by driving the video based on the training action; Obtaining predicted three-dimensional posture features based on the predicted motion video obtained by the video generation model for the training action-driven video; Obtaining a target loss value based on the predicted three-dimensional posture feature and the training three-dimensional posture feature; The video generation model is adjusted according to the target loss value.

7. The motion video generation method according to claim 6, characterized in that: Obtaining a target loss value according to the predicted three-dimensional posture feature and the training three-dimensional posture feature includes: The target loss value is obtained according to a second expression, the predicted three-dimensional posture feature, and the training three-dimensional posture feature; the second expression is: ； in, is the target loss value, n is the number of joint points in the three-dimensional posture feature, is the training 3D posture feature of the i-th joint point, is the predicted three-dimensional posture feature of the i-th joint point.

8. A motion video generating device, characterized in that: include: Data acquisition module, used to obtain the target person image, target background and action-driven video; a posture feature extraction module, configured to extract two-dimensional posture features and three-dimensional posture features corresponding to each frame of the action-driven video; the three-dimensional posture features including rotation angles corresponding to a first number of joint points; A posture feature fusion module is used to fuse the two-dimensional posture feature and the three-dimensional posture feature to obtain a fused posture feature; the fusing of the two-dimensional posture feature and the three-dimensional posture feature to obtain the fused posture feature includes: expanding the initial number of channels of the two-dimensional posture feature to a multiple of the first number, and dividing the two-dimensional posture feature into sub-feature graphs corresponding to each of the joint points in the channel dimension, and determining the first mean and first standard deviation corresponding to each channel of each sub-feature graph; transforming and mapping the three-dimensional posture feature to obtain a three-dimensional mapping posture feature, and determining the second mean and second standard deviation corresponding to each of the joint points in the three-dimensional mapping posture feature; each row of the three-dimensional mapping posture feature corresponds to one joint point, and each second number of columns corresponds to one rotation angle, and the second number is the ratio between the initial number of channels and the number of types of the rotation angles; for any of the sub-feature graphs, obtaining a fused sub-feature graph according to the corresponding first mean, first standard deviation, second mean and second standard deviation; obtaining the fused posture feature according to each of the fused sub-feature graphs; The video generation module is used to input the target person image, the target background and the fusion posture features into a video generation model to obtain a target motion video.

9. A computer device, characterized in that: The system comprises one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the motion video generation method according to any one of claims 1 to 7 are executed.

10. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the motion video generation method according to any one of claims 1 to 7.

Images