Method and device for generating hand gesture video based on fine-grained semantic description

Abstract

The present invention discloses a method and device for generating a gesture action video based on fine-grained semantic description, comprising: constructing sample data consisting of a video frame and a fine-grained semantic description text for describing gesture actions in the video frame after preprocessing and data enhancement of the original video; constructing a training framework consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module; constructing a loss function based on the difference between the added noise in the video encoding module and the denoised noise in the noise reduction generation module, using the loss function to optimize only the noise reduction generation module parameters, and forming a video generation model consisting of the text encoding module, the noise reduction generation module, and the video decoding module after parameter optimization; and using the video generation model to generate gesture action videos based on an input random vector and a fine-grained semantic description text describing the gesture action. In this way, a high-quality general digital human video containing gesture actions is generated based on the fine-grained semantic description text.

Description

Technical Field

[0001] The present invention belongs to the technical field of text action generation of computer vision, and in particular relates to a method and device for generating a gesture action video based on fine-grained semantic description. Background Art

[0002] With the rapid development of artificial intelligence technology, Text2Motion generation methods have gradually become a hot research area. Such tasks are to generate human motion sequences from given text descriptions, in which the model explores various mappings from natural language instructions to human motion. These methods not only bring new possibilities to content creation by converting text descriptions into human motion sequences, but also provide innovative solutions for entertainment, education and other fields.

[0003] Among the many text-driven motion generation methods, MotionDiffuse proposed the first framework for text-driven motion generation based on a diffusion model, verifying that text-driven motion generation has excellent motion manipulation capabilities.

[0004] FineMotionDiffuse has achieved good results in generating spatial or temporal composite actions by learning the implicit mapping from fine-grained descriptions to corresponding basic actions. These research works have demonstrated the feasibility of the task of generating gesture actions from fine-grained semantic descriptions.

[0005] Action-GPT proposed by Kalakonda et al. explored the effectiveness of the motion sequence generation scheme from coarse-grained text descriptions to different body parts by expanding the original coarse-grained text using GPT3.

[0006] The open source software TM2T facilitates the mapping between the two modes of motion and text by adapting the neural model of machine translation (NMT) to the context, achieving the generation of variable-length non-deterministic gesture sequences from input text.

[0007] The gesture generation method based on fine-grained semantic description belongs to the text generation method, which aims to use the generative model to generate high-fidelity digital human videos that are consistent with the hand movements in the fine-grained text semantic description. Most of the existing work tasks focus on coarse-grained action generation, such as walking, running, jumping, etc., and cannot focus on detailed action generation control.

[0008] However, the Wensheng digital human task is still in the initial exploration stage. Due to the high complexity of hand motion data and the lack of high-quality data sets of specific character gestures, the generated digital humans need to be improved in terms of authenticity and motion continuity. Motion controllability refers to whether the generated digital human's body movements and expressions can be controlled in a fine-grained manner. This requires improving the model's ability to understand driving information and adjust video structure. Summary of the invention

[0009] In view of the above, the purpose of the present invention is to provide a method and device for generating gesture action videos based on fine-grained semantic description, aiming to generate high-quality general digital human videos containing gesture actions based on fine-grained semantic description text.

[0010] To achieve the above-mentioned object of the invention, an embodiment provides a method for generating a gesture action video based on fine-grained semantic description, comprising the following steps:

[0011] After preprocessing and data enhancement of the original video, sample data consisting of video frames and fine-grained semantic description texts used to describe the gesture actions in the video frames are constructed;

[0012] Construct a training architecture consisting of a video encoding module, a text encoding module, a denoising generation module, and a video decoding module, wherein the video encoding module is used to encode, denoise, and re-encode video frames in the sample data to obtain latent vectors of the latent space, the text encoding module is used to embed the fine-grained semantic description text in the sample data to obtain a text vector as a condition, the denoising generation module is used to denoise the latent vector under the guidance of the text vector to obtain a denoised vector, and the video decoding module is used to decode the denoised vector to obtain a video frame with gesture actions and compose a gesture action video;

[0013] A loss function is constructed based on the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module. The loss function is used to optimize only the parameters of the denoising generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the denoising generation module, and the video decoding module.

[0014] The video generation model is used to generate gesture action videos based on the input random vector and the fine-grained semantic description text describing the gesture action.

[0015] The improved technical concept of the gesture video generation module based on fine-grained semantic description of the present invention is a phased innovative method, focusing on the task of generating gesture videos from text semantic descriptions. This technology first ensures the consistency of hand movements through accurate extraction of facial and body key points and data smoothing in the data preprocessing stage, and then generates human body key point representation images based on facial and body key points to provide standardized input data for the model. Most importantly, in the fine-tuning stage, this model is carefully fine-tuned on the public national common sign language dictionary dataset, using the gesture data of specific people to learn the cross-modal information between their gesture semantic descriptions and sign language action videos, and further optimize the loss function to adapt to individual differences. The fine-tuning at this stage significantly improves the performance of the model in sign language video generation, enhances hand clarity, and improves the model's ability to retain image details.

[0016] Preferably, the original video is preprocessed and data enhanced, including:

[0017] The original video is cleaned and segmented to obtain video clips. At this time, each video clip corresponds to a set of gesture action description texts, which include single action description texts and compound action description texts;

[0018] For single action description text, the corresponding video clip is disassembled into video frames, and the corresponding single action description text is matched for each video frame as fine-grained semantic description text;

[0019] For compound action description texts, the compound action description texts are decomposed into independent simple action description texts according to the action sequence. At the same time, the video clips are also decomposed into video frames according to the action sequence, and the corresponding simple action description texts are matched to the video frames as fine-grained semantic description texts.

[0020] For all fine-grained semantic description texts, a large language model is also used for text enhancement.

[0021] Preferably, the video encoding module includes a CausalVAE encoder, a denoising process, and a VIT model. The video frame is judged to be encoded by the CausalVAE encoder to obtain a visual block, the visual block is then denoised for multiple time steps through the denoising process to obtain a noisy visual block, and the noisy visual block is then encoded by the VIT model to obtain a latent vector of the latent space.

[0022] Preferably, the text encoding module adopts the text encoder in the mT5 translation model.

[0023] Preferably, the denoising generation module adopts a DiT diffusion model, which includes a Transformer encoder, linear decoding and shape adjustment operations. The input latent vector is denoised in the Transformer encoder based on the context guidance of the text vector through multiple time steps, and denoised noise is generated based on the context guidance of the text vector at each time step. The denoised noise is subtracted from the input vector to achieve one-step denoising. The denoised result is used as the input vector of the next time step. The denoised vector after denoising at all time steps is then subjected to linear decoding and shape adjustment operations to obtain the final denoised vector.

[0024] Preferably, the video decoding module adopts the CausalVAE decoder.

[0025] Preferably, the method further comprises extracting facial key points, body key points, and hand key points from the original video, and rendering the extracted key points in a dense neural rendering representation manner to obtain a CCBR image representing semantic information of the face and upper body and an eye gaze image representing eye features, and splicing the two images on the channel as a video frame image;

[0026] During rendering, enhanced rendering is performed by changing the camera position and camera viewing angle to obtain data-enhanced video frame images, and the video frame images are used as part of the sample data to train the training architecture.

[0027] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a gesture action video generation device based on fine-grained semantic description, comprising:

[0028] A sample data construction unit, which is used to construct sample data consisting of video frames and fine-grained semantic description texts for describing gesture actions in the video frames after preprocessing and data enhancement of the original video;

[0029] A training architecture construction unit, which is used to construct a training architecture consisting of a video encoding module, a text encoding module, a denoising generation module, and a video decoding module, wherein the video encoding module is used to encode, denoise, and re-encode video frames in the sample data to obtain a latent vector of the latent space, the text encoding module is used to perform text embedding on the fine-grained semantic description text in the sample data to obtain a text vector as a condition, the denoising generation module is used to denoise the latent vector under the guidance of the text vector to obtain a denoised vector, and the video decoding module is used to decode the denoised vector to obtain a video frame with gesture actions, and compose a gesture action video;

[0030] A training unit, which is used to construct a loss function based on the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module, and use the loss function to optimize only the parameters of the denoising generation module. After the parameters are optimized, a video generation model is composed of a text encoding module, a denoising generation module, and a video decoding module;

[0031] The video generation unit is used to generate a gesture action video based on an input random vector and a fine-grained semantic description text describing the gesture action by using a video generation model.

[0032] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned gesture action video generation method based on fine-grained semantic description.

[0033] To achieve the above-mentioned purpose of the invention, the embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned method for generating a gesture action video based on fine-grained semantic description is implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] By preprocessing and data enhancement of the original video, fine-grained semantic description text is constructed. On this basis, video generation is achieved through a training architecture consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module, combined with a noise addition and denoising diffusion process. The resulting video generation model can efficiently generate high-quality, highly synchronized, highly coherent, and highly detailed gesture action videos based on the fine-grained semantic description text to meet the needs of the digital human field. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a flow chart of a method for generating a gesture action video based on fine-grained semantic description provided by an embodiment;

[0038] Figure 2 is a large language model data enhancement graph provided by an embodiment;

[0039] Figure 3is a facial and body key point definition diagram provided by an embodiment;

[0040] Figure 4 is a definition diagram of eye and hand key points provided by an embodiment;

[0041] Figure 5 This is a three-dimensional key point rendering representation diagram provided by the embodiment;

[0042] Figure 6 is a schematic diagram of the structure of the training architecture provided in the embodiment;

[0043] Figure 7 It is a structural schematic diagram of a gesture action video generation device based on fine-grained semantic description provided in an embodiment. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] The inventive concept of the present invention is: to solve the technical problems existing in the prior art, the present invention provides a gesture action video generation method and device based on fine-grained semantic description, focusing on the digital human gesture action generation task in the half-body area, based on the proposed facial and upper body key point extraction and representation method, it can obtain the annotation data and sample data of the facial and body key points from the monocular video data, and realize the text-driven generation of gesture action video paradigm based on the constructed video generation model, and ensure the quality and timing continuity of the generated gesture action video.

[0046] like Figure 1 As shown, the embodiment provides a method for generating a gesture action video based on fine-grained semantic description, comprising the following steps:

[0047] S1, after preprocessing and data enhancement of the original video, constructs sample data consisting of video frames and fine-grained semantic description texts used to describe the gesture actions in the video frames.

[0048] In the embodiment, when constructing a video generation model, data preprocessing and data enhancement are crucial in model training, directly affecting the performance of the model and the final generation effect. The present invention improves the quality of training data through innovative data processing methods, and improves the generalization of the model through data enhancement. Data preprocessing is implemented through a series of precise steps to ensure the quality of input data and the effectiveness of subsequent model training. Specifically, the original video is preprocessed and data enhanced, including:

[0049] The original video is cleaned and segmented to obtain video clips. At this time, each video clip corresponds to a set of gesture action description texts, which include single action description texts and compound action description texts. The purpose of data cleaning and segmentation is to perform quality detection on the original video data to reduce the impact of distortion and noise on the training results of digital humanities generated videos. First, the original video is divided into multiple video clips according to the image and action characteristics of the characters. The length of each video is controlled to be about 5 to 10 seconds to ensure that the image and action of the characters in each video are relatively stable. Then, the video clips are checked frame by frame, and video frames with quality problems, such as flickering, black shadows, and blur, are deleted. At the same time, video frames with incomplete side views and hand gestures are deleted, such as videos shot at side angles and situations where characters leave the screen halfway.

[0050] For single action description text, the corresponding video clip is disassembled into video frames, and the corresponding single action description text is matched for each video frame as fine-grained semantic description text.

[0051] For the compound action description text, the compound action description text is decomposed into independent simple action description texts according to the order of actions. At the same time, the video clips are also decomposed into video frames according to the order of actions, and the corresponding simple action description texts are matched to the video frames as fine-grained semantic description texts. Compound action decomposition aims to decompose the action descriptions in the original video data according to the order of execution and the video actions corresponding to each clip, so as to obtain a more fine-grained correspondence between text and video action descriptions. In order to reduce the workload of compound action decomposition, this paper first adopts a batch processing method to evenly divide the video length according to the number of segments in the text description of the compound action, and then uses manual verification to adjust the start and end frames of each clip to ensure accurate correspondence with the text content.

[0052] For example, for a compound action description text, first a brief description of the action in the sign language dictionary is performed, and then a natural language description of the action in the sign language dictionary is performed, that is, data enhancement is performed, as shown in Table 1:

[0053]

[0054] The embodiment also performs data enhancement of fine-grained semantic description text, wherein text data enhancement methods generally include synonym replacement, random insertion, and back translation, etc. These methods have relatively limited effects in increasing the diversity of data sets and improving the generalization ability of models. To solve this problem, the present invention uses a large language model to reprocess the original fine-grained description text to generate a new action retelling that is consistent with the original description, such as Figure 2This process aims to enrich the dataset, increase the diversity of action descriptions, and significantly improve the model's ability to understand different expressions. Compared with traditional text data augmentation methods, using a large language model can not only generate more natural and semantically consistent paraphrases, but also better capture the subtle differences in action descriptions, thereby improving the model's ability to cope with diverse inputs. The advantage of this method is that it can create more diverse samples while retaining the original information, allowing the model to be exposed to a wider range of expressions during training, ultimately improving its generalization performance and practicality.

[0055] After text data enhancement, sample data consisting of video frames and fine-grained semantic description texts used to describe gesture actions in video frames can be constructed. The video generation model trained with these sample data can generate sign language action videos of specific characters.

[0056] In order to generate a video composed of a CCBR (Common Human Action Representation Method) image sequence in the embodiment, facial key points, body key points, and hand key points are also extracted from the original video when constructing the sample data. Specifically, when extracting key points, MediaPipe (an efficient facial key point detection model) and DWPose (an efficient full body posture estimation model) are used to extract the facial, body and hand key points of the characters in the video. Among them, MediaPipe is used to extract 478 facial key points, where facial key points include lips, eyes, nose and other key points. DWPose is used to extract 23 body key points and 21 hand key points for each hand, specifically including palms, fingertips, elbow joints, skeletons and other key points, defined as follows Figure 3 and Figure 4 As shown, the extracted facial and body key points are respectively smoothed to ensure the consistency of their movements and expressions between frames.

[0057] For data processing, the input is a sequence of images, and then each current frame is processed separately, and the processing of the current frame is further divided into processing the face and torso separately. For the face, the eye key points are first marked using facial key point detection, and the facial key points are extracted from the video. These points cover the basic contours of the face, and the eye image is obtained from them. According to Figure 4 The 20 eye key points marked by the yellow circle are used to obtain the pupil center position and pupil radius. A red circle with the pupil center as the origin and the pupil radius as the radius is drawn, and the eye edge points are connected and filled with white to generate an eye image. Then, the face is reconstructed using advanced methods, and the vertices and coefficients are saved. Then, the face is rendered based on the coefficients to obtain the facial image and head projection matrix. For the upper torso, advanced body key point recognition technology is used to extract 23 key points of the half torso from the video, such as Figure 3As shown in Figure 2, these points cover the specific postures of the body and hands. Since accurate reconstruction of the hands is required, additional 3D detection of the hand key points can be performed and the sequence of hand key points can be recorded.

[0058] The extracted key points are also rendered using a dense neural rendering representation to obtain CCBR images representing the semantic information of the face and upper body and eye gaze images representing the eye features, such as Figure 5 As shown, the two images are spliced ​​on the channel as a video frame image; during rendering, enhanced rendering is performed by changing the camera position and camera viewing angle to obtain a data-enhanced video frame image, and the video frame image is used as part of the sample data to train the training architecture.

[0059] S2, builds a training architecture consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module.

[0060] In the embodiment, a training framework is constructed to construct a video generation model for video generation, specifically, as Figure 6 As shown, the training architecture includes a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module. Among them, the video encoding module is used to encode, add noise and re-encode the video frames in the sample data to obtain the latent vector of the latent space, the text encoding module is used to embed the fine-grained semantic description text in the sample data to obtain the text vector as a condition, the noise reduction generation module is used to denoise the latent vector under the guidance of the text vector to obtain the denoised vector, and the video decoding module is used to decode the denoised vector to obtain the video frame with gesture action and form the gesture action video. This training architecture has the characteristics of temporal enhancement.

[0061] In an embodiment, a video encoding module includes a CausalVAE (causal autoencoder) encoder, a denoising process, and a VIT (visual Transformer) model. The video frame is judged to be encoded by the CausalVAE encoder to obtain a visual block, the visual block is then denoised for multiple time steps by the denoising process to obtain a noisy visual block, and the noisy visual block is then encoded by the VIT model to obtain a latent vector of the latent space.

[0062] Among them, the CausalVAE encoder maps the original video frame into the latent space, aiming to reduce the dimensionality of the original video frame and output a latent representation that has been compressed in time and space. The CausalVAE encoder mainly uses a causal convolutional network to ensure that when processing time series data, the encoder only relies on past information and avoids interference from future information. This structure enables the CausalVAE encoder to effectively capture and learn causal relationships in the data and enhance the temporal consistency when generating gesture action video inference.

[0063] During the denoising process, random noise is continuously added to the visual block in the latent space at each time step to obtain a noisy visual block. The VIT model is mainly used for visual encoding. The noisy visual block is further encoded in the latent space and the visual block is adjusted from a three-dimensional size to a one-dimensional size to adapt to the model size of the subsequent denoising generation module.

[0064] In the embodiment, the text encoding module has a high degree of flexibility and adaptability, and is mainly used to perform text embedding encoding on the fine-grained semantic description text describing the gesture action to obtain a text vector as a guiding condition for subsequent noise generation, so as to achieve effective interaction and reaction in a variety of application scenarios. In the embodiment, the text encoding module is developed and designed based on the mT5 translation model, and the text encoder part in the mT5 translation model is specifically selected to process the text input. mT5 is pre-trained on a wide range of mC4 corpora, and supports the understanding and generation of 101 languages ​​including Chinese, English, French, Arabic, etc., so it has a powerful cross-language text processing capability. This feature enables the action-driven text encoding module to process multilingual input, meet the needs of global users, and perform well in international application scenarios.

[0065] The text encoder part of the mT5 translation model uses the Transformer architecture to process the input text and convert it into a contextual embedding representation. The encoder is composed of multiple identical layers stacked together, each of which contains a self-attention mechanism and a feedforward neural network. This structure is specifically designed to extract deep semantic information about facial expressions and hand movements from fine-grained semantic description text. The text encoder can effectively associate action patterns with language information, thereby improving the accuracy and consistency of semantic understanding.

[0066] In the embodiment, the denoising generation module adopts the DiT diffusion model (Diffusion Transformer, a diffusion model based on Transformer), which includes a Transformer encoder, linear decoding and shape adjustment operations. The input latent vector is denoised in the Transformer encoder based on the context guidance of the text vector through multiple time steps, and denoised noise is generated based on the context guidance of the text vector at each time step. The denoised noise is subtracted from the input vector to achieve one-step denoising. The denoised result is used as the input vector of the next time step. The denoised vector after denoising at all time steps is then subjected to linear decoding and shape adjustment operations to obtain the final denoised vector.

[0067] The DiT diffusion model shows significant advantages in maintaining temporal consistency, which is mainly due to its architectural design and the synergy of key components. First, DiT uses contextual conditions in the Transformer encoder to strengthen the model's understanding of temporal information. This context-aware mechanism enables DiT to effectively capture and maintain dynamic changes in time series and avoid inconsistent frames during the generation process. Secondly, the introduction of cross-attention blocks in the Transformer encoder enables the model to have stronger correlation when processing time series data. This interaction not only enhances the consistency of local details, but also ensures the coherence of the global structure, effectively preventing jumps or breaks in the time dimension. Finally, the design of the adaptive layer normalization block in the Transformer encoder provides DiT with a flexible normalization strategy, enabling it to adaptively adjust according to the temporal characteristics of the input. This flexibility allows the model to dynamically optimize the generation process at different time points, thereby improving the stability and consistency of the generated results.

[0068] The DiT diffusion model's Transformer encoder significantly improves temporal consistency through the effective combination of contextual conditions, cross-attention blocks, and adaptive layer normalization blocks, ensuring the natural fluency and visual coherence of the generated content in the temporal dimension. This makes the DiT diffusion model have greater application potential in tasks such as dynamic scene generation and video synthesis.

[0069] In the embodiment, the video decoding module adopts the CausalVAE decoder. The CausalVAE and DiT diffusion model adopted by the training framework constructed in this way and the diffusion process can improve the consistency of action timing in the text generation sign language action task.

[0070] S3, constructs a loss function based on the difference between the noise added in the video encoding module and the denoised noise in the noise reduction generation module, and uses the loss function to optimize only the parameters of the noise reduction generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the noise reduction generation module, and the video decoding module.

[0071] In the embodiment, in order to save training time and improve training efficiency, the local fine-tuning strategy adopted by the training framework, that is, the CausalVAE encoder, VIT model, text encoding module, and CausalVAE decoder in the video encoding module of the pre-training process are frozen, and only the DIT diffusion model in the denoising generation module in the training framework is optimized. The loss function used in training is the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module to construct the loss function, and the back propagation and optimization algorithm (such as Adam) are used to update the network parameters, and the learning rate may be adjusted to adapt to more sophisticated learning needs. Through the small batch gradient descent method, the model parameters are continuously updated to minimize the loss function and improve the repair effect. In addition, fine-tuning also includes regularization techniques, such as R1 regularization, to prevent overfitting and ensure the generalization ability of the model on new data. Performance evaluation is analyzed by quantitative indicators and qualitative analysis, and hyperparameter adjustment ensures the efficiency and stability of model training. In this way, the denoising generation module after training can accurately denoise the input random noise to generate high-quality video frames.

[0072] During the training process, the fine-grained semantic description text in the sample data is used as the guiding information to drive the generation of gesture actions, and the video frame sequence is input into the video encoding module to generate the potential noise in the latent space. The denoising generation module and the video decoding module generate gesture action videos based on the guiding information and potential noise through denoising and decoding prediction. The connection between natural language and body movement language is established during training, and the corresponding gesture actions are accurately restored from the action description. After the training, the parameter-optimized denoising generation module and the parameter-unchanged text encoding module and video decoding module are extracted to form a video generation model.

[0073] S4, uses the video generation model to generate gesture action videos based on the input random vector and the fine-grained semantic description text describing the gesture action.

[0074] In an embodiment, when a gesture action video is generated, the fine-grained semantic description text guiding the generation of the gesture action is input into a text encoding module for encoding to obtain a text vector, and the text vector is input into a noise reduction generation module. At the same time, a random noise vector is input into the noise reduction generation module. After denoising under the guidance of the text vector, the text vector is decoded by a video decoding module to generate video frames and form a gesture action video.

[0075] The above-mentioned gesture action video generation method based on fine-grained semantic description is able to generate high-definition details and focus on the gesture actions represented by common characters. Through advanced data preprocessing and enhancement technology and superior model network architecture design, it significantly improves the clarity and realism of character action gesture generation.

[0076] The above-mentioned gesture action video generation method based on fine-grained semantic description has synchronization and coherence. While enhancing the clarity of gestures, it maintains the consistency of the character image in the video generation, ensures the naturalness and smoothness of the animation, and improves the user experience.

[0077] The above-mentioned gesture action video generation method based on fine-grained semantic description has strong generalization ability. Through text enhancement, this network model can adapt to different action description expression inputs, enhances the generalization ability of the model, and enables it to work stably in various scenarios.

[0078] The above-mentioned gesture action video generation method based on fine-grained semantic description has high computational efficiency. The model design focuses on computational efficiency. By freezing most module parameters and fine-tuning the training strategy and parallel processing technology of a small number of modules, it achieves faster image training and inference speed than other life video tasks, saving a lot of time.

[0079] The above-mentioned gesture action video generation method based on fine-grained semantic description also supports personalized customization. By not extracting key points from the training data, but directly using the gesture action video of a specific person as a training sample, it is possible to generate gesture actions for a specific person and meet the specific needs of users.

[0080] The above-mentioned gesture action video generation method based on fine-grained semantic description is highly robust. Through specific data preprocessing and enhancement techniques, this model has high robustness to the noise and variation of the input data, and can stably generate high-quality results on data of different qualities.

[0081] like Figure 7As shown, the embodiment also provides a gesture action video generation device 80 based on fine-grained semantic description, including: a sample data construction unit 81, a training architecture construction unit 82, a training unit 83, and a video generation unit 84, wherein the sample data construction unit 81 is used to construct sample data consisting of video frames and fine-grained semantic description texts for describing gesture actions in video frames after preprocessing and data enhancement of the original video; the training architecture construction unit 82 is used to construct a training architecture consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module, wherein the video encoding module is used to encode, add noise and re-encode the video frames in the sample data to obtain latent vectors of the latent space, and the text encoding module is used to The fine-grained semantic description text is embedded in the text to obtain a text vector as a condition. The denoising generation module is used to denoise the latent vector under the guidance of the text vector to obtain a denoised vector. The video decoding module is used to decode the denoised vector to obtain a video frame with gesture actions, and compose a gesture action video; the training unit 83 is used to construct a loss function based on the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module, and use the loss function to optimize only the parameters of the denoising generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the denoising generation module, and the video decoding module; the video generation unit 84 is used to use the video generation model to generate a gesture action video based on the input random vector and the fine-grained semantic description text describing the gesture action.

[0082] It should be noted that the gesture action video generation device based on fine-grained semantic description provided in the above embodiment should be illustrated by the division of the above-mentioned functional modules when performing gesture action video generation. The above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the gesture action video generation device based on fine-grained semantic description provided in the above embodiment belongs to the same concept as the gesture action video generation method embodiment based on fine-grained semantic description. The specific implementation process is detailed in the gesture action video generation method embodiment based on fine-grained semantic description, which will not be repeated here.

[0083] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for generating a gesture action video based on fine-grained semantic description is implemented, specifically including the following steps:

[0084] S1, after preprocessing and data enhancement of the original video, constructs sample data consisting of video frames and fine-grained semantic description texts used to describe the gesture actions in the video frames;

[0085] S2, constructs a training architecture consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module;

[0086] S3, constructing a loss function based on the difference between the noise added in the video encoding module and the denoised noise in the denoising generation module, and using the loss function to optimize only the parameters of the denoising generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the denoising generation module, and the video decoding module;

[0087] S4, uses the video generation model to generate gesture action videos based on the input random vector and the fine-grained semantic description text describing the gesture action.

[0088] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the gesture action video generation method based on fine-grained semantic description described in S1-S4 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0089] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for generating a gesture action video based on fine-grained semantic description is implemented, which specifically includes the following steps:

[0090] S1, after preprocessing and data enhancement of the original video, constructs sample data consisting of video frames and fine-grained semantic description texts used to describe the gesture actions in the video frames;

[0091] S2, constructs a training architecture consisting of a video encoding module, a text encoding module, a noise reduction generation module, and a video decoding module;

[0092] S3, constructing a loss function based on the difference between the noise added in the video encoding module and the denoised noise in the denoising generation module, and using the loss function to optimize only the parameters of the denoising generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the denoising generation module, and the video decoding module;

[0093] S4, uses the video generation model to generate gesture action videos based on the input random vector and the fine-grained semantic description text describing the gesture action.

[0094] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0095] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating gesture action videos based on fine-grained semantic description, characterized in that: The following steps are involved: After preprocessing and data enhancement of the original video, sample data consisting of video frames and fine-grained semantic description texts for describing gesture actions in the video frames are constructed, including: cleaning and segmenting the original video to obtain video clips, where each video clip corresponds to a set of gesture action description texts, which include single action description texts and compound action description texts; for single action description texts, the corresponding video clips are disassembled into video frames, and the corresponding single action description texts are matched for each video frame as fine-grained semantic description texts; for compound action description texts, the compound action description texts are disassembled into independent simple action description texts according to the action sequence, and the video clips are also disassembled into video frames according to the action sequence, and the corresponding simple action description texts are matched for the video frames as fine-grained semantic description texts; for all fine-grained semantic description texts, a large language model is also used for text enhancement; Construct a training architecture consisting of a video encoding module, a text encoding module, a denoising generation module, and a video decoding module, wherein the video encoding module is used to encode, denoise, and re-encode video frames in the sample data to obtain latent vectors of the latent space, the text encoding module is used to embed the fine-grained semantic description text in the sample data to obtain a text vector as a condition, the denoising generation module is used to denoise the latent vector under the guidance of the text vector to obtain a denoised vector, and the video decoding module is used to decode the denoised vector to obtain a video frame with gesture actions and compose a gesture action video; A loss function is constructed based on the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module. The loss function is used to optimize only the parameters of the denoising generation module. After the parameters are optimized, the video generation model is composed of the text encoding module, the denoising generation module, and the video decoding module. The video generation model is used to generate gesture action videos based on the input random vector and the fine-grained semantic description text describing the gesture action.

2. The method for generating gesture action videos based on fine-grained semantic description according to claim 1, characterized in that: The video encoding module includes a CausalVAE encoder, a denoising process, and a VIT model. The video frame is encoded by the CausalVAE encoder to obtain a visual block, the visual block is then denoised in multiple time steps by the denoising process to obtain a noisy visual block, and the noisy visual block is then encoded by the VIT model to obtain a latent vector of the latent space.

3. The method for generating gesture action videos based on fine-grained semantic description according to claim 1, characterized in that: The text encoding module adopts the text encoder in the mT5 translation model.

4. The method for generating gesture action videos based on fine-grained semantic description according to claim 1, characterized in that: The denoising generation module adopts the DiT diffusion model, which includes a Transformer encoder, linear decoding and shape adjustment operations. The input potential vector is denoised in the Transformer encoder based on the context guidance of the text vector through multiple time steps. De-noising noise is generated based on the context guidance of the text vector at each time step. The denoising noise is subtracted from the input vector to achieve one-step denoising. The denoised result is used as the input vector of the next time step. The denoised vector after denoising at all time steps is subjected to linear decoding and shape adjustment operations to obtain the final denoised vector.

5. The method for generating gesture action videos based on fine-grained semantic description according to claim 1, characterized in that: The video decoding module adopts the CausalVAE decoder.

6. The method for generating gesture action videos based on fine-grained semantic description according to claim 1, characterized in that: It also includes extracting facial key points, body key points, and hand key points from the original video, and rendering the extracted key points in a dense neural rendering representation manner to obtain a CCBR image representing semantic information of the face and upper body and an eye gaze image representing eye features, and splicing the two images on the channel as a video frame image; During rendering, enhanced rendering is performed by changing the camera position and camera viewing angle to obtain data-enhanced video frame images, and the video frame images are used as part of the sample data to train the training architecture.

7. A gesture action video generation device based on fine-grained semantic description, characterized in that: include: A sample data construction unit is used to construct sample data consisting of video frames and fine-grained semantic description texts for describing gesture actions in video frames after preprocessing and data enhancement of the original video, including: cleaning and segmenting the original video to obtain video segments, where each video segment corresponds to a set of gesture action description texts, which include single action description texts and compound action description texts; for single action description texts, the corresponding video segments are disassembled into video frames, and the corresponding single action description texts are matched for each video frame as fine-grained semantic description texts; for compound action description texts, the compound action description texts are disassembled into independent simple action description texts according to the action sequence, and the video segments are also disassembled into video frames according to the action sequence, and the corresponding simple action description texts are matched for the video frames as fine-grained semantic description texts; for all fine-grained semantic description texts, a large language model is also used for text enhancement; A training architecture construction unit, which is used to construct a training architecture consisting of a video encoding module, a text encoding module, a denoising generation module, and a video decoding module, wherein the video encoding module is used to encode, denoise, and re-encode video frames in the sample data to obtain a latent vector of the latent space, the text encoding module is used to perform text embedding on the fine-grained semantic description text in the sample data to obtain a text vector as a condition, the denoising generation module is used to denoise the latent vector under the guidance of the text vector to obtain a denoised vector, and the video decoding module is used to decode the denoised vector to obtain a video frame with gesture actions, and compose a gesture action video; A training unit, which is used to construct a loss function based on the difference between the added noise in the video encoding module and the denoised noise in the denoising generation module, and use the loss function to optimize only the parameters of the denoising generation module. After the parameters are optimized, a video generation model is composed of a text encoding module, a denoising generation module, and a video decoding module; The video generation unit is used to generate a gesture action video based on an input random vector and a fine-grained semantic description text describing the gesture action by using a video generation model.

8. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the method for generating gesture action videos based on fine-grained semantic description according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for generating a gesture action video based on fine-grained semantic description described in any one of claims 1 to 6 is implemented.

Images