Action generation method and device, model training method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses an action generation method and device, a model training method and device, equipment, a storage medium and a product. The action generation method comprises the steps of obtaining a to-be-processed text, performing M times of iterative prediction on the to-be-processed text through a key action generation module in an action generation model to obtain a key action feature sequence corresponding to the to-be-processed text and time sequence features corresponding to key action features in the key action feature sequence, and combining the mask features with the key action feature sequence according to the time sequence features to obtain a target feature sequence carrying the mask features, and performing action reconstruction on the target feature sequence through a reconstruction module in an action generation model to obtain a target action sequence corresponding to the to-be-processed text. Therefore, the key action features are predicted through the to-be-processed text, and then the target action sequence is directly predicted through the key action features, so that the action prediction time can be shortened (all action features do not need to be predicted), and the action sequence is generated more quickly and efficiently.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an action generation method, a model training method, an action generation device, a model training device, a computer device, a computer-readable storage medium, and an action generation product and a model training product. Background Technology

[0002] With advancements in scientific research, technologies related to artificial intelligence models are developing rapidly. More and more AI models are being applied in various fields of life; for example, generative models are used in video generation, and discriminative models are used in facial recognition. In video generation, action generation models can generate action sequences based on the text to be processed. How to generate action sequences more quickly and efficiently based on existing methods has become a hot research topic. Summary of the Invention

[0003] This application provides an action generation method, a model training method, an apparatus, a device, a computer-readable storage medium, and a product that can generate action sequences more quickly and efficiently.

[0004] On one hand, embodiments of this application provide an action generation method, including:

[0005] Get the text to be processed;

[0006] The key action generation module in the action generation model performs M iterations of prediction on the text to be processed, and obtains the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. The temporal features are used to indicate the arrangement position of the key action features in the target feature sequence, where M is an integer greater than 1.

[0007] The mask features are combined with the key action feature sequence based on the temporal features to obtain the target feature sequence carrying the mask features. The mask features are filled in the target feature sequence at positions other than those indicated by the temporal features.

[0008] The target feature sequence is reconstructed by the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed; the target action sequence is used to generate the action video of the target object.

[0009] In this embodiment, the text to be processed is obtained, and the key action generation module in the action generation model performs M iterations of prediction on the text to be processed to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. Based on the temporal features, the mask features are combined with the key action feature sequence to obtain the target feature sequence carrying the mask features. The reconstruction module in the action generation model performs action reconstruction on the target feature sequence to obtain the target action sequence corresponding to the text to be processed. It can be seen that predicting the key action features from the text to be processed first, and then directly predicting the target action sequence (i.e., the complete action sequence containing actions other than key actions) from the key action features can reduce the action prediction time (no need to predict all action features), thereby generating the action sequence more quickly and efficiently.

[0010] On the one hand, embodiments of this application provide a model training method, including:

[0011] Obtain the sample text and the corresponding sequence of sample actions;

[0012] Adaptive masking is performed on the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence;

[0013] The first training module performs M iterations of prediction on the sample text to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence; the temporal feature corresponding to any predicted action feature is used to indicate the arrangement position of the predicted action feature in the sample action sequence, where M is an integer greater than 1.

[0014] Based on the differences between the predicted action feature sequence and the key action feature sequence, as well as the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, the first training module is trained to obtain the action generation module; the temporal feature corresponding to any key action feature is used to indicate the arrangement position of the key action feature in the sample action sequence.

[0015] The mask features are combined with the key action feature sequence based on the temporal features corresponding to each key action feature to obtain the target feature sequence carrying the mask features. The mask features are filled in the target feature sequence at positions other than those indicated by the temporal features corresponding to the key action features.

[0016] The second training module reconstructs the target feature sequence to obtain the predicted action sequence corresponding to the sample text.

[0017] Based on the difference between the predicted action sequence and the sample action sequence, the second training module is trained to obtain the reconstruction module; the action generation module and the reconstruction module are included in the action generation model.

[0018] In this embodiment, sample text and corresponding sample action sequences are obtained. Adaptive masking is performed on the sample action sequences to obtain key action feature sequences. The sample text is iteratively predicted M times by a first training module to obtain predicted action feature sequences and temporal features corresponding to each predicted action feature in the predicted action feature sequences. Based on the differences between the predicted action feature sequences and the key action feature sequences, and the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, the first training module is trained to obtain an action generation module. The mask features are combined with the key action feature sequences according to the temporal features corresponding to each key action feature to obtain a target feature sequence carrying the mask features. The target feature sequence is reconstructed using a second training module to obtain the predicted action sequence corresponding to the sample text. Based on the differences between the predicted action sequence and the sample action sequence, the second training module is trained to obtain a reconstruction module. The action generation module and the reconstruction module are included in the action generation model. It is evident that by adaptively masking the sample action sequences and then training the action generation model in stages based on the key action feature sequences obtained after masking, the action generation model can gain a deeper understanding of the key actions corresponding to the text information, thereby improving the accuracy and stability of the action generation model.

[0019] On one hand, embodiments of this application provide an action generation device, which includes:

[0020] The acquisition unit is used to acquire the text to be processed.

[0021] The processing unit is used to perform M iterations of prediction on the text to be processed through the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. The temporal features are used to indicate the arrangement position of the key action features in the target feature sequence, where M is an integer greater than 1.

[0022] And for combining mask features with key action feature sequences based on temporal features to obtain a target feature sequence carrying mask features, with the mask features filling positions in the target feature sequence other than those indicated by the temporal features;

[0023] And it is used to reconstruct the target feature sequence through the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed; the target action sequence is used to generate the action video of the target object.

[0024] In one implementation, the key action generation module includes an action network and a temporal network; the process by which the processing unit performs M iterations of prediction on the text to be processed through the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence includes:

[0025] The action network is used to predict the action of the i-th input data to obtain the vector representation of the i-th key action, where i is a positive integer less than M.

[0026] The vector representation of the i-th key action is processed by feature mapping to obtain the features of the i-th key action;

[0027] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal feature corresponding to the i-th key action feature; the fused data is obtained by fusing the vector representation of the i-th key action and the i-th key action feature.

[0028] The i-th key action feature and the corresponding temporal feature are used as the input data for the (i+1)-th iteration of the action network.

[0029] The first input data for the action network is the embedding information, which is obtained by vector mapping of the text to be processed.

[0030] In one implementation, the processing unit is configured to perform temporal prediction on the fused data using a temporal network to obtain the temporal feature corresponding to the i-th key action feature, specifically for:

[0031] Temporal prediction of the fused data is performed by a temporal network to obtain the temporal vector representation corresponding to the i-th key action feature;

[0032] The temporal vector representation corresponding to the i-th key action feature is subjected to feature mapping to obtain the temporal feature corresponding to the i-th key action feature.

[0033] In one implementation, the processing unit is configured to perform feature mapping processing on the vector representation of the i-th key action to obtain the features of the i-th key action, specifically for:

[0034] The vector representation of the i-th key action is linearly mapped through a linear layer to obtain the mapping result of the i-th key action;

[0035] The mapping result of the i-th key action is normalized to obtain the feature of the i-th key action.

[0036] In one implementation, the reconstruction module includes a demasking layer and a decoding layer; the demasking layer carries contextual information of the text to be processed; the processing unit is used to perform action reconstruction on the target feature sequence through the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed, specifically for:

[0037] The masked features in the target feature sequence are demasked by a demasking layer to obtain the demasked target feature sequence; the demasking process is used to reconstruct the masked features.

[0038] A decoding layer is used to decode the demasked target feature sequence to obtain the target action sequence corresponding to the text to be processed.

[0039] In one implementation, the demasking layer includes a self-attention submodule and a residual submodule, the residual submodule including a one-dimensional convolution; the processing unit is used to perform demasking processing on the masked features in the target feature sequence through the demasking layer to obtain the demasked target feature sequence, specifically for:

[0040] Attention masking is applied to the masking features in the target feature sequence to obtain the attention-masked target feature sequence. The attention masking process indicates the masking features in the target feature sequence.

[0041] The self-attention submodule is used to reconstruct the target feature sequence after the attention mask, resulting in the reconstructed target feature sequence.

[0042] The target feature sequence after feature reconstruction is transformed by one-dimensional convolution of the residual submodule to obtain the target feature sequence after masking.

[0043] In one implementation, the target feature sequence includes action features corresponding to N temporal features, where N is an integer greater than 1; the process by which the processing unit combines the mask features with the key action feature sequence based on the temporal features to obtain the target feature sequence carrying the mask features includes:

[0044] If the key action feature sequence does not include the action feature corresponding to the temporal feature j, then the mask feature is used to fill the position corresponding to the temporal feature j in the target feature sequence, where j is a positive integer less than or equal to N;

[0045] If the key action feature sequence includes the action feature corresponding to the temporal feature j, then the action feature corresponding to the temporal feature j is used to fill the position corresponding to the temporal feature j in the target feature sequence.

[0046] In one implementation, each action in the target action sequence is represented by a K-dimensional vector, and each joint of the target object corresponds to at least one dimension of the K-dimensional vector, where K is an integer greater than 1; the processing unit is further configured to:

[0047] By driving the joints of the target object to perform corresponding actions through the target motion sequence, the motion video of the target object is obtained.

[0048] On one hand, embodiments of this application provide a model training apparatus, which includes:

[0049] The acquisition unit is used to acquire sample text and the corresponding sample action sequence;

[0050] The processing unit is used to perform adaptive masking on the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence.

[0051] And used to perform M iterations of prediction on the sample text through the first training module to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence; the temporal feature corresponding to any predicted action feature is used to indicate the arrangement position of the predicted action feature in the sample action sequence, where M is an integer greater than 1;

[0052] And to train the first training module based on the difference between the predicted action feature sequence and the key action feature sequence, and the difference between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action feature, to obtain the action generation module; the temporal feature corresponding to any key action feature is used to indicate the arrangement position of the key action feature in the sample action sequence.

[0053] And to combine the mask features with the key action feature sequence according to the temporal features corresponding to each key action feature to obtain the target feature sequence carrying the mask features, with the mask features filling the target feature sequence in positions other than those indicated by the temporal features corresponding to the key action features;

[0054] And used to reconstruct the action sequence of the target feature sequence through the second training module to obtain the predicted action sequence corresponding to the sample text;

[0055] And a second training module is trained based on the difference between the predicted action sequence and the sample action sequence to obtain the reconstruction module; the action generation module and the reconstruction module are included in the action generation model.

[0056] In one implementation, the processing unit is configured to combine the mask features with the key action feature sequence based on the temporal features corresponding to each key action feature to obtain a target feature sequence carrying the mask features, specifically configured to:

[0057] The key action feature sequence is quantized by using a vocabulary to obtain the quantized key action feature sequence.

[0058] The quantized key action feature sequence is filled with mask features to obtain the target feature sequence carrying the mask features; the time length corresponding to the target feature sequence matches the time length corresponding to the sample action sequence.

[0059] In one implementation, the lexicon includes K vectors, each vector corresponding to a different action, where K is an integer greater than 1; the processing unit is used to quantize the key action feature sequence using the lexicon to obtain a quantized key action feature sequence, specifically for:

[0060] Based on the similarity between each key action feature and K vectors in the key action feature sequence, each key action feature is transformed to obtain a quantitative representation of each key action.

[0061] The quantized representations of each key action are dequantized to obtain the quantized key action feature sequence.

[0062] In one implementation, the first training module includes an action network and a temporal network; the process by which the processing unit performs M iterations of prediction on the sample text through the first training module to obtain a predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence includes:

[0063] The action network is used to predict the action of the i-th input data, and the vector representation of the i-th predicted action is obtained; i is a positive integer less than or equal to M.

[0064] The vector representation of the i-th predicted action is processed by feature mapping to obtain the features of the i-th predicted action;

[0065] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal vector representation corresponding to the i-th predicted action feature; the fused data is obtained by fusing the vector representation of the i-th predicted action and the i-th predicted action feature.

[0066] The temporal vector representation corresponding to the i-th predicted action feature is processed by feature mapping to obtain the temporal feature corresponding to the i-th predicted action feature.

[0067] In one implementation, the first training module includes an action network and a temporal network; the processing unit trains the first training module based on the differences between the predicted action feature sequence and the key action feature sequence, and the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, to obtain the action generation module. The process includes:

[0068] The action network is trained based on the difference between the i-th predicted action feature and the i-th key action feature; i is a positive integer less than or equal to M.

[0069] The temporal network is trained based on the difference between the temporal features corresponding to the i-th predicted action feature and the temporal features corresponding to the i-th key action feature.

[0070] In one implementation, the processing unit is configured to perform adaptive masking processing on the sample action sequence to obtain a key action feature sequence corresponding to the sample action sequence, specifically for:

[0071] The sample action sequence is encoded to obtain the sample action features of each sample action;

[0072] Score prediction is performed on the action features of the samples to obtain the scores corresponding to each action feature of the samples.

[0073] The action features of samples with scores higher than the score threshold are packaged to obtain the key action feature sequence corresponding to the sample action sequence.

[0074] Accordingly, this application provides a computer device comprising:

[0075] Memory, which stores computer programs;

[0076] A processor is used to load computer programs to implement action generation methods or the aforementioned model training methods.

[0077] Accordingly, this application provides a computer-readable storage medium storing a computer program that is adapted to be loaded by a processor and executed by the above-described action generation method, or adapted to be loaded by a processor and executed by the above-described model training method.

[0078] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned action generation method or the aforementioned model training method.

[0079] In the action generation process of this application, key action features are first predicted from the text to be processed, and then the target action sequence (i.e., a complete action sequence containing all actions except the key actions) is directly predicted using the key action features. This reduces action prediction time (as it eliminates the need to predict all action features), thus generating action sequences more quickly and efficiently. During model training, the action generation model is trained in stages by adaptively masking the sample action sequences and using the key action feature sequence obtained after masking. This allows the action generation model to more deeply understand the key actions corresponding to the text information, improving the accuracy and stability of the action generation model. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 An action generation scene diagram provided in an embodiment of this application;

[0082] Figure 2 A flowchart illustrating an action generation method provided in an embodiment of this application;

[0083] Figure 3a A schematic diagram of an iterative prediction process provided in an embodiment of this application;

[0084] Figure 3b A schematic diagram of a human body model provided for an embodiment of this application;

[0085] Figure 4 A flowchart illustrating a model training method provided in this application embodiment;

[0086] Figure 5a A schematic diagram of an adaptive masking process provided in an embodiment of this application;

[0087] Figure 5b This is a schematic diagram illustrating a method for determining verification data provided in an embodiment of this application;

[0088] Figure 5c A schematic diagram of a feature mask provided for an embodiment of this application;

[0089] Figure 5d A diagram illustrating a reconstruction module training architecture is provided in this application embodiment;

[0090] Figure 6 This is a schematic diagram of the structure of an action generation device provided in an embodiment of this application;

[0091] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0092] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0094] This application relates to technologies related to artificial intelligence. The following is a brief introduction to relevant terms and concepts of artificial intelligence:

[0095] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0096] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0097] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0098] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0099] Pre-trained models (PTMs), also known as foundational models or large models, refer to deep neural networks (DNNs) with a large number of parameters. These DNNs are trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of large-parameter DNNs, PTMs extract common features from the data and are then fine-tuned using techniques such as fine-tuning, parameter efficient fine-tuning (PEFT), and prompt-tuning, making them suitable for downstream tasks. Therefore, pre-trained models can achieve ideal results in small-shot or zero-shot scenarios. PTMs can be categorized according to the data modality they process, such as language models, visual models, speech models, and multimodal models. Multimodal models refer to models that establish feature representations for two or more data modalities. Pre-trained models are important tools for outputting AI-generated content (AIGC) and can also serve as a general interface connecting multiple specific task models. In this embodiment, the main focus is on using a multimodal model (Clip) to perform vector mapping on the text to be processed, obtaining embedded information, and using this embedded information as the first input data for the action network.

[0100] Model compression and quantization refer to techniques used to reduce model size and accelerate model inference, thereby lowering the storage and computational costs of the model. Model compression typically includes pruning, low-rank decomposition, and knowledge distillation, while model quantization involves converting floating-point parameters in the model into fixed-point or integer parameters, thus reducing model size and accelerating model inference.

[0101] Based on the aforementioned artificial intelligence-related technologies, this application provides an action generation scheme and a model training scheme, which can improve the quality of action videos. Figure 1 An action generation scene graph provided in this application embodiment, such as Figure 1As shown, the action generation scenario provided in this application includes a terminal device 101 and a server 102. The action generation scheme and model training scheme provided in this application can be executed by either the terminal device 101 or the server 102. When both the action generation scheme and model training scheme provided in this application are executed by the terminal device 101, the action generation scenario may not include the server 102. The terminal device may include, but is not limited to: smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (Mobile Internet Devices, or MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and this application embodiment does not limit this. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and this application embodiment does not limit this.

[0102] It should be noted that, Figure 1 The number of terminal devices and servers is for illustrative purposes only and does not constitute an actual limitation of this application. Terminal device 101 and server 102 can be connected via wired or wireless means, and this application does not impose any restrictions on this.

[0103] The action generation scheme and the model training scheme can be independent of each other or combined with each other; for example, the model training scheme can be executed in server 102, and the trained action generation model can be deployed in terminal device 101, and then the action generation scheme can be executed in terminal device 101; or, for another example, the model training scheme and the action generation scheme can be executed in server 102.

[0104] The general process of the action generation scheme provided in this application is as follows:

[0105] Server 102 acquires the text to be processed, which may be extracted from selected multimodal data (such as images, audio, etc.). Server 102 performs M iterations of prediction on the text to be processed using the key action generation module in the action generation model, obtaining the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence, where M is an integer greater than 1. In one implementation, each iteration of prediction can obtain a key action feature and the temporal features corresponding to that key action feature. The key action generation module includes an action network and a temporal network. The process by which server 102 performs M iterations of prediction on the text to be processed using the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature includes: performing action prediction on the input data of the i-th iteration using the action network to obtain the vector representation of the i-th key action, where i is a positive integer less than M. Then, feature mapping processing is performed on the vector representation of the i-th key action to obtain the i-th key action feature. Furthermore, server 102 fuses the vector representation of the i-th key action and the i-th key action feature to obtain fused data. After obtaining the fused data, a temporal network is used to perform temporal prediction on the fused data to obtain the temporal feature corresponding to the i-th key action feature. Then, the i-th key action feature and the temporal feature corresponding to the i-th key action feature are used as the input data for the (i+1)-th iteration of the action network; wherein, the input data for the first iteration of the action network is the embedding information, which is obtained by vector mapping of the text to be processed using a multimodal model (Clip); the outputs of the action network and the temporal network at the M-th iteration are both terminators.

[0106] After obtaining the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence, server 102 combines the mask features with the key action feature sequence according to the temporal features to obtain a target feature sequence carrying the mask features. The arrangement position of any key action feature in the target feature sequence is determined according to the temporal feature corresponding to that key action feature. Specifically, the mask features are filled in the target feature sequence at positions other than those indicated by the temporal features. In one embodiment, the target feature sequence includes N action features corresponding to the temporal features, where N is an integer greater than 1. The process by which server 102 combines mask features with key action feature sequences based on temporal features to obtain a target feature sequence carrying mask features includes: if the key action feature sequence does not include the action feature corresponding to temporal feature j, server 102 fills the position corresponding to temporal feature j in the target feature sequence with mask features, where j is a positive integer less than or equal to N; correspondingly, if the key action feature sequence includes the action feature corresponding to temporal feature j, server 102 fills the position corresponding to temporal feature j in the target feature sequence with the action feature corresponding to temporal feature j. Further, server 102 performs action reconstruction on the target feature sequence through the reconstruction module in the action generation model to obtain a target action sequence corresponding to the text to be processed; wherein, the target action sequence is used to generate an action video of the target object. In one embodiment, the reconstruction module includes a demasking layer and a decoding layer, and the demasking layer carries contextual information of the text to be processed. Server 102 first performs demasking processing on the mask features in the target feature sequence through the demasking layer to obtain a demasked target feature sequence; wherein, the demasking processing is used to reconstruct the mask features. The masked target feature sequence is then decoded by a decoding layer to obtain the target action sequence corresponding to the text to be processed.

[0107] The general process of the model training scheme provided in this application is as follows:

[0108] Server 102 acquires sample text and corresponding sample action sequences, which can be extracted from sample videos. Server 102 performs adaptive masking on the sample action sequences to obtain key action feature sequences corresponding to the sample action sequences. In one implementation, server 102 encodes the sample action sequences to obtain individual sample action features, and then (through a score prediction network) predicts scores for each sample action based on these features to obtain scores corresponding to each sample action feature. After obtaining the scores for each sample action feature, server 102 packages sample action features with scores higher than a score threshold (e.g., sample action features ranked in the top P positions, where P is an integer greater than 1) to obtain key action feature sequences corresponding to the sample action sequences. In one stage, server 102 trains a first training module using sample text and key action feature sequences to obtain an action generation module. In one implementation, server 102 performs M iterations of prediction on the sample text using a first training module to obtain a predicted action feature sequence and temporal features corresponding to each predicted action feature in the predicted action feature sequence. The temporal feature corresponding to any predicted action feature is used to indicate the position of that predicted action feature in the sample action sequence, where M is an integer greater than 1. After obtaining the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence, server 102 trains the first training module based on the differences between the predicted action feature sequence and the key action feature sequence, as well as the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, to obtain an action generation module. The temporal feature corresponding to any key action feature is used to indicate the position of that key action feature in the sample action sequence. In another stage, server 102 trains a second training module using the key action feature sequence and the sample action sequence to obtain a reconstruction module. In one implementation, server 102 combines mask features with key action feature sequences based on temporal features corresponding to each key action feature to obtain a target feature sequence carrying mask features. The mask features are filled in all positions in the target feature sequence except those indicated by the temporal features corresponding to the key action features, and the time length of the target feature sequence matches the time length of the sample action sequence. After obtaining the target feature sequence carrying mask features, server 102 performs action reconstruction on the target feature sequence using a second training module to obtain a predicted action sequence corresponding to the sample text. Then, based on the difference between the predicted action sequence and the sample action sequence, the second training module is trained to obtain a reconstruction module. The action generation module and the reconstruction module are included in the action generation model.

[0109] In the action generation process of this application, key action features are first predicted from the text to be processed, and then the target action sequence (i.e., a complete action sequence containing all actions except the key actions) is directly predicted using the key action features. This reduces action prediction time (as it eliminates the need to predict all action features), thus generating action sequences more quickly and efficiently. During model training, the action generation model is trained in stages by adaptively masking the sample action sequences and using the key action feature sequence obtained after masking. This allows the action generation model to more deeply understand the key actions corresponding to the text information, improving the accuracy and stability of the action generation model.

[0110] Based on the above action generation scheme and model training scheme, this application proposes a more detailed action generation method and model training method. The action generation method and model training method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0111] Please see Figure 2 , Figure 2 A flowchart illustrating an action generation method provided in this application embodiment, which can be executed by a computer device; for example, by... Figure 1 The terminal device 101 or server 102 shown executes this. For example... Figure 2 As shown, the action generation method may include the following steps S201-S204:

[0112] S201. Obtain the text to be processed.

[0113] The text to be processed can be extracted from selected multimodal data (such as images, audio, etc.), or it can be any input text, or it can be text selected from text data. This application does not impose any restrictions on this.

[0114] S202. The key action generation module in the action generation model performs M iterations of prediction on the text to be processed to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence.

[0115] Each iteration of prediction yields a key action feature and its corresponding temporal feature. The temporal feature for each key action feature indicates its position within the target feature sequence. The target feature sequence is used to reconstruct the target action sequence, which includes both key and non-key actions. Key and non-key actions can be determined based on action scores; for example, actions with scores above a threshold are identified as key actions, while those with scores below the threshold are identified as non-key actions. In the target action sequence, the key action features obtained from M iterations are arranged according to the temporal feature corresponding to each key action feature, not according to the output order of the key action generation module, where M is an integer greater than 1.

[0116] It should be noted that the key action generation module is only used to predict the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. It does not predict the non-key action features in the target action sequence, nor the temporal features corresponding to the non-key action features. Compared to predicting all action features in the target action sequence and the temporal features corresponding to each action feature, this can reduce prediction time and thus improve prediction efficiency.

[0117] In one implementation, the key action generation module includes an action network and a temporal network. The process by which the computer device performs M iterations of prediction on the text to be processed using the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence includes: performing action prediction on the i-th input data through the action network to obtain the vector representation of the i-th key action, where i is a positive integer less than M. Then, feature mapping processing is performed on the vector representation of the i-th key action to obtain the i-th key action feature. In one embodiment, the computer device can first perform linear mapping on the vector representation of the i-th key action through a linear layer to obtain the mapping result of the i-th key action, and then normalize the mapping result of the i-th key action to obtain the i-th key action feature.

[0118] The computer device fuses the vector representation of the i-th key action and the feature of the i-th key action to obtain fused data. After obtaining the fused data, the computer device performs temporal prediction on the fused data through a temporal network to obtain the temporal feature corresponding to the i-th key action feature. In one embodiment, the computer device first performs temporal prediction on the fused data through a temporal network to obtain the temporal vector representation corresponding to the i-th key action feature, and then performs feature mapping processing on the temporal vector representation corresponding to the i-th key action feature to obtain the temporal feature corresponding to the i-th key action feature. The feature mapping process can refer to the process of performing feature mapping processing on the vector representation of the key action, and will not be repeated here.

[0119] Furthermore, when i is less than M, the computer device uses the i-th key action feature and the corresponding temporal feature as the input data for the (i+1)-th iteration of the action network; where the input data for the first iteration of the action network is the embedding information, which is obtained by vector mapping of the text to be processed using a multimodal model (Clip), and the outputs of the action network and the temporal network for the M-th iteration are both end symbols.

[0120] Figure 3a This is a schematic diagram of an iterative prediction process provided in an embodiment of this application. Figure 3a As shown, the process first uses a multimodal model (Clip) to perform vector mapping on the text to obtain embedded information. This embedded information is then used as the first input data to the Motion Transformer to obtain the vector representation of the first key action. Next, a linear layer performs a linear mapping on this vector representation to obtain the mapping result for the first key action. Finally, a softmax function normalizes this mapping result to obtain the first key action feature. Further, the first key action feature and its vector representation are fused (Fuse), and the fused data is input to a temporal Transformer to obtain the temporal vector representation corresponding to the first key action feature. Next, a linear layer is used to linearly map the temporal vector representation corresponding to the first key action feature, obtaining the mapping result of the temporal feature corresponding to the first key action feature. Then, a normalization function (Softmax) is used to normalize the mapping result of the temporal feature corresponding to the first key action feature, obtaining the temporal feature corresponding to the first key action feature. Furthermore, the first key action feature and the temporal feature corresponding to the first key action feature are used as the input data for the second time into the action network. The above steps are repeated until the termination symbol (e.g., ...) is obtained. <eos>), i.e. to obtain the key action feature sequence corresponding to the text to be processed and the time sequence feature corresponding to each key action feature.

[0121] S203, combining the mask feature with the key action feature sequence according to the time sequence feature to obtain a target feature sequence carrying the mask feature.

[0122] The mask feature can be a learnable mask feature (learnable mask token), and the mask feature is used to represent any non-key action feature, i.e. each non-key action feature is represented by the same mask feature. The mask feature is filled in the target feature sequence except the arrangement position indicated by the time sequence feature.

[0123] In an embodiment, the target feature sequence includes N action features corresponding to the time sequence features, and N is an integer greater than 1. For the position corresponding to the time sequence feature j in the target feature sequence, if the key action feature sequence does not include the action feature corresponding to the time sequence feature j (i.e. the action corresponding to the time sequence feature j is a non-key action), the computer device fills the position corresponding to the time sequence feature j in the target feature sequence with the mask feature, and j is a positive integer less than or equal to N. Accordingly, if the key action feature sequence includes the action feature corresponding to the time sequence feature j (i.e. the action corresponding to the time sequence feature j is a key action), the computer device fills the position corresponding to the time sequence feature j in the target feature sequence with the action feature corresponding to the time sequence feature j in the key action feature sequence.

[0124] For example, assuming that the key action feature sequence includes key action feature 1-key action feature 3, and key action feature 1 corresponds to time sequence feature A, key action feature 2 corresponds to time sequence feature B, and key action feature 3 corresponds to time sequence feature C, the position corresponding to time sequence feature A in the target feature sequence is 5, the position corresponding to time sequence feature B in the target feature sequence is 1, and the position corresponding to time sequence feature C in the target feature sequence is 4, then the position 1 in the target feature sequence is filled with key action feature 2, the positions 2 and 3 are filled with mask features, the position 4 is filled with key action feature 3, and the position 5 is filled with key action feature 1.

[0125] S204, performing action reconstruction on the target feature sequence through a reconstruction module in the action generation model to obtain a target action sequence corresponding to the text to be processed.

[0126] The key to action reconstruction is reconstructing masked features into masked (non-critical) action features. In one implementation, the reconstruction module includes a demasking layer and a decoder layer; the demasking layer carries contextual information of the text to be processed; for example, the demasking layer is a context-aware demasking layer. The computer device performs demasking processing on the masked features in the target feature sequence through the demasking layer to obtain a demasked target feature sequence; wherein, the demasking processing is used to reconstruct the masked features. In one embodiment, the demasking layer includes a self-attention submodule and a residual submodule, the residual submodule including a one-dimensional convolution. The computer device performs attention masking processing on the masked features in the target feature sequence to obtain an attention-masked target feature sequence; wherein, the attention masking process indicates the masked features in the target feature sequence. By indicating the masked features, the transmission of masked features to critical action features can be prevented. Attention masking can be simply understood as follows: by instructing and adjusting the attention of mask features and key action features (e.g., reducing the attention of mask features and increasing the attention of key action features), the transmission of mask features to key action features is prevented, thereby further improving reconstruction accuracy. Next, the computer device reconstructs the target feature sequence after attention masking using a self-attention submodule, obtaining the reconstructed target feature sequence. Then, a one-dimensional convolution in the residual submodule performs a dimensionality transformation on the reconstructed target feature sequence, resulting in the demasked target feature sequence. The one-dimensional convolution in the residual submodule is designed for action generation tasks; the convolution dimension can be adaptively adjusted when processing other generation tasks. After obtaining the demasked target feature sequence, the computer device uses a decoding layer to decode the demasked target feature sequence, thus obtaining the target action sequence corresponding to the text to be processed.

[0127] Furthermore, after obtaining the target action sequence corresponding to the text to be processed, the computer device can drive the corresponding target object (such as a virtual character model) through the target action sequence to obtain the action video of the target object. In one implementation, each action in the target action sequence is represented by a K-dimensional vector, and each joint of the target object corresponds to at least one dimension of the K-dimensional vector, where K is an integer greater than 1. The computer device drives each joint of the target object to perform the corresponding action through the target action sequence to obtain the action video of the target object.

[0128] Figure 3b This is a schematic diagram of a human body model provided for an embodiment of this application. Figure 3b As shown, the human body model provided in this application consists of 21 joints, plus one root joint (a total of 22 joints). Each movement is represented as... in, It is a 1-dimensional vector representing the angular velocity of the root node; r y Both are one-dimensional vectors, representing the root node's velocity in the x and z directions and its position in the y direction, respectively; j p It refers to the location of a local joint; j v j represents the velocity of a local joint; r This indicates the rotation of a local joint relative to the root node; c f Whether a foot is on the ground is indicated by a 4-dimensional vector. As can be seen from the above, a single movement of the human body model provided in this application is composed of 263-dimensional vectors.

[0129] It is understood that the specific representation of the target action sequence can be dynamically adjusted based on the driven object (such as adjusting the number of joints, the representation method, etc.), and this application does not impose any restrictions on this.

[0130] In this embodiment, the text to be processed is obtained, and the key action generation module in the action generation model performs M iterations of prediction on the text to be processed to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. Based on the temporal features, the mask features are combined with the key action feature sequence to obtain the target feature sequence carrying the mask features. The reconstruction module in the action generation model performs action reconstruction on the target feature sequence to obtain the target action sequence corresponding to the text to be processed. It can be seen that predicting the key action features from the text to be processed first, and then directly predicting the target action sequence (i.e., the complete action sequence containing actions other than key actions) from the key action features can reduce the action prediction time (no need to predict all action features), thereby generating the action sequence more quickly and efficiently.

[0131] Please see Figure 4 , Figure 4 A flowchart illustrating a model training method provided in this application embodiment, the model training method being executed by a computer device; for example, by... Figure 1 The terminal device 101 or server 102 shown executes this. For example... Figure 4 As shown, the model training method may include the following steps S401-S407:

[0132] S401. Obtain the sample text and the corresponding sample action sequence.

[0133] The sample text and the corresponding sample action sequence can be extracted separately from the same sample video.

[0134] S402. Perform adaptive masking on the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence.

[0135] Adaptive masking refers to masking non-critical action features (action features whose importance score is not higher than the score threshold) based on the importance score of each action feature in the action sequence.

[0136] In one implementation, the computer device encodes the sample action sequence to obtain individual sample action features. After obtaining these features, the computer device can use a scoring network (such as a Multi-Layer Perceptron (MLP)) to predict the score of each sample action feature, thus obtaining a score (i.e., an importance score) for each feature. The computer device then packages the sample action features with scores higher than a threshold to obtain the key action feature sequence corresponding to the sample action sequence. In one embodiment, the computer device multiplies each key action feature by its corresponding score to obtain a weighted feature for each key action, and then performs layer normalization on the weighted features to obtain the key action feature sequence corresponding to the sample action sequence.

[0137] Figure 5a This is a schematic diagram illustrating an adaptive masking process provided in an embodiment of this application. Figure 5a As shown, the computer device uses a scoring network to score the importance of action features (i.e., score prediction), obtaining importance scores for each action feature. The Q action features with the highest importance scores are selected as key action features. Then, each key action feature is multiplied by its corresponding importance score to obtain a weighted feature for each key action. Next, layer normalization is performed on the weighted features of each key action to obtain the key action feature sequence corresponding to the sample action sequence.

[0138] S403. The first training module performs M iterations of prediction on the sample text to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence.

[0139] During the training of the first training module, the sample text serves as the input data, and the key action feature sequence serves as the test data. The temporal feature corresponding to any predicted action feature is used to indicate the position of that predicted action feature in the sample action sequence (or target feature sequence), where M is an integer greater than 1.

[0140] In one implementation, the first training module includes an action network and a temporal network. The process by which a computer device performs M iterations of prediction on sample text using the first training module to obtain a predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence includes:

[0141] On one hand, the computer device performs action prediction on the i-th input data through the action network to obtain the vector representation of the i-th predicted action. Then, it performs feature mapping processing on the vector representation of the i-th predicted action to obtain the feature of the i-th predicted action, where i is a positive integer less than or equal to M.

[0142] On the other hand, the computer device performs temporal prediction on the fused data through a temporal network to obtain the temporal vector representation corresponding to the i-th predicted action feature. Then, it performs feature mapping processing on the temporal vector representation corresponding to the i-th predicted action feature to obtain the temporal feature corresponding to the i-th predicted action feature. Here, the fused data is obtained by fusing the vector representation of the i-th predicted action and the i-th predicted action feature.

[0143] S404. Based on the differences between the predicted action feature sequence and the key action feature sequence, and the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, the first training module is trained to obtain the action generation module.

[0144] In one implementation, the computer device trains the first training module to obtain the action generation module based on the differences between the predicted action feature sequence and the key action feature sequence, as well as the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action feature. The process includes: the computer device trains the action network based on the differences between the i-th predicted action feature and the i-th key action feature; and trains the temporal network based on the differences between the temporal features corresponding to the i-th predicted action feature and the temporal features corresponding to the i-th key action feature (i.e., the training of the temporal network does not affect the action network), where i is a positive integer less than or equal to M.

[0145] In another implementation, the computer device trains the first training module to obtain the action generation module based on the differences between the predicted action feature sequence and the key action feature sequence, as well as the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action feature. The process includes: on the one hand, the computer device trains the action network based on the differences between the i-th predicted action feature and the i-th key action feature; on the other hand, the computer device trains the action network and the temporal network based on the differences between the temporal features corresponding to the i-th predicted action feature and the temporal features corresponding to the i-th key action feature (i.e., the training of the temporal network will affect the action network), where i is a positive integer less than or equal to M.

[0146] Figure 5b This is a schematic diagram illustrating a method for determining verification data provided in an embodiment of this application. Figure 5b As shown, the verification data includes key action features and the corresponding temporal features. The key action feature sequence can be obtained by removing the masked (denoted as M) non-key action features from the sample action feature sequence. The temporal features corresponding to the key action features can be extracted by analyzing the arrangement of each key action in the sample action sequence.

[0147] S405. Based on the temporal features corresponding to each key action feature, combine the mask features with the key action feature sequence to obtain the target feature sequence carrying the mask features.

[0148] Mask features are filled into positions in the target feature sequence other than those indicated by the temporal features corresponding to the key action features. In one implementation, the computer device quantizes the key action feature sequence using a lexicon to obtain a quantized key action feature sequence. In one embodiment, the lexicon includes K vectors, where different vectors correspond to different actions, and K is an integer greater than 1. Based on the similarity between each key action feature in the key action feature sequence and the K vectors, the computer device transforms each key action feature to obtain a quantized representation of each key action feature. Then, the quantized representation of each key action feature is dequantized to obtain the quantized key action feature sequence.

[0149] After obtaining the quantized key action feature sequence, the computer device fills the quantized key action feature sequence with masking features to obtain a target feature sequence carrying the masking features; wherein the time length corresponding to the target feature sequence matches the time length corresponding to the sample action sequence. For details on the filling process, please refer to [link / reference]. Figure 2 The implementation method of step S203 will not be described in detail here.

[0150] S406. The target feature sequence is reconstructed through the second training module to obtain the predicted action sequence corresponding to the sample text.

[0151] Figure 5c This is a schematic diagram of a feature mask provided for an embodiment of this application. For example... Figure 5c As shown, the grayscale boxes in the third and last columns, labeled "M," represent the mask features. First, attention masking is applied to the mask features in the target feature sequence to obtain the attention-masked target feature sequence. Then, a self-attention submodule is used to reconstruct the features of the attention-masked target feature sequence (calculating the attention value of each token relative to other tokens). Figure 5c (The "4", "1", "0", and "M" in the text are all tokens), and the target feature sequence after feature reconstruction can be obtained.

[0152] For a detailed implementation of step S406, please refer to Figure 2 The implementation method of step S204 will not be described in detail here.

[0153] S407. Based on the difference between the predicted action sequence and the sample action sequence, the second training module is trained to obtain the reconstruction module.

[0154] The action generation module and the reconstruction module are contained in the action generation model.

[0155] Figure 5d This is a diagram illustrating a reconstruction module training architecture provided in an embodiment of this application. Figure 5d As shown, the sample action sequence is first encoded to obtain the sample action feature sequence. Next, adaptive masking is applied to the sample action feature sequence to obtain the importance score corresponding to each sample action feature. Based on the importance scores of each sample action feature, sample action features with scores higher than a threshold are retained to obtain the key action feature sequence. The threshold can be a fixed value or a dynamic value (e.g., sorting sample actions by importance score from high to low, using the importance score of the H-th sample action as the threshold). Next, the key action feature sequence is quantized using a vocabulary (including quantization and dequantization processes) to obtain the quantized key action feature sequence. Further, the quantized key action feature sequence is filled with learnable mask features to obtain the target feature sequence carrying the mask features. Then, the target feature sequence carrying the mask features is demasked using a demasking layer to obtain the demasked target feature sequence; the demasking layer includes a self-attention submodule and a residual submodule. Finally, the demasked target feature sequence is decoded by a decoder to obtain the reconstructed action sequence corresponding to the sample action sequence.

[0156] Optionally, the first and second training modules can be jointly trained using sample text and sample action sequences (e.g., the parameters in the first and second training modules can be adjusted based on the difference between the reconstructed action sequence output by the second training module and the sample action sequence so that the difference between the reconstructed action sequence and the sample action sequence is less than the difference threshold) to obtain the action generation model.

[0157] Understandable, Figure 4 The model training methods in [the text] can be compared with [the text] Figure 2 The action generation methods in the code are independent of each other and can also be used in... Figure 2 The action generation method is executed before the action generation method in the code.

[0158] In this embodiment, sample text and corresponding sample action sequences are obtained. Adaptive masking is performed on the sample action sequences to obtain key action feature sequences. The sample text is iteratively predicted M times by a first training module to obtain predicted action feature sequences and temporal features corresponding to each predicted action feature in the predicted action feature sequences. Based on the differences between the predicted action feature sequences and the key action feature sequences, and the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, the first training module is trained to obtain an action generation module. The mask features are combined with the key action feature sequences according to the temporal features corresponding to each key action feature to obtain a target feature sequence carrying the mask features. The target feature sequence is reconstructed using a second training module to obtain the predicted action sequence corresponding to the sample text. Based on the differences between the predicted action sequence and the sample action sequence, the second training module is trained to obtain a reconstruction module. The action generation module and the reconstruction module are included in the action generation model. It is evident that by adaptively masking the sample action sequences and then training the action generation model in stages based on the key action feature sequences obtained after masking, the action generation model can gain a deeper understanding of the key actions corresponding to the text information, thereby improving the accuracy and stability of the action generation model.

[0159] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0160] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an action generation device provided in an embodiment of this application. Figure 6 The motion generation device shown can be mounted in a computer device, which can be a terminal device or a server. Figure 6 The motion generation device shown can be used to perform the above. Figure 2 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 6 The motion generating device includes:

[0161] Acquisition unit 601 is used to acquire the text to be processed;

[0162] The processing unit 602 is used to perform M iterations of prediction on the text to be processed through the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. The temporal features are used to indicate the arrangement position of the key action features in the target feature sequence, where M is an integer greater than 1.

[0163] And for combining mask features with key action feature sequences based on temporal features to obtain a target feature sequence carrying mask features, with the mask features filling positions in the target feature sequence other than those indicated by the temporal features;

[0164] And it is used to reconstruct the target feature sequence through the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed; the target action sequence is used to generate the action video of the target object.

[0165] In one implementation, the key action generation module includes an action network and a temporal network; the processing unit 602 performs M iterations of prediction on the text to be processed through the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence, including:

[0166] The action network is used to predict the action of the i-th input data to obtain the vector representation of the i-th key action, where i is a positive integer less than M.

[0167] The vector representation of the i-th key action is processed by feature mapping to obtain the features of the i-th key action;

[0168] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal feature corresponding to the i-th key action feature; the fused data is obtained by fusing the vector representation of the i-th key action and the i-th key action feature.

[0169] The i-th key action feature and the corresponding temporal feature are used as the input data for the (i+1)-th iteration of the action network.

[0170] The first input data for the action network is the embedding information, which is obtained by vector mapping of the text to be processed.

[0171] In one implementation, the processing unit 602 is configured to perform temporal prediction on the fused data through a temporal network to obtain the temporal feature corresponding to the i-th key action feature, specifically configured to:

[0172] Temporal prediction of the fused data is performed by a temporal network to obtain the temporal vector representation corresponding to the i-th key action feature;

[0173] The temporal vector representation corresponding to the i-th key action feature is subjected to feature mapping to obtain the temporal feature corresponding to the i-th key action feature.

[0174] In one implementation, the processing unit 602 is configured to perform feature mapping processing on the vector representation of the i-th key action to obtain the features of the i-th key action, specifically for:

[0175] The vector representation of the i-th key action is linearly mapped through a linear layer to obtain the mapping result of the i-th key action;

[0176] The mapping result of the i-th key action is normalized to obtain the feature of the i-th key action.

[0177] In one implementation, the reconstruction module includes a demasking layer and a decoding layer; the demasking layer carries contextual information of the text to be processed; the processing unit 602 is used to perform action reconstruction on the target feature sequence through the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed, specifically for:

[0178] The masked features in the target feature sequence are demasked by a demasking layer to obtain the demasked target feature sequence; the demasking process is used to reconstruct the masked features.

[0179] A decoding layer is used to decode the demasked target feature sequence to obtain the target action sequence corresponding to the text to be processed.

[0180] In one implementation, the demasking layer includes a self-attention submodule and a residual submodule, the residual submodule including a one-dimensional convolution; the processing unit 602 is used to perform demasking processing on the masked features in the target feature sequence through the demasking layer to obtain the demasked target feature sequence, specifically for:

[0181] Attention masking is applied to the masking features in the target feature sequence to obtain the attention-masked target feature sequence. The attention masking process indicates the masking features in the target feature sequence.

[0182] The self-attention submodule is used to reconstruct the target feature sequence after the attention mask, resulting in the reconstructed target feature sequence.

[0183] The target feature sequence after feature reconstruction is transformed by one-dimensional convolution of the residual submodule to obtain the target feature sequence after masking.

[0184] In one implementation, the target feature sequence includes action features corresponding to N temporal features, where N is an integer greater than 1; the process by which the processing unit 602 combines the mask features with the key action feature sequence according to the temporal features to obtain the target feature sequence carrying the mask features includes:

[0185] If the key action feature sequence does not include the action feature corresponding to the temporal feature j, then the mask feature is used to fill the position corresponding to the temporal feature j in the target feature sequence, where j is a positive integer less than or equal to N;

[0186] If the key action feature sequence includes the action feature corresponding to the temporal feature j, then the action feature corresponding to the temporal feature j is used to fill the position corresponding to the temporal feature j in the target feature sequence.

[0187] In one implementation, each action in the target action sequence is represented by a K-dimensional vector, and each joint of the target object corresponds to at least one dimension of the K-dimensional vector, where K is an integer greater than 1; the processing unit 602 is further configured to:

[0188] By driving the joints of the target object to perform corresponding actions through the target motion sequence, the motion video of the target object is obtained.

[0189] According to one embodiment of this application, Figure 2 The steps involved in the action generation method shown can be derived from... Figure 6 The actions are executed by the individual units in the shown motion generation device. For example, Figure 2 Step S201 shown can be performed by Figure 6 The acquisition unit 601 shown is executed, and steps S202-S204 can be performed by... Figure 6 The processing unit 602 shown executes. Figure 6 The various units in the illustrated motion generation device can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the motion generation device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0190] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 2 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The image shows an action generation apparatus and an action generation method for implementing the embodiments of this application. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0191] Based on the same inventive concept, the principle and beneficial effects of the action generation device provided in the embodiments of this application are similar to the principle and beneficial effects of the action generation method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.

[0192] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Figure 7 The model training device shown can be mounted on a computer device, which can be a terminal device or a server. Figure 7 The model training apparatus shown can be used to perform the above. Figure 4 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 7 The model training device includes:

[0193] The acquisition unit 701 is used to acquire sample text and the sample action sequence corresponding to the sample text;

[0194] Processing unit 702 is used to perform adaptive masking processing on sample action sequences to obtain key action feature sequences corresponding to sample action sequences.

[0195] And used to perform M iterations of prediction on the sample text through the first training module to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence; the temporal feature corresponding to any predicted action feature is used to indicate the arrangement position of the predicted action feature in the sample action sequence, where M is an integer greater than 1;

[0196] And to train the first training module based on the difference between the predicted action feature sequence and the key action feature sequence, and the difference between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action feature, to obtain the action generation module; the temporal feature corresponding to any key action feature is used to indicate the arrangement position of the key action feature in the sample action sequence.

[0197] And to combine the mask features with the key action feature sequence according to the temporal features corresponding to each key action feature to obtain the target feature sequence carrying the mask features, with the mask features filling the target feature sequence in positions other than those indicated by the temporal features corresponding to the key action features;

[0198] And used to reconstruct the action sequence of the target feature sequence through the second training module to obtain the predicted action sequence corresponding to the sample text;

[0199] And a second training module is trained based on the difference between the predicted action sequence and the sample action sequence to obtain the reconstruction module; the action generation module and the reconstruction module are included in the action generation model.

[0200] In one embodiment, the processing unit 702 is configured to combine the mask features with the key action feature sequence according to the temporal features corresponding to each key action feature to obtain a target feature sequence carrying the mask features, specifically configured to:

[0201] The key action feature sequence is quantized by using a vocabulary to obtain the quantized key action feature sequence.

[0202] The quantized key action feature sequence is filled with mask features to obtain the target feature sequence carrying the mask features; the time length corresponding to the target feature sequence matches the time length corresponding to the sample action sequence.

[0203] In one implementation, the lexicon includes K vectors, each vector corresponding to a different action, where K is an integer greater than 1; the processing unit 702 is used to quantize the key action feature sequence using the lexicon to obtain a quantized key action feature sequence, specifically for:

[0204] Based on the similarity between each key action feature and K vectors in the key action feature sequence, each key action feature is transformed to obtain a quantitative representation of each key action.

[0205] The quantized representations of each key action are dequantized to obtain the quantized key action feature sequence.

[0206] In one implementation, the first training module includes an action network and a temporal network; the processing unit 702 performs M iterations of prediction on the sample text through the first training module to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence, including the following process:

[0207] The action network is used to predict the action of the i-th input data, and the vector representation of the i-th predicted action is obtained; i is a positive integer less than or equal to M.

[0208] The vector representation of the i-th predicted action is processed by feature mapping to obtain the features of the i-th predicted action;

[0209] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal vector representation corresponding to the i-th predicted action feature; the fused data is obtained by fusing the vector representation of the i-th predicted action and the i-th predicted action feature.

[0210] The temporal vector representation corresponding to the i-th predicted action feature is processed by feature mapping to obtain the temporal feature corresponding to the i-th predicted action feature.

[0211] In one implementation, the first training module includes an action network and a temporal network; the processing unit 702 trains the first training module based on the differences between the predicted action feature sequence and the key action feature sequence, and the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, to obtain the action generation module. The process includes:

[0212] The action network is trained based on the difference between the i-th predicted action feature and the i-th key action feature; i is a positive integer less than or equal to M.

[0213] The temporal network is trained based on the difference between the temporal features corresponding to the i-th predicted action feature and the temporal features corresponding to the i-th key action feature.

[0214] In one embodiment, the processing unit 702 is configured to perform adaptive masking processing on the sample action sequence to obtain a key action feature sequence corresponding to the sample action sequence, specifically for:

[0215] The sample action sequence is encoded to obtain the sample action features of each sample action;

[0216] Score prediction is performed on the action features of the samples to obtain the scores corresponding to each action feature of the samples.

[0217] The action features of samples with scores higher than the score threshold are packaged to obtain the key action feature sequence corresponding to the sample action sequence.

[0218] According to one embodiment of this application, Figure 4 The model training method shown can be partially implemented by... Figure 7 The individual units in the model training device shown are used to execute this. For example, Figure 4 Step S401 shown can be performed by Figure 7 The acquisition unit 701 shown is executed, and steps S402-S407 can be performed by... Figure 7 The processing unit 702 shown is executed. Figure 7 The units in the model training device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0219] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 4 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The diagram illustrates a model training apparatus and a model training method for implementing embodiments of this application. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the same medium, and executed therein.

[0220] Based on the same inventive concept, the principle and beneficial effects of the model training device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the model training method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0221] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 8 As shown, the computer device includes at least a processor 801, a communication interface 802, and a memory 803. The processor 801, communication interface 802, and memory 803 can be connected via a bus or other means. The processor 801 (or Central Processing Unit, CPU) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data. For example, the CPU can parse power-on / off commands issued by an object to the computer device and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device, and so on. The communication interface 802 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and can be used to send and receive data under the control of the processor 801; the communication interface 802 can also be used for data transmission and interaction within the computer device. The memory 803 is the storage device in the computer device, used to store programs and data. It can be understood that the memory 803 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 803 provides storage space for storing the operating system of the computer device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0222] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by the processor 801. It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0223] In one embodiment, processor 801 performs the following operations by running a computer program stored in memory 803:

[0224] Get the text to be processed;

[0225] The key action generation module in the action generation model performs M iterations of prediction on the text to be processed, and obtains the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence. The temporal features are used to indicate the arrangement position of the key action features in the target feature sequence, where M is an integer greater than 1.

[0226] The mask features are combined with the key action feature sequence based on the temporal features to obtain the target feature sequence carrying the mask features. The mask features are filled in the target feature sequence at positions other than those indicated by the temporal features.

[0227] The target feature sequence is reconstructed by the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed; the target action sequence is used to generate the action video of the target object.

[0228] As an optional embodiment, the key action generation module includes an action network and a temporal network; the process by which the processor 801 performs M iterations of prediction on the text to be processed through the key action generation module in the action generation model to obtain the key action feature sequence corresponding to the text to be processed and the temporal features corresponding to each key action feature in the key action feature sequence includes:

[0229] The action network is used to predict the action of the i-th input data to obtain the vector representation of the i-th key action, where i is a positive integer less than M.

[0230] The vector representation of the i-th key action is processed by feature mapping to obtain the features of the i-th key action;

[0231] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal feature corresponding to the i-th key action feature; the fused data is obtained by fusing the vector representation of the i-th key action and the i-th key action feature.

[0232] The i-th key action feature and the corresponding temporal feature are used as the input data for the (i+1)-th iteration of the action network.

[0233] The first input data for the action network is the embedding information, which is obtained by vector mapping of the text to be processed.

[0234] As an optional embodiment, the processor 801 performs temporal prediction on the fused data through a temporal network to obtain the temporal feature corresponding to the i-th key action feature. A specific embodiment of this is as follows:

[0235] Temporal prediction of the fused data is performed by a temporal network to obtain the temporal vector representation corresponding to the i-th key action feature;

[0236] The temporal vector representation corresponding to the i-th key action feature is subjected to feature mapping to obtain the temporal feature corresponding to the i-th key action feature.

[0237] As an optional embodiment, the processor 801 performs feature mapping processing on the vector representation of the i-th key action to obtain the features of the i-th key action. A specific embodiment of this is as follows:

[0238] The vector representation of the i-th key action is linearly mapped through a linear layer to obtain the mapping result of the i-th key action;

[0239] The mapping result of the i-th key action is normalized to obtain the feature of the i-th key action.

[0240] As an optional embodiment, the reconstruction module includes a demasking layer and a decoding layer; the demasking layer carries contextual information of the text to be processed; the processor 801 performs action reconstruction on the target feature sequence through the reconstruction module in the action generation model to obtain the target action sequence corresponding to the text to be processed. A specific embodiment is as follows:

[0241] The masked features in the target feature sequence are demasked by a demasking layer to obtain the demasked target feature sequence; the demasking process is used to reconstruct the masked features.

[0242] A decoding layer is used to decode the demasked target feature sequence to obtain the target action sequence corresponding to the text to be processed.

[0243] As an optional embodiment, the demasking layer includes a self-attention submodule and a residual submodule, the residual submodule including a one-dimensional convolution; the processor 801 performs demasking processing on the masked features in the target feature sequence through the demasking layer, and the specific embodiment of obtaining the demasked target feature sequence is as follows:

[0244] Attention masking is applied to the masking features in the target feature sequence to obtain the attention-masked target feature sequence. The attention masking process indicates the masking features in the target feature sequence.

[0245] The self-attention submodule is used to reconstruct the target feature sequence after the attention mask, resulting in the reconstructed target feature sequence.

[0246] The target feature sequence after feature reconstruction is transformed by one-dimensional convolution of the residual submodule to obtain the target feature sequence after masking.

[0247] As an optional embodiment, the target feature sequence includes action features corresponding to N temporal features, where N is an integer greater than 1; the process by which the processor 801 combines the mask features with the key action feature sequence according to the temporal features to obtain the target feature sequence carrying the mask features includes:

[0248] If the key action feature sequence does not include the action feature corresponding to the temporal feature j, then the mask feature is used to fill the position corresponding to the temporal feature j in the target feature sequence, where j is a positive integer less than or equal to N;

[0249] If the key action feature sequence includes the action feature corresponding to the temporal feature j, then the action feature corresponding to the temporal feature j is used to fill the position corresponding to the temporal feature j in the target feature sequence.

[0250] As an optional embodiment, each action in the target action sequence is represented by a K-dimensional vector, and each joint of the target object corresponds to at least one dimension of the K-dimensional vector, where K is an integer greater than 1; the processor 801 further performs the following operations by running the computer program in the memory 803:

[0251] By driving the joints of the target object to perform corresponding actions through the target motion sequence, the motion video of the target object is obtained.

[0252] In another embodiment, processor 801 performs the following operations by running a computer program stored in memory 803:

[0253] Obtain the sample text and the corresponding sequence of sample actions;

[0254] Adaptive masking is performed on the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence;

[0255] The first training module performs M iterations of prediction on the sample text to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence; the temporal feature corresponding to any predicted action feature is used to indicate the arrangement position of the predicted action feature in the sample action sequence, where M is an integer greater than 1.

[0256] Based on the differences between the predicted action feature sequence and the key action feature sequence, as well as the differences between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, the first training module is trained to obtain the action generation module; the temporal feature corresponding to any key action feature is used to indicate the arrangement position of the key action feature in the sample action sequence.

[0257] The mask features are combined with the key action feature sequence based on the temporal features corresponding to each key action feature to obtain the target feature sequence carrying the mask features. The mask features are filled in the target feature sequence at positions other than those indicated by the temporal features corresponding to the key action features.

[0258] The second training module reconstructs the target feature sequence to obtain the predicted action sequence corresponding to the sample text.

[0259] Based on the difference between the predicted action sequence and the sample action sequence, the second training module is trained to obtain the reconstruction module; the action generation module and the reconstruction module are included in the action generation model.

[0260] As an optional embodiment, the processor 801 combines the mask features with the key action feature sequence according to the temporal features corresponding to each key action feature to obtain the target feature sequence carrying the mask features. A specific embodiment of this is as follows:

[0261] The key action feature sequence is quantized by using a vocabulary to obtain the quantized key action feature sequence.

[0262] The quantized key action feature sequence is filled with mask features to obtain the target feature sequence carrying the mask features; the time length corresponding to the target feature sequence matches the time length corresponding to the sample action sequence.

[0263] As an optional embodiment, the vocabulary includes K vectors, with different vectors corresponding to different actions, where K is an integer greater than 1; the processor 801 quantizes the key action feature sequence using the vocabulary to obtain the quantized key action feature sequence. A specific embodiment is as follows:

[0264] Based on the similarity between each key action feature and K vectors in the key action feature sequence, each key action feature is transformed to obtain a quantitative representation of each key action.

[0265] The quantized representations of each key action are dequantized to obtain the quantized key action feature sequence.

[0266] As an optional embodiment, the first training module includes an action network and a temporal network; the processor 801 performs M iterations of prediction on the sample text through the first training module to obtain the predicted action feature sequence and the temporal features corresponding to each predicted action feature in the predicted action feature sequence, including the following process:

[0267] The action network is used to predict the action of the i-th input data, and the vector representation of the i-th predicted action is obtained; i is a positive integer less than or equal to M.

[0268] The vector representation of the i-th predicted action is processed by feature mapping to obtain the features of the i-th predicted action;

[0269] Temporal prediction is performed on the fused data using a temporal network to obtain the temporal vector representation corresponding to the i-th predicted action feature; the fused data is obtained by fusing the vector representation of the i-th predicted action and the i-th predicted action feature.

[0270] The temporal vector representation corresponding to the i-th predicted action feature is processed by feature mapping to obtain the temporal feature corresponding to the i-th predicted action feature.

[0271] As an optional embodiment, the first training module includes an action network and a temporal network; the processor 801 trains the first training module based on the difference between the predicted action feature sequence and the key action feature sequence, and the difference between the temporal features corresponding to each predicted action feature and the temporal features corresponding to the corresponding key action features, to obtain the action generation module. The process includes:

[0272] The action network is trained based on the difference between the i-th predicted action feature and the i-th key action feature; i is a positive integer less than or equal to M.

[0273] The temporal network is trained based on the difference between the temporal features corresponding to the i-th predicted action feature and the temporal features corresponding to the i-th key action feature.

[0274] As an optional embodiment, the processor 801 performs adaptive masking processing on the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence. A specific embodiment of this is as follows:

[0275] The sample action sequence is encoded to obtain the sample action features of each sample action;

[0276] Score prediction is performed on the action features of the samples to obtain the scores corresponding to each action feature of the samples.

[0277] The action features of samples with scores higher than the score threshold are packaged to obtain the key action feature sequence corresponding to the sample action sequence.

[0278] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the action generation method and model training method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0279] This application also provides a computer-readable storage medium storing a computer program, which is adapted to be loaded by a processor and execute the action generation method of the above method embodiments, or adapted to be loaded by a processor and execute the model training method of the above method embodiments.

[0280] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned action generation method or the aforementioned model training method.

[0281] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0282] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0283] In the embodiments of this application, the term "module" or "unit" refers to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0284] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0285] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.< / eos>

Claims

1. An action generation method characterized by comprising: The method comprises: acquiring a text to be processed; performing M times of iterative prediction on the text to be processed by a key action generation module in an action generation model to obtain a key action feature sequence corresponding to the text to be processed and a time sequence feature corresponding to each key action feature in the key action feature sequence, the time sequence feature being used to indicate an arrangement position of the key action feature in a target feature sequence, M being an integer greater than 1; combining a mask feature with the key action feature sequence according to the time sequence feature to obtain a target feature sequence carrying the mask feature, the mask feature being filled in positions other than the arrangement position indicated by the time sequence feature in the target feature sequence; performing action reconstruction on the target feature sequence by a reconstruction module in the action generation model to obtain a target action sequence corresponding to the text to be processed, the target action sequence being used to generate an action video of a target object.

2. The method of claim 1, wherein, The key action generation module comprises an action network and a time sequence network; the process of performing M times of iterative prediction on the text to be processed by the key action generation module in the action generation model to obtain a key action feature sequence corresponding to the text to be processed and a time sequence feature corresponding to each key action feature in the key action feature sequence comprises: performing action prediction on input data of the i-th time by the action network to obtain a vector representation of the i-th key action, i being a positive integer less than M; performing feature mapping processing on the vector representation of the i-th key action to obtain an i-th key action feature; performing time sequence prediction on fusion data by the time sequence network to obtain a time sequence feature corresponding to the i-th key action feature; the fusion data is obtained by fusing the vector representation of the i-th key action and the i-th key action feature; taking the i-th key action feature and the time sequence feature corresponding to the i-th key action feature as input data of the i+1-th time of the action network; wherein the input data of the first time of the action network is embedding information, and the embedding information is obtained by performing vector mapping on the text to be processed.

3. The method of claim 2, wherein, The process of performing time sequence prediction on the fusion data by the time sequence network to obtain a time sequence feature corresponding to the i-th key action feature comprises: performing time sequence prediction on the fusion data by the time sequence network to obtain a time sequence vector representation corresponding to the i-th key action feature; performing feature mapping processing on the time sequence vector representation corresponding to the i-th key action feature to obtain a time sequence feature corresponding to the i-th key action feature.

4. The method of claim 2, wherein, The process of performing feature mapping processing on the vector representation of the i-th key action to obtain an i-th key action feature comprises: performing linear mapping on the vector representation of the i-th key action by a linear layer to obtain a mapping result of the i-th key action; performing normalization processing on the mapping result of the i-th key action to obtain an i-th key action feature.

5. The method of claim 1, wherein, The reconstruction module comprises a de-masking layer and a decoding layer; the de-masking layer carries context information of the text to be processed; the action reconstruction of the target feature sequence by the reconstruction module in the action generation model obtains a target action sequence corresponding to the text to be processed, comprising: The de-masking layer is used for de-masking processing of the mask feature in the target feature sequence to obtain a de-masked target feature sequence; the de-masking processing is used for reconstructing the mask feature; The decoding layer is used for decoding the de-masked target feature sequence to obtain the target action sequence corresponding to the text to be processed.

6. The method of claim 5, wherein, The de-masking layer comprises a self-attention submodule and a residual submodule, and the residual submodule comprises a one-dimensional convolution; the de-masking processing of the mask feature in the target feature sequence by the de-masking layer comprises: The mask feature in the target feature sequence is subjected to attention mask processing to obtain an attention-masked target feature sequence, and the attention mask processing indicates the mask feature in the target feature sequence; The self-attention submodule is used for feature reconstruction of the attention-masked target feature sequence to obtain a feature-reconstructed target feature sequence; The one-dimensional convolution of the residual submodule is used for dimension conversion of the feature-reconstructed target feature sequence to obtain the de-masked target feature sequence.

7. The method of claim 1, wherein, The target feature sequence comprises N action features corresponding to N time sequence features, N is an integer greater than 1; the process of combining the mask feature with the key action feature sequence according to the time sequence feature to obtain the target feature sequence carrying the mask feature comprises: If the key action feature sequence does not comprise an action feature corresponding to time sequence feature j, the mask feature is used to fill the position corresponding to time sequence feature j in the target feature sequence, j is a positive integer less than or equal to N; If the key action feature sequence comprises an action feature corresponding to time sequence feature j, the action feature corresponding to time sequence feature j is used to fill the position corresponding to time sequence feature j in the target feature sequence.

8. The method of claim 1, wherein, Each action in the target action sequence is represented by a K-dimensional vector, and each joint of the target object corresponds to at least one dimension in the K-dimensional vector, K is an integer greater than 1; the method further comprises: The target action sequence is used to drive each joint of the target object to perform a corresponding action to obtain an action video of the target object.

9. A model training method, comprising: The method comprises: Obtaining a sample text and a sample action sequence corresponding to the sample text; Adaptive mask processing is performed on the sample action sequence to obtain a key action feature sequence corresponding to the sample action sequence; The first to-be-trained module is used for M times of iterative prediction of the sample text to obtain a predicted action feature sequence and a time sequence feature corresponding to each predicted action feature in the predicted action feature sequence; any predicted action feature corresponds to a time sequence feature for indicating the arrangement position of the predicted action feature in the sample action sequence, M is an integer greater than 1; The first to-be-trained module is trained based on differences between the predicted action feature sequence and the key action feature sequence and differences between time sequence features corresponding to each predicted action feature and time sequence features corresponding to the corresponding key action feature, to obtain an action generation module; the time sequence feature corresponding to any key action feature is used to indicate the arrangement position of the key action feature in the sample action sequence; The mask feature is combined with the key action feature sequence according to the time sequence feature corresponding to each key action feature, to obtain a target feature sequence carrying the mask feature, and the mask feature is filled in positions other than the arrangement position indicated by the time sequence feature corresponding to the key action feature in the target feature sequence; An action reconstruction is performed on the target feature sequence by a second to-be-trained module, to obtain a predicted action sequence corresponding to the sample text; The second to-be-trained module is trained based on differences between the predicted action sequence and the sample action sequence, to obtain a reconstruction module; the action generation module and the reconstruction module are included in an action generation model.

10. The method of claim 9, wherein, The mask feature is combined with the key action feature sequence according to the time sequence feature corresponding to each key action feature, to obtain a target feature sequence carrying the mask feature, and the mask feature is filled in positions other than the arrangement position indicated by the time sequence feature corresponding to the key action feature in the target feature sequence; The key action feature sequence is quantized by using a vocabulary, to obtain a quantized key action feature sequence; The quantized key action feature sequence is filled with the mask feature, to obtain a target feature sequence carrying the mask feature; the time length corresponding to the target feature sequence matches the time length corresponding to the sample action sequence.

11. The method of claim 10, wherein, The vocabulary includes K vectors, different vectors correspond to different actions, and K is an integer greater than 1; the key action feature sequence is quantized by using the vocabulary, to obtain a quantized key action feature sequence, including: Each key action feature is converted based on a similarity between the key action feature and the K vectors, to obtain a quantized representation of each key action; The quantized representation of each key action is dequantized, to obtain the quantized key action feature sequence.

12. The method of claim 9, wherein, The first to-be-trained module includes an action network and a time sequence network; the process of predicting the sample text by the first to-be-trained module for M times of iteration to obtain a predicted action feature sequence and a time sequence feature corresponding to each predicted action feature in the predicted action feature sequence includes: An i-th predicted action vector representation is obtained by performing action prediction on i-th input data by the action network; i is a positive integer less than or equal to M; A feature mapping is performed on the i-th predicted action vector representation, to obtain an i-th predicted action feature; A time sequence vector representation corresponding to the i-th predicted action feature is obtained by performing time sequence prediction on fused data by the time sequence network; the fused data is obtained by fusing the i-th predicted action vector representation and the i-th predicted action feature; perform feature mapping processing on the time sequence vector representation corresponding to the i th predicted action feature, to obtain a time sequence feature corresponding to the i th predicted action feature.

13. The method of claim 9, wherein, The first to-be-trained module includes an action network and a time sequence network; the process of training the first to-be-trained module based on the difference between the predicted action feature sequence and the key action feature sequence, and the difference between the time sequence feature corresponding to each predicted action feature and the time sequence feature corresponding to the corresponding key action feature, to obtain an action generation module includes: training the action network based on the difference between the i th predicted action feature and the i th key action feature; i is a positive integer less than or equal to M; training the time sequence network based on the difference between the time sequence feature corresponding to the i th predicted action feature and the time sequence feature corresponding to the i th key action feature.

14. The method of claim 9, wherein, The adaptive mask processing of the sample action sequence to obtain the key action feature sequence corresponding to the sample action sequence includes: encoding the sample action sequence to obtain a sample action feature of each sample action; performing score prediction on the sample action feature to obtain a score corresponding to each sample action feature; packaging the sample action feature with a score higher than the score threshold to obtain the key action feature sequence corresponding to the sample action sequence.

15. An action generation apparatus characterized by comprising: The action generation device includes: an acquisition unit configured to acquire a to-be-processed text; a processing unit configured to perform M times of iterative prediction on the to-be-processed text by a key action generation module in an action generation model, to obtain a key action feature sequence corresponding to the to-be-processed text and a time sequence feature corresponding to each key action feature in the key action feature sequence, the time sequence feature being used to indicate an arrangement position of the key action feature in a target feature sequence, and M being an integer greater than 1; and configured to combine a mask feature with the key action feature sequence according to the time sequence feature, to obtain a target feature sequence carrying the mask feature, the mask feature being filled in positions other than the arrangement position indicated by the time sequence feature in the target feature sequence; and configured to perform action reconstruction on the target feature sequence by a reconstruction module in the action generation model, to obtain a target action sequence corresponding to the to-be-processed text; the target action sequence is used to generate an action video of a target object.

16. A model training apparatus, comprising: The model training device includes: an acquisition unit configured to acquire a sample text and a sample action sequence corresponding to the sample text; a processing unit configured to perform adaptive mask processing on the sample action sequence to obtain a key action feature sequence corresponding to the sample action sequence; and configured to perform M times of iterative prediction on the sample text by a first to-be-trained module, to obtain a predicted action feature sequence and a time sequence feature corresponding to each predicted action feature in the predicted action feature sequence; the time sequence feature corresponding to any predicted action feature is used to indicate an arrangement position of the predicted action feature in the sample action sequence, and M is an integer greater than 1; and the first to-be-trained module is trained based on differences between the predicted action feature sequence and the key action feature sequence and differences between time sequence features corresponding to respective predicted action features and time sequence features corresponding to respective key action features, to obtain an action generation module; a time sequence feature corresponding to any key action feature is used to indicate an arrangement position of the key action feature in the sample action sequence; and a mask feature is combined with the key action feature sequence according to time sequence features corresponding to respective key action features, to obtain a target feature sequence carrying the mask feature, the mask feature being filled in positions other than the arrangement positions indicated by the time sequence features corresponding to the key action features in the target feature sequence; and the target feature sequence is subjected to action reconstruction by a second to-be-trained module, to obtain a predicted action sequence corresponding to the sample text; and the second to-be-trained module is trained based on differences between the predicted action sequence and the sample action sequence, to obtain a reconstruction module; the action generation module and the reconstruction module are included in an action generation model.

17. A computer device, comprising: comprising: a memory having a computer program stored therein; a processor configured to load the computer program and implement the action generation method according to any one of claims 1-8; or to load the computer program and implement the model training method according to any one of claims 9-14.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by the processor and execute the action generation method according to any one of claims 1-8; or adapted to be loaded by the processor and execute the model training method according to any one of claims 9-14.

19. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is adapted to be loaded by the processor and execute the action generation method according to any one of claims 1-8; or adapted to be loaded by the processor and execute the model training method according to any one of claims 9-14.

Images