Data processing method for task processing model and virtual character video generation method

Abstract

[0013] The present disclosure provides a data processing method for a task processing model and a virtual character animation generation method, the data processing method for the task processing model including the steps of: acquiring a first sample set, the first sample set including multimodal sample guide information; inputting the sample guide information and a sample task sequence into an initial processing model to acquire predicted task features corresponding to the sample guide information; training the initial processing model based on the predicted task features and the sample task features corresponding to the sample task sequence; and, when a first predetermined stopping condition is reached, acquiring model parameters of the processing model obtained by the training, the sample task features being obtained by quantizing and encoding the sample task sequence; and transmitting the model parameters of the trained task processing model to an end device. Because the task processing model is trained based on the multimodal sample guide information, multimodal task integration can be achieved, improving the accuracy and versatility of the model.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application filed with the China National Intellectual Property Administration on December 15, 2022, with an application number of 202211611176.1 and an application title of "Data Processing Method of Task Processing Model and Virtual Character Video Generation Method", and all of its content is incorporated herein by reference.

[0002] The embodiments of this specification relate to the field of computer technology, particularly to the data processing method of a task processing model. One or more embodiments of this specification relate simultaneously to a virtual character video generation method, a data processing system of a task processing model, a data processing device of a task processing model, a virtual character video generation device, a computing device, and a computer-readable storage medium.

Background Art

[0003] With the development of computer technology, motion generation has gradually become a key step in video, games, and digital human applications. For example, it includes the motion generation of roles in videos, roles in games, customer objects in web pages or application software, and virtual roles in movie production. The reality of motion is one of the important elements reflecting the reality and naturalness of the interaction between the role and the environment.

[0004] Currently, motion generation depends on the motion collection of many real characters, which is inefficient, has high requirements for the environment and collection hardware, and the accuracy of the generated motion is low, requiring post-production. Therefore, there is a strong demand for a highly versatile and accurate motion generation means.

Summary of the Invention

Problems to be Solved by the Invention

[0005] In view of the above, the embodiments of this specification provide a data processing method for a task processing model. One or more embodiments of this specification simultaneously relate to a virtual character video generation method, a data processing system for a task processing model, a data processing device for a task processing model, a virtual character video generation device, a computing device, a computer-readable storage medium, and a computer program, and solve the technical shortcomings of the prior art. [Means for solving the problem]

[0006] According to a first embodiment of the embodiments of this specification, a data processing method for a task processing model executed by a cloud device connected to a plurality of end devices is provided, the method is: A step of obtaining a first sample set, wherein the first sample set includes multimodal sample guide information, The steps include inputting sample guide information and sample task sequences into an initial processing model and obtaining predictive task features corresponding to the sample guide information, A step of training an initial processing model based on prediction task features and sample task features corresponding to a sample task sequence, and when a first predetermined stopping condition is reached, obtaining the model parameters of the training processing model, wherein the sample task features are obtained by quantizing and encoding a sample task sequence. The process includes the step of sending the model parameters of the task processing model obtained through training to an end device.

[0007] According to a second embodiment of the embodiments herein, a data processing method for a task processing model performed by an end device connected to a cloud device is provided, the method being: The steps include receiving model parameters of a task processing model sent by a cloud device and constructing a task processing model based on those model parameters, A step of receiving a task processing request entered by a user, wherein the task processing request includes task guide information, A step comprising inputting task guide information and the entire mask task sequence into a task processing model, and obtaining target task features corresponding to the task guide information through processing by the task processing model, wherein the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, and the sample task features are obtained by quantizing and encoding the sample task sequences. The process includes the steps of quantizing and decoding target task features and obtaining task processing results corresponding to task guide information.

[0008] According to a third embodiment of the embodiments described herein, a method for generating a virtual character video is provided, and the method is The step of receiving a virtual character video generation request sent by the front end, wherein the virtual character video generation request includes video guide information, The steps include: inputting video guide information and the entire mask video sequence into a virtual character video generation model, and obtaining virtual character video features corresponding to the video guide information through processing by the virtual character video generation model, wherein the virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences; The steps include: quantizing and decoding virtual character video features to obtain virtual character motion sequences corresponding to video guide information; The process includes the step of generating a virtual character video based on a virtual character motion sequence and sending it to the front end, thereby causing the front end to display the virtual character video.

[0009] According to a fourth aspect of the embodiments of this specification, a data processing device is provided for a task processing model executed by a cloud device connected to a plurality of end devices, the device being An acquisition module configured to acquire a first sample set, wherein the first sample set includes an acquisition module containing multimodal sample guide information, A first input module is configured to input sample guide information and sample task sequences into an initial processing model and to acquire predictive task features corresponding to the sample guide information. A training module configured to train an initial processing model based on prediction task features and sample task features corresponding to a sample task sequence, and to obtain the model parameters of the training processing model when a first predetermined stopping condition is reached, wherein the sample task features are obtained by quantizing and encoding a sample task sequence. It includes a transmission module configured to send model parameters of a task processing model obtained through training to an end device.

[0010] According to a fifth embodiment of the embodiments herein, a data processing device for a task processing model executed by an end device connected to a cloud device is provided, and the device is A build module configured to receive model parameters of a task processing model sent by a cloud device and build the task processing model based on those model parameters, A first receiving module configured to receive a task processing request entered by a user, wherein the task processing request includes task guide information, A second input module is configured to input task guide information and the entire mask task sequence into a task processing model, and to obtain target task features corresponding to the task guide information through processing by the task processing model, wherein the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, and the sample task features are obtained by quantizing and encoding the sample task sequences. The system includes a first decoding module configured to quantize and decode target task features and obtain task processing results corresponding to task guide information.

[0011] According to a sixth embodiment of the embodiments described herein, a virtual character video generation apparatus is provided, the apparatus is A second receiving module configured to receive a virtual character video generation request sent by the front end, wherein the virtual character video generation request includes video guide information, A third input module is configured to input video guide information and the entire mask video sequence into a virtual character video generation model, and to obtain virtual character video features corresponding to the video guide information through processing by the virtual character video generation model, wherein the virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences. A first decoding module is configured to quantize and decode virtual character video features and obtain virtual character motion sequences corresponding to video guide information, The system includes a generation module configured to generate a virtual character video based on a virtual character motion sequence and send it to the front end, thereby causing the front end to display the virtual character video.

[0012] According to a seventh embodiment of the embodiments herein, a data processing system for a task processing model is provided, the system is: An end device for constructing a first sample set and sending the first sample set to a cloud device, wherein the first sample set includes multimodal sample guide information. The system includes a cloud device for inputting sample guide information and a sample task sequence into an initial processing model, obtaining predicted task features corresponding to the sample guide information, training the initial processing model based on the predicted task features and sample task features corresponding to the sample task sequence, obtaining the model parameters of the trained processing model when a first predetermined stop condition is reached, and transmitting the model parameters of the trained task processing model to an end device, wherein the sample task features are obtained by quantizing and encoding the sample task sequence.

[0013] According to an eighth aspect of the embodiments of this specification, a computing device is provided, said computing device is Including memory and processor, The memory is used to store computer executable instructions, the processor is used to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, realize steps of the method provided in the first, second, or third embodiment.

[0014] According to a ninth embodiment of the embodiments described herein, a computer-readable storage medium is provided which stores computer-executable instructions, which, when executed by a processor, realize steps of the method provided in the first, second, or third embodiment.

[0015] According to a tenth embodiment of the embodiments of this specification, a computer program is provided, and when the computer program is executed on a computer, the computer is made to perform steps of the method provided in the first, second, or third embodiment. [Effects of the Invention]

[0016] The data processing method of the task processing model according to one embodiment of this specification is as follows: obtain a first sample set, where the first sample set contains multimodal sample guidance information, input the sample guidance information and the sample task sequence into an initial processing model, obtain the predicted task features corresponding to the sample guidance information, train the initial processing model based on the predicted task features and the sample task features corresponding to the sample task sequence, when reaching the first predetermined stop condition, obtain the model parameters of the trained processing model, the sample task features are obtained by quantizing and encoding the sample task sequence, and send the model parameters of the trained task processing model to the end device. Since the task processing model is obtained by training based on multimodal sample guidance information, multimodal task integration can be realized, improving the accuracy and generality of the model.

Brief Description of Drawings

[0017] [Figure 1] It is a configuration diagram of the data processing system of the task processing model according to one embodiment of this specification. [Figure 2] It is a configuration diagram of the data processing system of another task processing model according to one embodiment of this specification. [Figure 3] It is a flowchart of the data processing method of the task processing model according to one embodiment of this specification. [Figure 4] It is a flowchart of the data processing method of another task processing model according to one embodiment of this specification. [Figure 5] It is a flowchart of the virtual character video generation method according to one embodiment of this specification. [Figure 6] It is a schematic diagram of the processing process of the virtual character video generation method according to one embodiment of this specification. [Figure 7] It is a schematic diagram of the virtual character video generation interface according to one embodiment of this specification. [Figure 8]This is a flowchart of a data processing method for a quantization generative model according to one embodiment of this specification. [Figure 9] This is a flowchart illustrating the training of a task processing model and a quantization generation model according to one embodiment of this specification. [Figure 10] This is a schematic diagram showing the structure of a data processing device for a task processing model according to one embodiment of this specification. [Figure 11] This is a schematic diagram showing the structure of a data processing device for another task processing model according to one embodiment of this specification. [Figure 12] This is a schematic diagram showing the structure of a virtual character video generation device according to one embodiment of this specification. [Figure 13] This is a block diagram showing the structure of a computing device according to one embodiment of this specification. [Modes for carrying out the invention]

[0018] Many specific details are provided in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other forms different from those described herein, and a person skilled in the art can similarly infer these without departing from the intent of this specification; therefore, this specification is not limited to the specific implementations disclosed below.

[0019] The terms used in one or more embodiments of this specification are for the sole purpose of describing a particular embodiment and are not intended to limit one or more embodiments of this specification. The singular forms “one kind,” “the said,” and “the” used in one or more embodiments of this specification and the appended claims are intended to include the plural form unless the context clearly indicates otherwise. Furthermore, the terms “and / or” used in one or more embodiments of this specification should be understood to refer to and include any or all possible combinations of one or more related enumerated items.

[0020] In one or more embodiments of this specification, various pieces of information may be described using terms such as First, Second, etc., but it should be understood that this information should not be limited by these terms. These terms are used solely to distinguish the same kind of information. For example, First may be referred to as Second without departing from the scope of one or more embodiments of this specification, and similarly, Second may be referred to as First. Depending on the context, the word “if” as used herein may be interpreted as “when…,” “in the event that…,” or “in response to deciding….”

[0021] First, we will interpret the terms relating to one or more embodiments of this specification.

[0022] A Transformer is a neural network structure.

[0023] Seq2seq is an encoding-decoding structure in which the input and output sequences may be of different lengths.

[0024] Human motion refers to the skeleton data used to drive a human body model to perform limb movements in 3D (three-dimensional) digital humans, 3D games, and 3D videos. It consists of multiple frames, and the data in each frame describes the orientation and displacement of the entire body, as well as the rotation angles of each joint.

[0025] Diffusion models are a novel type of deep generative model. During training, samples are given noise of different intensities (intensity = 0, 1, ..., or T), and the model is asked to reconstruct the sample before noise was added, depending on the noise intensity and the sample after noise was added. During inference, the model is asked to progressively remove noise from a single random noise sample (noise intensity = T) to obtain samples with noise intensities of T-1, T-2, ..., 0, respectively, with the sample with noise intensity 0 being the final sample generated by the model.

[0026] With the advancement of computer technology, motion generation is gradually becoming a key step in video, games, and digital human applications. Traditional motion generation methods rely on collecting motion data from many real-world characters, which is inefficient, demanding on the environment and collection hardware, and requires final polishing. Currently, learning-based methods have advanced motion generation, enabling the generation of high-quality, continuous motion under different conditions through learning based on large amounts of motion capture data. However, currently, to meet the diverse needs of motion generation tasks, it is common to use specific frameworks and perform individual training for corresponding tasks, failing to benefit from different task or type datasets.

[0027] To solve the above problems, the embodiments of this specification provide a task processing framework that integrates multiple tasks, eliminating the gap between different tasks, enabling the learning of more meaningful information from different cross-modal data signals, and efficiently representing and learning different generation tasks. This allows for the realization of various different generation requests and provides rich motion material for video, games, digital human applications, etc., at low cost, efficiently and accurately. Specifically, a first sample set is acquired, the first sample set includes multimodal sample guide information, the sample guide information and sample task sequences are input into an initial processing model, predicted task features corresponding to the sample guide information are acquired, the initial processing model is trained based on the predicted task features and sample task features corresponding to the sample task sequences, and when a first predetermined stop condition is reached, the model parameters of the trained processing model are acquired, the sample task features are obtained by quantizing and encoding the sample task sequences, and the model parameters of the trained task processing model are transmitted to the end device. Since the task processing model is obtained by training based on multimodal sample guide information, multimodal task integration can be realized, improving the accuracy and versatility of the model.

[0028] This specification provides a data processing method for a task processing model, and also describes in detail, in the following embodiments, a method for generating virtual character videos, a data processing system for a task processing model, a data processing device for a task processing model, a virtual character video generation device, a computing device, a computer-readable storage medium, and a computer program.

[0029] Referring to Figure 1, Figure 1 shows a configuration diagram of a data processing system for a task processing model according to one embodiment of this specification, the data processing system for the task processing model includes a cloud device 100 and an end device 200. The end device 200 constructs the first sample set, sends the first sample set to the cloud device 100, and the first sample set contains multimodal sample guide information. The cloud device 100 inputs sample guide information and sample task sequences into the initial processing model, acquires predicted task features corresponding to the sample guide information, trains the initial processing model based on the predicted task features and sample task features corresponding to the sample task sequences, and when the first predetermined stop condition is reached, acquires the model parameters of the training processing model, the sample task features are obtained by quantizing and encoding the sample task sequences, and transmits the model parameters of the training task processing model to the end device 200.

[0030] Using the embodiment described herein, a first sample set is obtained, which includes multimodal sample guide information. The sample guide information and sample task sequences are input to an initial processing model. Predicted task features corresponding to the sample guide information are obtained. The initial processing model is trained based on the predicted task features and sample task features corresponding to the sample task sequences. When a first predetermined stop condition is reached, the model parameters of the trained processing model are obtained. The sample task features are obtained by quantizing and encoding the sample task sequences. The model parameters of the trained task processing model are transmitted to the end device. Since the task processing model is trained based on multimodal sample guide information, multimodal task integration can be achieved, improving the accuracy and versatility of the model.

[0031] Referring to Figure 2, Figure 2 shows a configuration diagram of a data processing system for another task processing model according to one embodiment of this specification, the system may include a cloud device 100 and a plurality of end devices 200. The plurality of end devices 200 can establish communication connections with each other via the cloud device 100, and in a task processing scene, the cloud device 100 is used to provide task processing services to the plurality of end devices 200, and the plurality of end devices 200 can each achieve real-time communication via the cloud device 100 as either a sender or a receiver.

[0032] The user can interact with the cloud device 100 via the end device 200 to receive data sent by other end devices 200, or to send data to other end devices 200. In a task processing scene, the user may deliver a data stream to the cloud device 100 via the end device 200, the cloud device 100 may generate an action based on the data stream, and push the action generation result to other end devices with established communication.

[0033] A connection is established between the end device 200 and the cloud device 100 via a network. The network is the medium that provides the communication link between the end device and the cloud device. The network can include various types of connections, such as wired, wireless, or fiber optic cables. Data transmitted by the end device 200 may require processing such as encoding, transcoding, or compression before being delivered to the cloud device 100.

[0034] The end device 200 may be a web page application such as a browser, an APP (Application program), or an H5 (HyperText Markup Language 5) application, a mini-application (also called an applet, a small application program), or a cloud application. The end device 200 can be developed and obtained based on a software development kit (SDK) for the corresponding service provided by the cloud device, for example, based on a Real Time Communication (RTC) SDK. The end device 200 can be deployed on an electronic device and may depend on the device's execution or execution of an APP within the device. The electronic device may have a display screen and support the viewing of information, and may be a personal mobile terminal such as a mobile phone, tablet, or personal computer. Various other types of applications may be deployed on the electronic device, such as human-machine interactive applications, model training applications, text processing applications, web page browser applications, shopping applications, search applications, instant communication tools, mailbox end devices, and social platform software.

[0035] The cloud device 100 may include servers that provide various services, such as a server that provides communication services to multiple end devices, a server for background training that provides support for models used by end devices, and a server that processes data sent by end devices. The cloud device 100 may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server may be a server for a distributed system, or a server combined with a blockchain. The server may be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud hosting equipped with artificial intelligence technology.

[0036] Referring to Figure 3, Figure 3 shows a flowchart of a data processing method for a task processing model according to one embodiment of this specification, the data processing method for the task processing model is applied to an end device connected to a cloud device, and specifically includes the following steps 302 to 308.

[0037] In step 302, the cloud device sends model parameters for the task processing model, and the task processing model is constructed based on these model parameters.

[0038] In the embodiments described herein, in order to perform task processing accurately and efficiently, model parameters of a task processing model transmitted by a cloud device can be received, a task processing model can be constructed based on the model parameters, and task processing can be implemented using the task processing model.

[0039] In step 304, the task processing request entered by the user is received, and the task processing request includes task guide information.

[0040] In one or more embodiments of this specification, task guide information can be acquired in the initial stage of job processing, the task processing process can be guided based on the task guide information, and processing results that match the task guide information can be generated efficiently and accurately.

[0041] Specifically, a task processing request may be a processing request for a different task, and the tasks include, but are not limited to, text generation tasks, motion generation tasks, and speech processing tasks. Task guide information is multimodal and includes, but is not limited to, text guide information, image guide information, speech guide information, trajectory guide information, and video guide information, and is specifically selected according to the actual situation, and the embodiments of this specification are not limited thereto. Taking a motion generation task as an example, the motion guide information may be motion guide information for different digital objects such as virtual characters, virtual animals, and virtual vehicles, the text guide information includes, but is not limited to, motion categories and natural language description text, and the image guide information may be understood as a visual signal, for example, a reference image, and the reference image includes reference motion, motion at a partial time point, etc., and aims to ensure that the motion at corresponding time points in the generated motion sequence, for example, motion insertion (in-betweening) and motion filling, match. Trajectory guide information may be understood as a trajectory signal, which represents a position corresponding to a different point in time in an action sequence, controls the direction of movement of an object, and can be used for motion control tasks.

[0042] In actual use, there are various ways in which a user can input a task processing request, and the specific method is selected depending on the actual situation; the embodiments described herein are not limited to this. In one possible implementation of this specification, a user can voluntarily submit a task processing request, and the task processing request includes task guide information. In another possible implementation of this specification, the task processing request includes an information identifier for the task guide information, and the task guide information corresponding to the information identifier can be retrieved from a guide information base, and the guide information base includes multiple task guide information entries.

[0043] In step 306, task guide information and the entire mask task sequence are input to the task processing model. The task processing model then obtains target task features corresponding to the task guide information. The task processing model is trained based on multimodal sample guide information, sample task sequences, and sample task features. The sample task features are obtained by quantizing and encoding the sample task sequences.

[0044] In one or more embodiments of this specification, after receiving a task processing request entered by a user, task guide information and the entire mask task sequence are further input into the task processing model, and target task features corresponding to the task guide information can be obtained by processing the task processing model.

[0045] Specifically, the task processing model is a Transformer model capable of extracting features from task guide information, and through processing by the task processing model, the task guide information can be processed as discrete target task features.

[0046] The task processing model includes a first encoder and a first decoder, and the above step involves inputting task guide information and the entire mask task sequence into the task processing model, and obtaining target task features corresponding to the task guide information through processing by the task processing model. The steps include inputting task guide information into the first encoder and obtaining task guide features corresponding to the task guide information, The process may include the step of inputting task guide features and the entire mask task sequence into a first decoder and obtaining target task features corresponding to the task guide information.

[0047] Specifically, a task sequence is a continuous representation of the task itself. For example, a motion sequence includes 23 principal joints, and the motion is primarily represented by the rotation angle of each skeleton, with the angles being continuous.

[0048] Furthermore, task guide information can be input to the first encoder, encoded, and a coded representation corresponding to the task guide information, i.e., task guide features, can be generated. After obtaining the task guide features, unlike training a task processing model, there are no sample task sequences that will be referenced when actually used. Therefore, the mask is repeated based on the entire mask task sequence to realize task processing in task sequences that are not referenced, predict the complete task sequence, and further, target task features corresponding to the task guide information can be obtained.

[0049] Using the embodiment described herein, task guide information is input to a first encoder to obtain the task guide function corresponding to the task guide information, and the task guide features and the entire mask task sequence are input to a first decoder to obtain the target task features corresponding to the task guide information. This enables efficient and accurate acquisition of the target task features.

[0050] In step 308, the target task features are quantized and decoded to obtain the task processing results corresponding to the task guide information.

[0051] In one or more embodiments of this specification, a task processing request entered by a user is received, the task processing request includes task guide information, the task guide information and the entire mask task sequence are input to a task processing model, the target task features corresponding to the task guide information are obtained by processing the task processing model, and then the target task features are further quantized and decoded to obtain a task processing result corresponding to the task guide information.

[0052] Using the embodiment described herein, a cloud device sends model parameters of a task processing model, a task processing model is constructed based on the model parameters, a user inputs a task processing request, the request includes task guide information, the task guide information and the entire mask task sequence are input to the task processing model, the task processing model processes to obtain target task features corresponding to the task guide information, the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, the sample task features are obtained by quantizing and encoding the sample task sequences, the target task features are quantized and decoded to obtain task processing results corresponding to the task guide information. Since the task processing model is obtained by training on multimodal sample guide information, multimodal task integration can be realized, and therefore the task processing model efficiently and accurately generates target task features, further improving the accuracy of the task processing results.

[0053] In the embodiments of this specification, the task processing model includes a first encoder and a first decoder, and the training method for the first encoder and the first decoder will be described in detail in each of the following embodiments.

[0054] Referring to Figure 4, Figure 4 shows a flowchart of a data processing method for another task processing model according to one embodiment of this specification, the data processing method for the task processing model is applied to a cloud device connected to multiple end devices, and specifically includes the following steps 402 to 408.

[0055] In step 402, the first sample set is obtained, which includes multimodal sample guide information.

[0056] Specifically, the sample guide information is a multimodal control signal and includes at least two of the following types of information: sample text information, sample screen information, sample trajectory information, sample audio information, and sample video information.

[0057] There are multiple methods for acquiring the first sample set. The first sample set may be constructed by artificially inputting a large amount of multimodal sample guide information, or by reading a large amount of multimodal sample guide information from another data acquisition device or database. The method for acquiring the first sample set is specifically selected according to the actual situation, and the embodiments described herein are not limited to this.

[0058] In step 404, sample guide information and sample task sequences are input into the initial processing model, and predictive task features corresponding to the sample guide information are obtained.

[0059] In step 406, an initial processing model is trained based on the predicted task features and the sample task features corresponding to the sample task sequence. If the first predetermined stopping condition is reached, the model parameters of the trained processing model are obtained. The sample task features are obtained by quantizing and encoding the sample task sequence.

[0060] In step 408, the model parameters of the task processing model obtained through training are sent to the end device.

[0061] Specifically, the sample task sequence is a continuous representation of the sample tasks themselves. The initial processing model includes a first encoder and a first decoder, and the first predetermined stop condition includes a first stop subcondition.

[0062] Using the embodiment described herein, a first sample set is obtained, which includes multimodal sample guide information. The sample guide information and sample task sequences are input to an initial processing model. Predicted task features corresponding to the sample guide information are obtained. The initial processing model is trained based on the predicted task features and sample task features corresponding to the sample task sequences. When a first predetermined stop condition is reached, the model parameters of the trained processing model are obtained. The sample task features are obtained by quantizing and encoding the sample task sequences. The model parameters of the trained task processing model are transmitted to the end device. Since the task processing model is trained based on multimodal sample guide information, multimodal task integration can be achieved, improving the accuracy and versatility of the model.

[0063] In one selectable embodiment of this specification, the above step of inputting sample guide information and a sample task sequence into an initial processing model and obtaining predictive task features corresponding to the sample guide information is: A step of extracting first sample guide information from a first sample set, wherein the first sample guide information is any one of the sample guide information in the first sample set, The first step involves inputting the first sample guide information into a pre-trained first encoder to obtain the first sample guide features, The steps include: masking a sample task sequence and obtaining a masked task sequence; The process may include the steps of inputting a first sample guide feature and a mask task sequence into a first decoder to obtain a first prediction task feature, The above steps involve training an initial processing model based on the predicted task features and sample task features corresponding to the sample task sequence, and obtaining the model parameters of the training processing model when a first predetermined stopping condition is reached. The steps include: calculating the decoding loss value based on the first predicted task features and the sample task features corresponding to the sample task sequence; The process may include steps such as adjusting the parameters of the first decoder based on the decoding loss value, then going back to extract the first sample guide information from the first sample set, and if the first stop subcondition is met, obtaining the trained first decoder.

[0064] The pre-trained first encoder is the first encoder of the pre-trained initial processing model. When training the first decoder, a training method similar to that of a feature extraction model (ViT, Vision Transformer) can be used. This method involves decomposing the sample task sequence corresponding to the sample guide information, treating adjacent time points and keypoints as a single patch, obtaining multiple patches, further smoothing each patch, mapping them to the input embedding with a linear layer, and then adding a learnable [Motion MASK] token to represent the masked positions.

[0065] Furthermore, in the training process of the first decoder, a repeating mask training method can be employed. The repeating mask process includes degeneration and recovery processes, specifically degenerating at the discrete code level, with each iteration replacing features (embeddings) in the sample task sequence with mask token embeddings with a predetermined probability. For the sampling method, the network can be used to predict the sampling position, thereby improving the convergence speed and making the prediction results more stable.

[0066] In one possible implementation of this specification, the first stop subcondition includes the decoding loss value being less than or equal to a first predetermined threshold, the first predetermined threshold being specifically selected according to the actual circumstances, and the embodiments of this specification are not limited thereto. The guide features and mask task sequence of the first sample are input to the first decoder, the first prediction task features are obtained, and after obtaining the first prediction task features, the decoding loss value is calculated based on the first prediction task features and the sample task features corresponding to the sample task sequence, and the decoding loss value is compared with the first predetermined threshold.

[0067] Specifically, if the decoding loss value is greater than the first predetermined threshold, it indicates that the difference between the first predicted task features and the sample task features corresponding to the sample task sequence is large, and that the predictive ability of the first decoder for the first sample guide features and mask task sequence is low. In this case, the parameters of the first decoder are adjusted, and the first sample guide information is extracted from the multimodal sample guide information, allowing the first decoder to continue training. When the decoding loss value falls below the first predetermined threshold, it indicates that the difference between the first predicted task features and the sample task features corresponding to the sample task sequence is small, reaching the first stop subcondition, and a trained first decoder is obtained.

[0068] Using the embodiments described herein, a first sample guide feature and a mask task sequence are input to a first decoder, a first prediction task feature is obtained, a decoding loss value is calculated based on the first prediction task feature and the sample task feature corresponding to the sample task sequence, the decoding loss value is compared with a first predetermined threshold, and if it is greater than the first predetermined threshold, the first decoder is trained until the decoding loss value is less than or equal to the first predetermined threshold, training for the first decoder is completed, and the parameters of the first decoder are continuously adjusted to make the final obtained first decoder more accurate.

[0069] In another possible implementation of this specification, in addition to comparing the magnitude of the decoding loss value with a first predetermined threshold, it is possible to determine whether the current first decoder is trained or not, in accordance with the number of iterations.

[0070] Specifically, if the decoding loss value is greater than a first predetermined threshold, the parameters of the first decoder are adjusted, and the step of extracting the first sample guide information from the multimodal sample guide information is performed, the first decoder is trained, and when a first predetermined number of iterations is reached, the iteration is stopped and a trained first decoder is obtained. The first predetermined number of iterations is specifically selected according to the actual situation, and the embodiments herein are not limited thereto.

[0071] Using the embodiments described herein, a first sample guide feature and a mask task sequence are input to a first decoder, a first prediction task feature is obtained, a decoding loss value is calculated based on the first prediction task feature and the sample task feature corresponding to the sample task sequence, the decoding loss value is compared with a first predetermined threshold, and if it is greater than the first predetermined threshold, the first decoder is trained until a first predetermined number of iterations is reached to complete the training of the first decoder, and the parameters of the first decoder are continuously adjusted to make the final obtained first decoder more accurate.

[0072] In practice, there are many functions for calculating the decoding loss value, such as the cross-entropy loss function, L1 norm loss function, maximum loss function, mean squared error loss function, and logarithmic loss function. The specific function to be selected depends on the actual situation, and the examples in this specification are not limited to these.

[0073] In one selectable embodiment of this specification, the first predetermined stop condition includes a second stop subcondition, and the training method for the first encoder of the task processing model is: A step of obtaining a second sample set, wherein the second sample set includes multimodal sample guide information, and the sample guide information includes sample guide features, A step of extracting second sample guide information from the second sample set, wherein the second sample guide information is any one of the sample guide information in the second sample set, The steps include inputting second sample guide information into the first encoder and obtaining first predictive guide features corresponding to the second sample guide information, A step of calculating the coding loss value based on the first prediction guide feature and the second sample guide feature included in the second sample guide information, The process may include the steps of adjusting the parameters of the first encoder based on the encoding loss value, then going back and extracting second sample guide information from the second sample set, and if a second stop subcondition is met, obtaining the trained first encoder.

[0074] Specifically, the task processing model structurally employs an encoder-decoder framework, where the first encoder and first decoder are connected via a cross-attention layer to achieve seq2seq association and no longer rely on the compression of a single implicit variable. Both the first encoder and first decoder are multi-layer bidirectional transformer structures. The first encoder receives multimodal control signals, i.e., multimodal sample guide information, which includes at least two of the following types of information: sample text information, sample image information, sample trajectory information, sample audio information, and sample video information.

[0075] When training the first encoder, a large-scale pre-training model may be used, such as a multimodal generative model (OFA, One-For-All) or an image-text correlation matching model (CLIP, Contrastive Language-Image Pre-training). The specific model will be selected according to the actual situation, and the examples in this specification are not limited to this.

[0076] In actual use, there are multiple methods for obtaining the second sample set. The second sample set may be constructed by artificially inputting a large amount of multimodal sample guide information, or by reading a large amount of multimodal sample guide information from another data acquisition device or database. The method for obtaining the second sample set is specifically selected according to the actual situation, and the embodiments described herein are not limited to this.

[0077] In one possible implementation of this specification, the second stopping subcondition includes the coding loss value being less than or equal to a second predetermined threshold, the second predetermined threshold being specifically selected according to the actual circumstances, and the embodiments of this specification are not limited thereto. Guide information of the second sample is input to the first decoder, a first predicted guide feature corresponding to the second sample guide information is obtained, and after obtaining the first predicted guide feature, the coding loss value is calculated based on the first predicted guide feature and the second sample guide feature included in the second sample guide information, and the coding loss value is compared with the second predetermined threshold.

[0078] Specifically, if the coding loss value is greater than the second predetermined threshold, it indicates that the difference between the first predicted guide feature and the second sample guide feature contained in the second sample guide information is large, and that the predictive ability of the first encoder for the second sample guide information is low. In this case, the parameters of the first encoder are adjusted, and the step of going back to extract the second sample guide information from the multimodal sample guide information is performed, allowing the first encoder to continue training. When the coding loss value becomes less than or equal to the second predetermined threshold, it indicates that the difference between the first predicted guide feature and the second sample guide feature contained in the second sample guide information is small, and the second stop subcondition is reached, and the trained first encoder is obtained.

[0079] Using the embodiments described herein, second sample guide information is input to the first encoder, first predicted guide features corresponding to the second sample guide information are obtained, an encoded loss value is calculated based on the first predicted guide features and the second sample guide features included in the second sample guide information, the encoded loss value is compared with a second predetermined threshold, and if it is greater than the second predetermined threshold, the first encoder is trained until the encoded loss value is less than or equal to the second predetermined threshold, training for the first encoder is completed, and the parameters of the first encoder are continuously adjusted to make the final obtained first encoder more accurate.

[0080] In another possible implementation of this specification, in addition to comparing the magnitude of the coding loss value with a second predetermined threshold, it is possible to determine whether the current first encoder is trained or not, in accordance with the number of iterations.

[0081] Specifically, if the coding loss value is greater than a second predetermined threshold, the parameters of the first encoder are adjusted, and the second sample guide information is extracted from the multimodal sample guide information. The first encoder is then trained, and when the second predetermined number of iterations is reached, the iterations are stopped, and the trained first encoder is obtained. The second predetermined number of iterations is specifically selected according to the actual situation, and the embodiments herein are not limited thereto.

[0082] Using the embodiments described herein, second sample guide information is input to the first encoder, a first predicted guide feature corresponding to the second sample guide information is obtained, an encoding loss value is calculated based on the first predicted guide feature and the second sample guide feature included in the second sample guide information, the encoding loss value is compared with a second predetermined threshold, and if it is greater than the second predetermined threshold, the first encoder is trained until a second predetermined number of iterations is reached to complete the training of the first encoder, and the parameters of the first encoder are continuously adjusted to make the final obtained first encoder more accurate.

[0083] In practice, there are many functions for calculating the coding loss value, such as the cross-entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, and logarithmic loss function. The specific function is selected according to the actual situation, and the embodiments described herein are not limited to these. Preferably, the coding loss value can be calculated using the cross-entropy loss function. By using the cross-entropy loss function to calculate the cross-entropy of the first prediction guide feature and the second sample guide feature included in the second sample guide information, and using this to obtain the coding loss value, the efficiency of calculating the coding loss value is improved, thereby improving the training efficiency of the first encoder.

[0084] In one selectable embodiment of this specification, the sample task feature corresponding to the sample task sequence is obtained by quantizing and encoding the sample task sequence, that is, before the above step of calculating the decoding loss value based on the first prediction task feature and the sample task feature corresponding to the sample task sequence, The process may further include the step of inputting a sample task sequence into a second encoder of a pre-trained quantization generative model, and obtaining sample task features corresponding to the sample task sequence through the encoding process of the second encoder.

[0085] The quantization generative model includes a second encoder and a second decoder. The second encoder can quantize a continuous task sequence to obtain discrete task features, and the second decoder can reconstruct the task sequence by quantizing and decoding the discrete task features. The second encoder of the quantization generative model (VQ, Vector Quantization) is an autoencoder structure in image generation. When training the quantization generative model, a discrete coding dictionary (codebook) is introduced in the network's intermediate process, and by targeting a reconstruction task, the model parameters of the quantization generative model can be iteratively updated, thereby obtaining the quantization generative model.

[0086] Using the embodiments described herein, a sample task sequence is input to a second encoder of a pre-trained quantization generative model, and the second encoder's encoding process is used to obtain sample task features corresponding to the sample task sequence. The efficiency and accuracy of obtaining sample task features corresponding to the sample task sequence are improved.

[0087] In one selectable embodiment of this specification, when training a task processing model, the accuracy of the task processing model can be improved by inputting corresponding timestep features (timestep embeddings) as training supplements in each iteration process, i.e., the step of inputting a first sample guide feature and a mask task sequence into a first decoder to obtain a first predicted task feature, A step to acquire pre-set time step features, wherein there is a one-to-one correspondence between the time step features and the number of training iterations, The process may include the step of inputting time-step features, first sample-guide features, and a mask task sequence into a first decoder to obtain first prediction task features.

[0088] Furthermore, since different time steps correspond to different learnable features, when training a task processing model, it is possible to obtain a timestep embedding corresponding to the current iteration count. By inputting the timestep embedding, the first sample guide feature, and the mask task sequence together into the first decoder, it is possible to obtain accurate first predicted task features.

[0089] As an example, assuming the current number of training iterations is 2, we determine the time step feature corresponding to the current number of training iterations 2 as X, and then input the time step feature X, the first sample guide feature, and the mask task sequence together into the first decoder to obtain the first prediction task feature.

[0090] When using the embodiments described herein, a pre-defined time step feature is obtained, and there is a one-to-one correspondence between the time step feature and the number of training iterations. The time step feature, the first sample guide feature, and the mask task sequence are input to the first decoder to obtain the first prediction task feature. By using the time step feature as a supplement to the training task processing model, the accuracy of the task processing model is improved.

[0091] In one selectable embodiment of this specification, the quantization generative model includes a second encoder and a second decoder, and the training method for the quantization generative model is: A step of obtaining a third sample set, wherein the third sample set includes multiple training sample sequences, A step of extracting a first training sample sequence from a third sample set, wherein the first training sample sequence is any one of the training sample sequences in the third sample set, The first training sample sequence is input to the second encoder to obtain the first test feature, The first test features and predetermined presentation information are input to the second decoder to obtain the first test sequence, A step of calculating the quantization loss value based on the first training sample sequence and the first test sequence, Based on the quantization loss value, the parameters of the second encoder and second decoder are adjusted, and the first training sample sequence is extracted from the third sample set. If the second predetermined stop condition is reached, the model parameters of the quantization generation model obtained through training are acquired. This may include the step of sending the model parameters of the quantization generative model obtained through training to an end device.

[0092] Specifically, in the training process of the quantization generative model, predetermined presentation information control can be introduced. That is, by introducing predetermined presentation information to the second decoder side of the quantization generative model, the second decoder reconstructs the task sequence according to the predetermined presentation information and the first test features compressed by the second encoder. Taking the motion generation task as an example, the predetermined presentation information is predetermined trajectory information, and trajectory decoupling control can be realized using the predetermined trajectory information. Furthermore, by randomly selecting the direction of motion, that is, setting different predetermined trajectory information, the quantization generative model can learn the association between motion and trajectory direction, and an intuitive and simple motion control function can be realized at the feature level.

[0093] In the embodiments of this specification, the multiple training sample sequences in the third sample set may be sample sequences corresponding to multimodal sample information, and the multimodal sample information includes at least two of the following: sample text information, sample image information, sample trajectory information, sample audio information, sample video information, etc.

[0094] In actual use, there are multiple methods for obtaining the third sample set. The third sample set may be constructed by artificially inputting a large number of training sample sequences, or by reading a large number of training sample sequences from another data acquisition device or database. The method for obtaining the third sample set is specifically selected according to the actual situation, and the examples in this specification are not limited to this.

[0095] In one possible implementation of this specification, a third predetermined stopping condition includes the quantization loss value being less than or equal to a third predetermined threshold, the third predetermined threshold being specifically selected according to the actual circumstances, and the embodiments of this specification are not limited thereto. A first test feature and predetermined presentation information are input to a second decoder, a first test sequence is obtained, and after obtaining the first test sequence, a quantization loss value is calculated based on the first test sequence and the first training sample sequence, and the quantization loss value is compared with a third predetermined threshold.

[0096] Specifically, if the quantization loss value is greater than a third predetermined threshold, it indicates that the difference between the first test sequence and the first training sample sequence is large, and that the predictive ability of the quantization generative model to the first test features and predetermined presented information is low. In this case, the model parameters of the quantization generative model are adjusted, and the first training sample sequence is extracted from multiple training sample sequences, allowing the quantization generative model to continue training. When the quantization loss value falls below the third predetermined threshold, it indicates that the difference between the first test sequence and the first training sample sequence is small, reaching the third predetermined stopping condition, and a trained quantization generative model is obtained.

[0097] Using the embodiments described herein, a first training sample sequence is input to a second encoder to obtain a first test feature, the first test feature and predetermined presentation information are input to a second decoder to obtain a first test sequence, a quantization loss value is calculated based on the first training sample sequence and the first test sequence, the quantization loss value is compared with a third predetermined threshold, and if it is greater than the third predetermined threshold, the quantization generative model is trained until the quantization loss value is less than or equal to the third predetermined threshold, completing the training of the quantization generative model, and the model parameters of the quantization generative model are continuously adjusted to make the finally obtained quantization generative model more accurate.

[0098] In another possible implementation of this specification, in addition to comparing the magnitude of the quantization loss value with a third predetermined threshold, it is possible to determine whether the current quantization generation model is trained or not, in accordance with the number of iterations.

[0099] Specifically, if the quantization loss value is greater than a third predetermined threshold, the model parameters of the quantization generative model are adjusted, and the first training sample sequence is extracted from multiple training sample sequences. The quantization generative model continues to be trained, and when the third predetermined number of iterations is reached, the iterations are stopped, and the trained quantization generative model is obtained. The third predetermined number of iterations is specifically selected according to the actual situation, and the embodiments herein are not limited thereto.

[0100] Using the embodiments described herein, a first training sample sequence is input to a second encoder to obtain a first test feature, the first test feature and predetermined presentation information are input to a second decoder to obtain a first test sequence, a quantization loss value is calculated based on the first training sample sequence and the first test sequence, the quantization loss value is compared with a third predetermined threshold, and if it is greater than the third predetermined threshold, the quantization generative model is trained until a third predetermined number of iterations is reached to complete the training of the quantization generative model, and the model parameters of the quantization generative model are continuously adjusted to make the finally obtained quantization generative model more accurate.

[0101] In practice, there are many functions for calculating quantization loss values, such as the cross-entropy loss function, L1 norm loss function, maximum loss function, mean squared error loss function, and logarithmic loss function. The specific function is selected according to the actual situation, and the examples in this specification are not limited to these.

[0102] In one selectable embodiment of this specification, in order to ensure temporal continuity in the task sequence, the first test feature and predetermined presentation information are input to the second decoder, and after the above step of obtaining the first test sequence, The steps include dividing the first test sequence at random points in time to obtain the first test subsequence and the second test subsequence, The step of calculating a continuity loss value based on a first test subsequence and a second test subsequence may further include: Based on the quantization loss value, the parameters of the second encoder and second decoder are adjusted, and the first training sample sequence is extracted from the third sample set. If the second predetermined stop condition is reached, the above step of obtaining the model parameters of the quantization generation model obtained through training is performed. The process may include adjusting the parameters of the second encoder and second decoder based on the quantization loss value and the continuity loss value, then returning to extract the first training sample sequence from the third sample set, and, if a second predetermined stop condition is reached, obtaining the model parameters of the trained quantization generation model.

[0103] Taking the motion generation task as an example, in the motion generation process, even if two discrete motion features are joined together, a continuous motion sequence can be generated, resulting in a smooth transition. Therefore, to avoid time-series disruption in the motion generation process and to satisfy the requirement of temporal continuity of the motion sequence, when training a quantized generative model, a single continuous motion sequence is divided into a first test subsequence (seq1) and a second test subsequence (seq2) at random points in time, and a time-series reconstruction consistency loss function for the motion sequence is introduced to calculate the continuity loss value L, which is shown in equation (1) below.

[0104] JPEG0007871496000001.jpg15170 Using the embodiments of this specification, the first test sequence is divided at random points in time to obtain a first test subsequence and a second test subsequence; based on the first and second test subsequences, a continuity loss value is calculated; based on the quantization loss value and the continuity loss value, the parameters of the second encoder and second decoder are adjusted; and the first training sample sequence is extracted from the third sample set. If a second predetermined stop condition is reached, the model parameters of the trained quantization generation model are obtained to ensure that the generated task sequence satisfies the temporal continuity requirement and further improve the accuracy of the task processing model.

[0105] The motion generation methods described in this specification can be applied to different motion generation scenarios, such as generating motion for objects like virtual humans and animals in game scenes, or generating motion for virtual customers in e-commerce scenes. The specific examples will be selected according to the actual situation, and the examples described in this specification are not limited to these.

[0106] The data processing method of the task processing model described herein will be further explained below, with reference to Figure 5, using its application in the field of virtual character video generation as an example. Referring to Figure 5, Figure 5 shows a flowchart of a virtual character video generation method according to one embodiment of this specification, which specifically includes the following steps 502 to 508.

[0107] In step 502, the virtual character video generation request sent by the frontend is received, and the virtual character video generation request includes video guide information.

[0108] In step 504, the video guide information and the entire mask video sequence are input to the virtual character video generation model, and the virtual character video features corresponding to the video guide information are obtained through processing by the virtual character video generation model.

[0109] The virtual character video generation model is obtained by training it based on multiple sample video guide information, sample video sequences, and sample video features. The sample video features are obtained by quantizing and encoding the sample video sequences.

[0110] In step 506, the virtual character video features are quantized and decoded to obtain the virtual character motion sequence corresponding to the video guide information.

[0111] In step 508, a virtual character video is generated based on the virtual character motion sequence and sent to the front end, causing the front end to display the virtual character video.

[0112] Specifically, video guide information refers to information used to guide the generation of virtual character videos. Video guide information includes, but is not limited to, video guide text information, video guide image information, video guide audio information, video guide video information, and video guide trajectory information. Specifically, it is selected according to the actual situation, and the embodiments described herein are not limited to this.

[0113] The implementations of steps 502, 504, and 506 are the same as those of steps 404, 406, and 408 described above, and will not be described in detail in the examples of this specification.

[0114] Furthermore, after obtaining a virtual character motion sequence corresponding to the video guide information, a virtual character video can be generated by controlling the virtual character's skeleton motion based on the time order of the virtual character motion sequence.

[0115] Using the embodiment described herein, a virtual character video generation request is received from the front end, which includes video guide information. The video guide information and the entire mask video sequence are input to the virtual character video generation model. The virtual character video generation model processes the video guide information to obtain virtual character video features corresponding to the video guide information. The virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features. The sample video features are obtained by quantizing and encoding the sample video sequences. The virtual character video features are quantized and decoded to obtain a virtual character motion sequence corresponding to the video guide information. Based on the virtual character motion sequence, a virtual character video is generated and transmitted to the front end, causing the front end to display the virtual character video. Because the virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences, the virtual character video generation model can efficiently and accurately generate virtual character motion sequences, and furthermore, can generate accurate virtual character videos based on the virtual character motion sequences.

[0116] In one selectable embodiment of this specification, before the above step in which video guide information and the entire mask video sequence are input to a virtual character video generation model and virtual character video features corresponding to the video guide information are obtained by processing the virtual character video generation model, A step of receiving specified presentation information entered by a user, the specified presentation information may further include a step of including specified presentation text and / or specified presentation audio, The steps of inputting video guide information and the entire mask video sequence into a virtual character video generation model, and obtaining virtual character video features corresponding to the video guide information through processing by the virtual character video generation model, are as follows: The process may include the step of inputting specified presentation information, video guide information, and the entire mask video sequence into a virtual character video generation model, and obtaining virtual character video features corresponding to the video guide information through processing by the virtual character video generation model.

[0117] Specifically, the designated presentation information is presentation information for video guide information, and the designated presentation information may be in text format, audio format, or a combination of text and audio format, and will be specifically selected according to the actual situation, and the embodiments of this specification are not limited thereto.

[0118] For example, let's assume that the video guide information is a video guide image, and the specified presentation information may be text that presents the video guide image, such as "The character in the video guide image is dancing."

[0119] Using the embodiments described herein, user-inputted specified presentation information is received, which includes specified presentation text and / or specified presentation audio. The specified presentation information, video guide information, and the entire mask video sequence are input into a virtual character video generation model. The virtual character video generation model processes the information to obtain virtual character video features corresponding to the video guide information and improve the accuracy of the virtual character video features.

[0120] In one selectable embodiment of this specification, after the above step of quantizing and decoding virtual character video features and obtaining a virtual character motion sequence corresponding to video guide information, A step to receive scene information for the current scene entered by the user, The method may further include the steps of adjusting the virtual character motion sequence using a virtual character video generation model based on scene information, and obtaining the adjusted virtual character motion sequence.

[0121] Specifically, scene information refers to video scenes in a virtual character video, including background information, weather information, geographical information, etc., and is selected according to the actual situation; the embodiments described herein are not limited to this.

[0122] Furthermore, when adjusting the virtual character motion sequence using a virtual character video generation model based on scene information, the scene information and video guide information are input to the encoder of the virtual character video generation model to obtain scene encoding features and video guide features. These are then input to the decoder of the virtual character video generation model to obtain the adjusted virtual character motion sequence. Finally, the adjusted virtual character video features are quantized and decoded to obtain the adjusted virtual character motion sequence.

[0123] Using the embodiments described herein, the system receives current scene information input by the user, adjusts the virtual character motion sequence using a virtual character video generation model based on the scene information, and obtains the adjusted virtual character motion sequence. By adjusting the virtual character motion sequence based on the current scene information, the virtual character motion sequence is made more realistic, further improving the realism of the virtual character video.

[0124] In one selectable embodiment of this specification, virtual character video features are input to a second decoder of a pre-trained quantization generation model, and the decoding process of the second decoder can obtain a virtual character motion sequence corresponding to video guide information. Furthermore, in order to generate a more accurate virtual character motion sequence that meets the user's needs, the user inputs target trajectory information, uses the target trajectory information to control the direction of motion generation, that is, the above steps of quantizing and decoding virtual character video features and obtaining a virtual character motion sequence corresponding to video guide information are performed as follows: The steps include receiving target trajectory information entered by the user, The process may include the steps of inputting virtual character video features and target trajectory information into a second decoder of a quantization generation model, and obtaining a virtual character motion sequence corresponding to the video guide information by decoding the second decoder, wherein the quantization generation model is trained based on a plurality of training sample sequences.

[0125] Furthermore, target trajectory information can be represented by the relative displacement of two adjacent frames. The quantization generative model learns an encoding dictionary (codebook) such as a text dictionary, converts a continuous motion sequence into discrete motion features, and can control the direction of motion generation by trajectory decoupling with different trajectory information inputs. The virtual character motion sequence includes motion-related parameters, and the virtual character can be driven by the virtual character motion sequence. Taking a T×N motion sequence as an example, T is the number of motion frames, N is the number of motion skeletons, and each skeleton in each frame is represented by a 6D rotation, meaning the motion sequence is represented in dimensions of T×N×6.

[0126] Using the embodiment described herein, user-inputted target trajectory information is received, virtual character video features and target trajectory information are input to a second decoder of a quantization generation model, and the decoding process of the second decoder obtains a virtual character motion sequence corresponding to the video guide information. The quantization generation model is obtained by training on multiple training sample sequences. Since the motion trajectory of the motion often relates to the orientation of the virtual object, performing data augmentation by simultaneously rotating the orientation of the virtual object and the trajectory allows the model to learn relevant information such as orientation in the motion from the trajectory information, ensuring the generation of a virtual character motion sequence that meets the user's needs and improving the accuracy of the virtual character video.

[0127] Referring to Figure 6, Figure 6 shows a schematic diagram of the processing process of a virtual character video generation method according to one embodiment of this specification. As shown in Figure 6, the user inputs video guide information, which includes one of various combinations of motion trajectory (3D displacement), motion type (text), partial motion (6D display and 3D position of keyframe motion), text description, image, image, text description, and trajectory. The user inputs each piece of video guide information into a virtual character video generation model, and the virtual character video generation model processes to obtain virtual character video features corresponding to the video guide information. The virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features. The sample video features are obtained by quantizing and encoding sample video sequences. The virtual character video features are quantized and decoded to obtain virtual character motion sequences corresponding to the video guide information.

[0128] By utilizing the embodiments described herein, it is possible to integrate multiple multimodal virtual character video generation tasks and improve the accuracy of virtual character motion sequences by converting video guide information from different modals into an integrated format and simultaneously accepting integrated input of different information during the learning process.

[0129] Referring to Figure 7, Figure 7 shows a schematic diagram of a virtual character video generation interface according to one embodiment of this specification.

[0130] The virtual character video generation interface includes a video guide information upload box, a "Confirm" control, a "Cancel" control, and a virtual character video display box. When a user uploads video guide information using the video guide information upload box displayed on the frontend and clicks the "Confirm" control, the server inputs the video guide information into the virtual character video generation model. The virtual character video generation model then processes the video guide information to obtain a virtual character action sequence corresponding to it. Based on the virtual character action sequence, it generates a virtual character video and sends it to the frontend, where it displays the virtual character video corresponding to the video guide information in the virtual character video display box.

[0131] The methods by which the user operates the controls include any one of the following: click, double-click, touch, mouse hover, slide, long press, voice control, or shake, and will be specifically selected depending on the actual situation. The embodiments described herein are not limited to these methods.

[0132] Referring to Figure 8, Figure 8 shows a flowchart of a data processing method for a quantization generative model according to one embodiment of this specification, the quantization generative model comprising a second encoder and a second decoder, and the data processing method for the quantization generative model is applied to a cloud device and specifically includes the following steps 802 to 810.

[0133] In step 802, a third sample set is obtained, which includes multiple training sample sequences.

[0134] In step 804, the training sample sequence is input to the second encoder to acquire test behavioral features.

[0135] In step 806, the test operation characteristics and predetermined trajectory information are input to the second decoder to obtain the test operation sequence.

[0136] In step 808, the quantization generative model is trained based on the training sample sequence and the test operation sequence. If the third predetermined stop condition is reached, the model parameters of the quantization generative model obtained through training are acquired.

[0137] In step 810, the model parameters of the quantization generation model obtained through training are transmitted to the end device.

[0138] The specific implementations of steps 802 to 808 are the same as the implementations of data processing in the task processing model shown in Figure 4, and will not be described in detail in the examples of this specification.

[0139] In actual use, the cloud device sends the model parameters of the quantization generative model to the end device, which can then construct the quantization generative model based on those parameters, thereby enabling behavior generation on the end side.

[0140] Furthermore, because the quantization generative model is small, the data processing method of the above quantization generative model can be performed by an end device.

[0141] Using the embodiment described herein, the cloud device acquires a third sample set, which includes multiple training sample sequences, inputs the training sample sequences to a second encoder to acquire test operation features, inputs the test operation features and predetermined trajectory information to a second decoder to acquire a test operation sequence, trains a quantization generation model based on the training sample sequences and test operation sequences, and when a third predetermined stop condition is reached, acquires the model parameters of the trained quantization generation model and transmits the model parameters of the trained quantization generation model to the end device. By continuously adjusting the model parameters of the quantization generation model, the model parameters of the final quantization generation model are made more accurate.

[0142] Referring to Figure 9, which shows a flowchart of the training of a task processing model and a quantization generation model according to one embodiment of this specification, taking the motion generation task as an example, the task processing model is a motion generation model, and specifically includes the following:

[0143] For training a quantization generative model, the network learns one codebook. The network first converts a multimodal training sample sequence into several test action features (embeddings), the number of which is less than the original number of frames multiplied by the number of keypoints. Next, each test action feature is discretely placed in the latest embedding of the codebook according to its similarity. Finally, the network decodes the embedding and predetermined trajectory information corresponding to the codebook to obtain the test action sequence. The entire process is optimized based on the error between the training sample sequence and the test action sequence, and the distance between the latest embedding in the codebook and the test action feature.

[0144] Furthermore, the second encoder of the quantization generative model is used only during the training phase, and the output of the second encoder is the label of the output of the first decoder of the behavior generative model.

[0145] For training the motion generation model, multimodal sample guide information is input to the first encoder of the motion generation model, and sample guide features corresponding to the sample guide information are obtained. The original motion sequence is masked, and the sample guide features and the original motion sequence after masking are input to the first decoder of the motion generation model, and a decoded motion sequence is obtained. The decoded motion sequence is further discretized to obtain predicted motion features, and the first decoder is optimized based on the error between the predicted motion features and the motion features of the samples corresponding to the original motion sequence, and the motion generation model is obtained. The first encoder and the first decoder are connected via a cross-attention layer.

[0146] The training process for the motion generation model includes degeneration and recovery processes. The degeneration process uses a masking method, training with multimodal sample guide information. If there is a missing value in the sample guide information of a modal, the corresponding position is replaced with a mask token to perform a prediction mask. Each time, a different step is randomized, corresponding to a different mask probability distribution, and masking is performed according to the corresponding mask probability. Next, the model is trained to predict the original motion sequence, and then additional mask heads are placed, and the model prediction results are re-inputted into the network to have the model predict the masked positions. In the testing phase, various motion generation tasks are completed by adjusting different input conditions. Initially, a sequence with all masks is input, and each prediction is masked based on the mask head structure against the code head prediction result. The prediction is repeated, and the final prediction result is output. Specifically, the iterative process involves first inputting the original operation sequence of all mask tokens into the first decoder, predicting it to obtain the complete sequence features of the first step, then using the second decoder to obtain a test operation sequence from the complete sequence features, inputting the test operation sequence into the first decoder to predict the mask position, performing the mask, and then inputting the complete sequence features of the first step after masking and the timestamp of the second step into the first decoder to re-predict (i.e., recover), and repeating this process until the iteration is complete.

[0147] In actual use, the motion generation model converts the sequence into an embedding position in the codebook, that is, predicts the discretized motion sequence, and when discretizing the decoded motion sequence, specifically, it obtains predicted motion features by directly discretizing the motion distances of the X and Y axes included in the motion of each step. For example, the range of 0 to 1 meter is divided into 200 intervals, and 0 to 0.005 meters are all used as the first predicted motion feature, and the motion feature prediction by discrete motion corresponds to the learned codebook.

[0148] By utilizing the methods of the embodiments described herein, a method is used to convert operation sequences into discrete operation feature sequences, thereby enabling the representation of different control signals as seq2seq. By replacing the differences in different operation guide information with mask tokens, the relationship with operation sequences is learned from multimodal signals. Furthermore, by converting different modal operation guide information into an integrated representation, an improvement in data volume is ensured, model scaling up is supported, and multitasking avoids model overfitting. The iterative mask modeling method learns multiple mask degradation and recovery processes at the discrete code level, predicts mask positions using a network for sampling, and ensures training-test consistency in generative tasks compared to methods that add random noise to a diffusion model. Cross-modal learning allows the model to exhibit a certain degree of migration and zero-shot learning, enabling collaborative control of multimodal input information.

[0149] Furthermore, the information and data related to the embodiment of the above method, such as sample guide information, sample task sequence, training sample sequence, predetermined presentation information, video guide information, full mask video sequence, specified presentation information, scene information, and target trajectory information, are all information and data authorized by the user or fully authorized by each party, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entries are provided for the user to authorize or reject.

[0150] This specification further provides embodiments of a data processing device for a task processing model, corresponding to embodiments of the data processing method for the task processing model executed by a cloud device, and Figure 10 shows a schematic diagram of the structure of a data processing device for a task processing model according to one embodiment of this specification, the device is applied to a cloud device connected to multiple cloud devices, and as shown in Figure 10, the device is An acquisition module configured to acquire a first sample set, wherein the first sample set includes an acquisition module 1002 containing multimodal sample guide information, A first input module 1004 is configured to input sample guide information and sample task sequences into an initial processing model and to acquire predicted task features corresponding to the sample guide information, A training module 1006 is configured to train an initial processing model based on prediction task features and sample task features corresponding to a sample task sequence, and to acquire the model parameters of the training processing model when a first predetermined stopping condition is reached, wherein the sample task features are obtained by quantizing and encoding a sample task sequence. The system includes a transmission module 1008 configured to send model parameters of a task processing model obtained through training to an end device.

[0151] Selectively, the initial processing model includes a first encoder and a first decoder, the first predetermined stop condition includes a first stop subcondition, the first input module 1004 is configured to further acquire first sample guide information from a first sample set, the first sample guide information being any one of the sample guide information in the first sample set, input the first sample guide information to a pre-trained first encoder to acquire first sample guide features, mask the sample task sequence to acquire the mask task sequence, input the first sample guide features and the mask task sequence to a first decoder to acquire first prediction task features, the training module 1006 is configured to further calculate a decoding loss value based on the first prediction task features and the sample task features corresponding to the sample task sequence, adjust the parameters of the first decoder based on the decoding loss value, and then perform the steps of extracting first sample guide information from the first sample set, and if the first stop subcondition is reached, acquire a trained first decoder.

[0152] Selectively, the first predetermined stop condition includes a second stop subcondition, and the first input module 1004 is configured to further acquire a second sample set, the second sample set includes multimodal sample guide information, the sample guide information has sample guide features, extract the second sample guide information from the second sample set, the second sample guide information is any one of the sample guide information in the second sample set, input the second sample guide information to the first encoder, acquire a first prediction guide feature corresponding to the second sample guide information, calculate an encoding loss value based on the first prediction guide feature and the second sample guide feature included in the second sample guide information, adjust the parameters of the first encoder based on the encoding loss value, and then return to extract the second sample guide information from the second sample set, and if the second stop subcondition is reached, acquire the trained first encoder.

[0153] Optionally, the first input module 1004 is configured to further input a sample task sequence to a second encoder of a pre-trained quantization generative model, and to obtain sample task features corresponding to the sample task sequence through the encoding process of the second encoder.

[0154] Optionally, the first input module 1004 is configured to acquire a pre-set time step feature, with a one-to-one correspondence between the time step feature and the number of training iterations, and to input the time step feature, the first sample guide feature, and the mask task sequence to the first decoder to acquire the first prediction task feature.

[0155] Optionally, the quantization generative model includes a second encoder and a second decoder, and the device further includes a quantization generative model training module, which is configured to acquire a third sample set, the third sample set includes multiple training sample sequences, extract a first training sample sequence from the third sample set, the first training sample sequence being any one of the training sample sequences in the third sample set, input the first training sample sequence to the second encoder, acquire a first test feature, input the first test feature and predetermined presentation information to the second decoder, acquire a first test sequence, calculate a quantization loss value based on the first training sample sequence and the first test sequence, adjust the parameters of the second encoder and the second decoder based on the quantization loss value, and return to extract the first training sample sequence from the third sample set, and if a second predetermined stop condition is reached, acquire the model parameters of the trained quantization generative model and transmit the model parameters of the trained quantization generative model to the end device.

[0156] Optionally, the quantization generative model training module is configured to further perform the steps of splitting the first test sequence at random points in time to obtain the first and second test subsequences, calculating the continuity loss value based on the first and second test subsequences, adjusting the parameters of the second encoder and second decoder based on the quantization loss value and the continuity loss value, and then extracting the first training sample sequence from the third sample set, and if a second predetermined stop condition is reached, to obtain the model parameters of the quantization generative model obtained by training.

[0157] Using the embodiment described herein, a first sample set is obtained, which includes multimodal sample guide information. The sample guide information and sample task sequences are input to an initial processing model. Predicted task features corresponding to the sample guide information are obtained. The initial processing model is trained based on the predicted task features and sample task features corresponding to the sample task sequences. When a first predetermined stop condition is reached, the model parameters of the trained processing model are obtained. The sample task features are obtained by quantizing and encoding the sample task sequences. The model parameters of the trained task processing model are transmitted to the end device. Since the task processing model is trained based on multimodal sample guide information, multimodal task integration can be achieved, improving the accuracy and versatility of the model.

[0158] The above is a schematic solution for the data processing device of the task processing model in this embodiment. The technical solution for the data processing device of the task processing model belongs to the same concept as the technical solution for the data processing method of the task processing model executed by the cloud device, and details not explained in detail in the technical solution for the data processing device of the task processing model can be found by referring to the explanation of the technical solution for the data processing method of the task processing model executed by the cloud device.

[0159] This specification further provides embodiments of a data processing device for a task processing model, corresponding to embodiments of the data processing method for the task processing model executed by an end device, and Figure 11 shows a schematic diagram of the structure of another data processing device for a task processing model according to one embodiment of this specification, which is executed by an end device connected to a cloud device. As shown in Figure 11, the device is A build module 1102 is configured to receive model parameters of a task processing model sent by a cloud device and to build a task processing model based on those model parameters, A first receiving module configured to receive a task processing request entered by a user, wherein the task processing request includes 1104 containing task guide information, A second input module 1106 is configured to input task guide information and the entire mask task sequence into a task processing model, and to obtain target task features corresponding to the task guide information through processing by the task processing model, wherein the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, and the sample task features are obtained by quantizing and encoding the sample task sequences. The system includes a first decoding module 1108 configured to quantize and decode target task features and obtain task processing results corresponding to task guide information.

[0160] Selectively, the task processing model includes a first encoder and a first decoder, and the second input module 1106 is further configured to input task guide information to the first encoder, obtain task guide functions corresponding to the task guide information, input task guide features and the entire mask task sequence to the first decoder, and obtain target task features corresponding to the task guide information.

[0161] Using the embodiment described herein, a cloud device sends model parameters of a task processing model, a task processing model is constructed based on the model parameters, a user inputs a task processing request, the request includes task guide information, the task guide information and the entire mask task sequence are input to the task processing model, the task processing model processes to obtain target task features corresponding to the task guide information, the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, the sample task features are obtained by quantizing and encoding the sample task sequences, the target task features are quantized and decoded to obtain task processing results corresponding to the task guide information. Since the task processing model is obtained by training on multimodal sample guide information, multimodal task integration can be realized, and therefore the task processing model efficiently and accurately generates target task features, further improving the accuracy of the task processing results.

[0162] The above is an exemplary solution for the data processing device of the task processing model of this embodiment. The technical solution for the data processing device of the task processing model belongs to the same concept as the technical solution for the data processing method of the task processing model executed by the end device, and details not explained in detail in the technical solution for the data processing device of the task processing model can be found in the explanation of the technical solution for the data processing method of the task processing model executed by the end device.

[0163] This specification further provides embodiments of a virtual character video generation apparatus corresponding to the embodiments of the virtual character video generation method described above, and Figure 12 shows a schematic diagram illustrating the structure of a virtual character video generation apparatus according to one embodiment of this specification. As shown in Figure 12, the apparatus is A second receiving module configured to receive a virtual character video generation request sent by the front end, wherein the virtual character video generation request includes video guide information, and the second receiving module 1202 A third input module 1204 is configured to input video guide information and the entire mask video sequence into a virtual character video generation model, and to obtain virtual character video features corresponding to the video guide information through processing by the virtual character video generation model, wherein the virtual character video generation model is obtained by training on a plurality of sample video guide information, sample video sequences and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences. A first decoding module 1206 is configured to quantize and decode virtual character video features and obtain virtual character motion sequences corresponding to video guide information, The system includes a generation module 1208 configured to generate a virtual character video based on a virtual character motion sequence and send it to the front end, thereby causing the front end to display the virtual character video.

[0164] Selectively, the device further includes a fourth input module configured to receive user-entered specified presentation information, the specified presentation information comprising specified presentation text and / or specified presentation audio, and a third input module 1204 further configured to input the specified presentation information, video guide information, and the entire mask video sequence into a virtual character video generation model, and to obtain virtual character video features corresponding to the video guide information through processing by the virtual character video generation model.

[0165] Optionally, the device further includes an adjustment module configured to receive scene information of the current scene input by a user, and to adjust a virtual character motion sequence using a virtual character video generation model based on the scene information, and to obtain the adjusted virtual character motion sequence.

[0166] Optionally, the first decoding module 1206 is configured to receive user-inputted target trajectory information, input virtual character video features and target trajectory information to the second decoder of the quantization generative model, and obtain a virtual character motion sequence corresponding to the video guide information through the decoding process of the second decoder, the quantization generative model is obtained by training on multiple training sample sequences.

[0167] Using the embodiment described herein, a virtual character video generation request is received from the front end, which includes video guide information. The video guide information and the entire mask video sequence are input to the virtual character video generation model. The virtual character video generation model processes the video guide information to obtain virtual character video features corresponding to the video guide information. The virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features. The sample video features are obtained by quantizing and encoding the sample video sequences. The virtual character video features are quantized and decoded to obtain a virtual character motion sequence corresponding to the video guide information. Based on the virtual character motion sequence, a virtual character video is generated and transmitted to the front end, causing the front end to display the virtual character video. Because the virtual character video generation model is obtained by training on multiple sample video guide information, sample video sequences, and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences, the virtual character video generation model can efficiently and accurately generate virtual character motion sequences, and furthermore, can generate accurate virtual character videos based on the virtual character motion sequences.

[0168] The above is an exemplary solution for the virtual character video generation device of this embodiment. The technical solution for the virtual character video generation device belongs to the same concept as the technical solution for the virtual character video generation method described above, and details not explained in detail in the technical solution for the virtual character video generation device can all be found by referring to the explanation of the technical solution for the virtual character video generation method described above.

[0169] Figure 13 shows a block diagram of a computing device according to one embodiment of this specification. The components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 and the memory 1310 are connected via a bus 1330, and a database 1350 stores data.

[0170] The computing device 1300 further includes an access device 1340, which enables the computing device 1300 to communicate over one or more networks 1360. Examples of these networks include combinations of communication networks such as Public Switched Telephone Networks (PSTN), Local Area Networks (LAN), Wide Area Networks (WAN), Personal Area Networks (PAN), or the Internet. The access device 1340 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface (Bluetooth is a registered trademark), or a Near Field Communication (NFC) interface.

[0171] In one embodiment of this specification, the above-mentioned components of the computing device 1300, as well as other components not shown in Figure 13, can be connected to one another, for example, via a bus. It should be understood that the block diagram of the structure of the computing device shown in Figure 13 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0172] The computing device 1300 may be any type of fixed or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.), or other types of mobile devices, or fixed computing devices such as desktop computers or personal computers (PCs). The computer device 1300 may also be a mobile or fixed server.

[0173] Of these, processor 1320 is used to execute computer executable instructions that realize the steps of the data processing method or virtual character video generation method of the task processing model described above when executed by the processor.

[0174] The above is an exemplary solution for the computing device of this embodiment. The technical solution for this computing device belongs to the same concept as the technical solution for the data processing method and virtual character video generation method of the task processing model described above. Details not explained in detail in the technical solution for the computing device can be found by referring to the explanation of the technical solution for the data processing method or virtual character video generation method of the task processing model described above.

[0175] One embodiment of this specification further provides a computer-readable storage medium that stores computer-executable instructions that, when executed by a processor, realize steps of the data processing method or virtual character video generation method of the task processing model described above.

[0176] The above is an exemplary solution for a computer-readable storage medium in this embodiment. The technical solution for this storage medium belongs to the same concept as the technical means of the data processing method and virtual character video generation method of the task processing model described above. Details not explained in detail in the technical solution for the storage medium can be found by referring to the explanation of the technical means of the data processing method or virtual character video generation method of the task processing model described above.

[0177] Furthermore, one embodiment of this specification further provides a computer program that, when executed on a computer, causes the computer to perform steps of the data processing method or virtual character video generation method of the task processing model described above.

[0178] The above is an exemplary solution for the computer program of this embodiment. The technical solutions of this computer program belong to the same concept as the technical solutions of the data processing method and virtual character video generation method of the task processing model described above. Details not explained in detail in the technical solutions of the computer program can be found by referring to the explanation of the technical solutions of the data processing method or virtual character video generation method of the task processing model described above.

[0179] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the operations or steps described in the claims may be performed in a different order than those described in the embodiments, and the desired results may still be achieved. Furthermore, the processes shown in the drawings do not necessarily require that the desired results cannot be achieved by not following the specific order or sequence shown. In some embodiments, multitasking and parallel processing may also be possible or advantageous.

[0180] The computer instructions include computer program code, which may be in source code format, object code format, executable file format, or some intermediate format. The computer-readable medium may include any entity or device capable of having the computer program code, recording media, U disks, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media.

[0181] Furthermore, while the embodiments of each method described above have all been described as a series of operations for the sake of simplicity, those skilled in the art will understand that the embodiments herein are not limited by the order of operations described, and that according to the embodiments herein, certain steps can be performed in other orders or simultaneously. Next, those skilled in the art will understand that all embodiments described herein are preferred embodiments, and that the operations and modules mentioned are not necessarily required for the embodiments herein.

[0182] In the above embodiments, each embodiment has its own emphasis, and for parts not described in detail in one embodiment, you can refer to the relevant descriptions in other embodiments.

[0183] The preferred embodiments disclosed herein are provided solely to aid in the explanation herein. Selected embodiments are not exhaustive and do not limit the invention to the specified embodiments. Clearly, various modifications and variations are possible according to the embodiments herein. These embodiments have been selected and described in detail to better illustrate the principles and practical applications of the embodiments herein, thereby enabling those skilled in the art to fully understand and utilize this specification. This specification is limited only by the claims and their entirety and equivalents.

Claims

1. A data processing method for a task processing model executed by a cloud device connected to multiple end devices, A step of obtaining a first sample set, wherein the first sample set includes multimodal sample guide information, The steps include inputting the aforementioned sample guide information and sample task sequence into an initial processing model and obtaining predictive task features corresponding to the sample guide information, The steps include training the initial processing model based on the prediction task features and the sample task features corresponding to the sample task sequence, and, when a first predetermined stop condition is reached, obtaining the model parameters of the training processing model, wherein the sample task features are obtained by quantizing and encoding the sample task sequence; The steps include sending the model parameters of the task processing model obtained through the training to an end device, including, Data processing method for task processing models.

2. The initial processing model includes a first encoder and a first decoder, and the first predetermined stop condition includes a first stop subcondition. The step of inputting the aforementioned sample guide information and sample task sequence into an initial processing model and obtaining predictive task features corresponding to the sample guide information is: A step of obtaining first sample guide information from the first sample set, wherein the first sample guide information is any one of the sample guide information in the first sample set, The first sample guide information is input to a pre-trained first encoder to obtain the first sample guide features, The steps include: masking the aforementioned sample task sequence and obtaining a masked task sequence; The steps include inputting the first sample guide feature and the mask task sequence into the first decoder to obtain the first prediction task feature, Includes, The step of training the initial processing model based on the prediction task features and the sample task features corresponding to the sample task sequence, and obtaining the model parameters of the training processing model when the first predetermined stop condition is reached, A step of calculating a decoding loss value based on the first prediction task features and the sample task features corresponding to the sample task sequence, Based on the decoding loss value, the steps include adjusting the parameters of the first decoder, returning to extract the first sample guide information from the first sample set, and if the first stop subcondition is reached, obtaining the trained first decoder. including, The method according to claim 1.

3. The first predetermined stop condition includes a second stop subcondition, The training method for the first encoder is: A step of obtaining a second sample set, wherein the second sample set includes multimodal sample guide information, and the sample guide information includes sample guide features, A step of extracting second sample guide information from the second sample set, wherein the second sample guide information is any one of the sample guide information in the second sample set, The steps include inputting the second sample guide information to the first encoder and obtaining a first predictive guide feature corresponding to the second sample guide information, A step of calculating the coding loss value based on the first prediction guide feature and the second sample guide feature included in the second sample guide information, Based on the encoding loss value, the parameters of the first encoder are adjusted, and the steps of going back and extracting the second sample guide information from the second sample set are performed, and if the second stop subcondition is reached, the trained first encoder is obtained. including, The method according to claim 2.

4. Before the step of calculating the decoding loss value based on the first prediction task features and the sample task features corresponding to the sample task sequence, The further step includes inputting a sample task sequence into a second encoder of a pre-trained quantization generative model, and obtaining sample task features corresponding to the sample task sequence by encoding the second encoder. The method according to claim 2.

5. The step of inputting the first sample guide feature and the mask task sequence into the first decoder and obtaining the first prediction task feature is: A step of acquiring a pre-set time step feature, wherein the time step feature and the number of training iterations correspond one-to-one, The steps include inputting the time step feature, the first sample guide feature, and the mask task sequence to the first decoder to obtain the first prediction task feature, including, The method according to claim 2.

6. The quantization generation model includes a second encoder and a second decoder, The training method for the aforementioned quantization generation model is: A step of obtaining a third sample set, wherein the third sample set includes a plurality of training sample sequences, A step of extracting a first training sample sequence from the third sample set, wherein the first training sample sequence is any one of the training sample sequences from the third sample set, The steps include inputting the first training sample sequence to the second encoder and obtaining the first test feature, The steps include inputting the first test feature and predetermined presentation information into the second decoder to obtain the first test sequence, A step of calculating the quantization loss value based on the first training sample sequence and the first test sequence, Based on the quantization loss value, the parameters of the second encoder and the second decoder are adjusted, and the first training sample sequence is extracted from the third sample set. If the second predetermined stop condition is reached, the model parameters of the quantization generation model obtained by training are acquired. The steps include transmitting the model parameters of the quantization generation model obtained through the training to an end device, including, The method according to claim 4.

7. After the step of inputting the first test feature and predetermined presentation information into the second decoder and obtaining the first test sequence, The steps include dividing the first test sequence at random points in time to obtain a first test subsequence and a second test subsequence, A step of calculating a continuity loss value based on the first test subsequence and the second test subsequence, It further includes, Based on the quantization loss value, the parameters of the second encoder and the second decoder are adjusted, and the step of extracting the first training sample sequence from the third sample set is performed. If the second predetermined stop condition is reached, the step of obtaining the model parameters of the quantization generation model obtained by training is performed. The process includes adjusting the parameters of the second encoder and the second decoder based on the quantization loss value and the continuity loss value, and then performing the step of extracting the first training sample sequence from the third sample set, and if a second predetermined stop condition is reached, obtaining the model parameters of the quantization generation model obtained by training, The method according to claim 6.

8. A data processing method for a task processing model executed by an end device connected to a cloud device, The steps include receiving model parameters of a task processing model sent by a cloud device and constructing a task processing model based on said model parameters, A step of receiving a task processing request entered by a user, wherein the task processing request includes task guide information, A step comprising inputting the task guide information and the entire mask task sequence into the task processing model, and obtaining target task features corresponding to the task guide information through processing by the task processing model, wherein the task processing model is obtained by training on multimodal sample guide information, sample task sequences, and sample task features, and the sample task features are obtained by quantizing and encoding the sample task sequences. The steps include: quantizing and decoding the target task features and obtaining task processing results corresponding to the task guide information; including, Data processing method for task processing models.

9. The task processing model includes a first encoder and a first decoder, The step of inputting the task guide information and the entire mask task sequence into a task processing model, and obtaining target task features corresponding to the task guide information through processing by the task processing model, The steps include inputting the task guide information into the first encoder and obtaining task guide features corresponding to the task guide information, The steps include inputting the task guide features and the entire mask task sequence into the first decoder and obtaining target task features corresponding to the task guide information, including, The method according to claim 8.

10. The step of receiving a virtual character video generation request sent by the front end, wherein the virtual character video generation request includes video guide information, The steps include: inputting the aforementioned video guide information and the entire mask video sequence into a virtual character video generation model; and obtaining virtual character video features corresponding to the video guide information through processing by the virtual character video generation model, wherein the virtual character video generation model is obtained by training on a plurality of sample video guide information, sample video sequences, and sample video features, and the sample video features are obtained by quantizing and encoding the sample video sequences; The steps include: quantizing and decoding the virtual character video features and obtaining a virtual character motion sequence corresponding to the video guide information; The steps include generating a virtual character video based on the virtual character operation sequence, sending it to the front end, and causing the front end to display the virtual character video, including, A method for generating virtual character videos.

11. Before the step in which the aforementioned video guide information and the entire mask video sequence are input to a virtual character video generation model, and the virtual character video features corresponding to the video guide information are obtained by processing the aforementioned character video generation model, The step of receiving specified presentation information entered by a user, further comprising the step of the specified presentation information including specified presentation text and / or specified presentation audio, The step of inputting the aforementioned video guide information and the entire mask video sequence into a virtual character video generation model, and obtaining virtual character video features corresponding to the video guide information through processing by the aforementioned character video generation model, The process includes inputting the specified presentation information, video guide information, and the entire mask video sequence into a virtual character video generation model, and obtaining virtual character video features corresponding to the video guide information through processing by the virtual character video generation model. The method according to claim 10.

12. After the step of quantizing and decoding the virtual character video features and obtaining a virtual character motion sequence corresponding to the video guide information, A step to receive scene information for the current scene entered by the user, Based on the scene information, the virtual character motion sequence is adjusted using the virtual character video generation model, and the adjusted virtual character motion sequence is obtained. Further including, The method according to claim 10 or 11.

13. The step of quantizing and decoding the virtual character video features and obtaining a virtual character motion sequence corresponding to the video guide information is: The steps include receiving target trajectory information entered by the user, The steps include inputting the virtual character video features and the target trajectory information into a second decoder of a quantization generation model, and obtaining a virtual character motion sequence corresponding to the video guide information by decoding the second decoder, wherein the quantization generation model is obtained by training based on a plurality of training sample sequences, including, The method according to claim 10.

14. An end device for constructing a first sample set and transmitting the first sample set to a cloud device, wherein the first sample set includes multimodal sample guide information. A cloud device that inputs the sample guide information and sample task sequence into an initial processing model, acquires predicted task features corresponding to the sample guide information, trains the initial processing model based on the predicted task features and sample task features corresponding to the sample task sequence, and acquires the model parameters of the trained processing model when a first predetermined stop condition is reached, wherein the sample task features are obtained by quantizing and encoding the sample task sequence, and the cloud device transmits the model parameters of the trained task processing model to an end device. including, A data processing system based on a task processing model.

15. Including memory and processor, The aforementioned memory is used to store computer executable instructions. The processor is used to execute the computer executable instructions, The computer executable instruction, when executed by the processor, implements a step of the method according to any one of claims 1 to 7, or any one of claims 8 to 9, or claim 10. Computing device.

Images