Data processing method, electronic device, storage medium and computer program product

Abstract

The invention discloses a data processing method, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining a token sequence which is obtained through the conversion of skeleton animation data of a first digital human; dividing tokens in the token sequence into a plurality of clusters; predicting a cluster center of each cluster in the plurality of clusters through an expert network in the multi-expert network model to obtain a first prediction value of each cluster center; and for each token in the token sequence, an estimated value of each token is obtained based on the first predicted value of the clustering center corresponding to each token and the compensation value of each token, the compensation value is used for compensating the difference between each token and the clustering center, and the estimated value of each token is used for generating the action of the first digital human. According to the invention, the searching and positioning requirements of a user in an emergency can be met, and the sensing and positioning capabilities of the terminal are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, a Mixture-of-Experts (MoE) model can be used for digital human motion generation. The MoE model consists of multiple expert networks, each an independent sub-network. During digital human motion generation, the expert networks process the input token to generate the digital human's actions. Currently, transmitting the input token to the GPU where the corresponding expert network resides for processing is time-consuming, resulting in low efficiency in digital human motion generation. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide a data processing method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, the method comprising: Obtain the token sequence, which is derived from the skeletal animation data of the first digital human; Divide the tokens in the token sequence into multiple clusters; The expert network in the multi-expert network model predicts the cluster center of each of the multiple clusters, and obtains the first predicted value of each cluster center. For each token in the token sequence, an estimated value for each token is obtained based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. The compensation value is used to compensate for the difference between each token and the cluster center, and the estimated value of each token is used to generate the action of the first digital human.

[0005] In the above scheme, before obtaining the estimated value of each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token, the method further includes: The compensation value for each token is determined based on neighborhood information, which is the association information of each data point in one or more first data points. The distance between one or more data points in the skeletal animation data and the first data point is less than a first threshold.

[0006] In the above scheme, the neighborhood information includes one or more of the following: Information used to indicate changes in the motion state of the first digital human; Information used to indicate the state of the second digital human, wherein the difference between the feature vector of the first digital human and the feature vector of the second digital human is less than a second threshold; Information used to indicate changes in the environment in which the first digital human is located; Human-computer interaction information.

[0007] In the above scheme, the compensation value for each token is determined based on neighborhood information, including: For each token, a compensation value is determined based on the neighborhood information and the first vector corresponding to each token. The first vector is obtained by randomly projecting each token.

[0008] In the above scheme, determining the compensation value for each token based on the neighborhood information and the first vector corresponding to each token includes: Based on the neighborhood information and the first vector, determine the K nearest neighbors of each token from the cluster in which each token belongs, where K is a positive integer; The compensation value for each token is calculated based on the K nearest neighbors.

[0009] In the above scheme, the token sequence is divided into multiple clusters, including: The token sequence is divided into multiple clusters using locality-sensitive hashing based on random projection.

[0010] The above scheme further includes the following method: Based on the first motion library, the estimated value of each token is processed by the multi-expert network model to generate the first motion of the first digital human, which is different from the motion of the first digital human in the skeletal animation data.

[0011] This application also provides a data processing apparatus, including: An acquisition unit is used to acquire a token sequence, which is obtained by converting the skeletal animation data of the first digital human. A partitioning unit is used to divide the tokens in the token sequence into multiple clusters; The prediction unit is used to predict the cluster center of each of the multiple clusters through the expert network in the multi-expert network model, and obtain the first predicted value of each cluster center. The processing unit is configured to, for each token in the token sequence, obtain an estimated value for each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token, wherein the compensation value is used to compensate for the difference between each token and the cluster center, and the estimated value of each token is used to generate the action of the first digital human.

[0012] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor. The processor is used to execute the steps of any of the above-mentioned technical solutions when running the computer program.

[0013] This application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0015] The embodiments of this application have the following beneficial effects: In this embodiment, the token sequence is obtained by converting the skeletal animation data of the first digital human. The token sequence is acquired, and then the tokens in the sequence are divided into multiple clusters, i.e., the token sequence is classified. Then, the expert network in a multi-expert network model predicts the cluster center of each cluster, obtaining a first predicted value for each cluster center. To more accurately estimate the tokens in the token sequence, this application also introduces a compensation value for each token. This compensation value can compensate for the difference between the token and the cluster center, thus obtaining an estimated value for each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. In this application, the expert network generates the digital human's actions based on the estimated value of each token. The estimated value of each token is determined based on the cluster center and the compensation value of each token. Therefore, it is not necessary to transmit the tokens to the expert network. Instead, the estimated value of each token can be obtained by predicting the cluster center and obtaining the compensation value of each token. The estimated value of each token is used as a substitute for each token in the digital human's action generation. This method can reduce the time spent transmitting tokens, thereby shortening the communication time between GPUs caused by token transmission and improving the efficiency of digital human action generation. Furthermore, this application uses an expert network to predict cluster centers, which can shorten the time for calculating cluster centers. For example, it can obtain some tokens in a cluster and analyze these tokens through an expert network to predict the cluster center, thereby shortening the time for calculating cluster centers and improving the efficiency of digital human motion generation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of the multi-expert network model provided in the embodiments of this application; Figure 2 A schematic diagram illustrating the data processing procedure of the multi-expert network model provided in this application embodiment; Figure 3 A flowchart illustrating a data processing method provided for an application embodiment of this application. Figure 4 A schematic diagram of the data processing procedure provided for an application embodiment of this application; Figure 5 Another schematic diagram illustrating the data processing procedure provided for an application embodiment of this application; Figure 6 A schematic diagram illustrating the generation of actions applied to a virtual anchor, provided as an example of an application embodiment of this application; Figure 7 Another flowchart illustrating a data processing method provided for an application embodiment of this application. Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "one or more" in this document refers to any combination of at least two of any one or more elements from a set of A, B, and C. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set of A, B, and C.

[0020] It should also be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0021] It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship of instruction and being instructed, configuration and being configured, etc.

[0022] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0024] The multi-expert network model in this application is described below: Multi-expert network models, also known as Mixtures-of-Expert Architectures, can improve the training efficiency of Transformer models. The main components of this model are expert networks and sparse gate networks (also called gated networks). To efficiently train these models on multiple GPUs, the gated network can be replicated on each GPU, and the different expert networks can be distributed across multiple GPUs for parallel processing. See also... Figure 1 , Figure 1 This describes a parallel architecture for training multi-expert network models across multiple GPUs. Specifically, the input token is first processed by a gating network to match the corresponding expert network. Then, the token is passed from the original GPU to the GPU containing the corresponding expert network via peer-to-peer (P2P) network communication. After the corresponding expert network completes its computation, the token is returned to the original GPU for further processing via another peer-to-peer network communication. For example, Figure 1 In the process, x0 is processed through a gating network, which matches x0 with expert network 3. At this point, x0 needs to be passed from GPU 0 to GPU 2, where expert network 3 resides, via peer-to-peer (P2P) network communication. For example, after expert network 3 processes x0, x0 needs to be returned from GPU 2 to GPU 0 via another peer-to-peer network communication. From the above processing flow, it can be seen that when processing input tokens, each GPU typically needs to send tokens to many other GPUs; these peer-to-peer transmissions form a full-to-full communication.

[0025] Multi-expert network models can effectively balance parameter size and training cost. Studies have shown that, with the same number of parameters, the performance of multi-expert network models can significantly outperform dense models. First, please refer to... Figure 2, Figure 2 This provides the framework for a multi-expert network model, which consists of multiple specialized, distributed networks, commonly referred to as expert networks, denoted as [missing information]. Where N represents the number of expert networks, This represents the output of expert network i after processing input x. Each expert network is trained to handle a specific sub-task, thereby improving the overall model's capabilities. In the base model, multi-expert network layers replace traditional feedforward neural networks (FNNs). In multi-expert network layers, the feedforward neural network layer works like an independent expert, significantly improving the model's ability to handle dense and complex input data. Secondly, gating networks play a crucial role in the multi-expert network structure. In a K-way gating network, the gating network outputs a series of integers to assign specific tasks to the expert networks, with the specific assignment method being... This allocation process is based on data type to achieve dynamic and efficient allocation of computing resources. Finally, a multi-expert network model integrates the processing results of multiple specialized expert networks; the specific integration method is as follows: .

[0026] Because the computation of this model depends on the communication process, training the model across multiple GPU clusters encounters a severe performance bottleneck. In related research, the TA-MoE framework aims to minimize cross-machine communication by modifying the model definition. This approach involves optimizing the routing strategy of the gating network to prioritize expert networks on the same server. Simultaneously, Pre-gated Hybrid Expert Models (Pre-gated MoE) attempt to decouple the sequential dependency between full-to-full communication and expert network computation. This is achieved by implementing a pre-gating mechanism that establishes a routing plan for subsequent multi-expert network layers. However, these methods involve modifying the routing mechanism of the gating network during multi-expert network model training and changing the model definition, making it difficult to ensure their applicability in all multi-expert network training scenarios.

[0027] The main advantage of multi-expert network (MEN) models is decoupling parameter size from network cost. However, a fundamental challenge lies in effectively learning the gating function, stemming from the non-differentiability of the MEN model output, which in turn is due to the model's sparsity. Therefore, research in MEN primarily focuses on developing techniques for learning gating functions. These techniques can be categorized into three types: routing through learned weights; routing through deterministic hashing; and routing through reinforcement learning. These methods primarily focus on the design of the gating network, not the expert network itself. Consequently, they all face the same scalability problem: they cannot scale to scenarios with larger datasets. In scenarios with larger datasets, the number of tokens increases. When using these methods, each GPU still needs to send an increased number of tokens to many other GPUs, resulting in longer all-to-all communication times and consequently, inefficient model processing, making them unsuitable for such scenarios.

[0028] In existing large-scale multi-expert network training systems, expert networks are distributed across different GPUs for parallelization. Therefore, input data requires additional full-to-full communication to access the target expert network and perform corresponding computations. Existing research indicates that full-to-full communication time accounts for an average of approximately 45% of the training process for three mainstream multi-expert network models on commonly used GPU clusters, significantly hindering the training efficiency and scalability of multi-expert network models.

[0029] In related technologies, multi-expert network models can be used for digital human motion generation. A multi-expert network model consists of multiple expert networks, each an independent sub-network that can be used to compute input tokens. However, since the different expert networks in the multi-expert network model are distributed across different GPUs, the input token needs to be transmitted to the corresponding expert network's GPU for computation, which consumes considerable time and results in low efficiency for digital human motion generation.

[0030] Based on this, in this embodiment, the token sequence is obtained by converting the skeletal animation data of the first digital human. The token sequence is acquired, and then the tokens in the sequence are divided into multiple clusters, i.e., the token sequence is classified. Then, the expert network in the multi-expert network model predicts the cluster center of each cluster, obtaining a first predicted value for each cluster center. To more accurately estimate the tokens in the token sequence, this application also introduces a compensation value for each token. This compensation value can compensate for the difference between the token and the cluster center, thereby obtaining an estimated value for each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. In this application, the expert network generates the digital human's actions based on the estimated value of the tokens. This estimated value is determined according to the cluster center and the compensation value of each token. Therefore, it is not necessary to transmit the tokens to the expert network. Instead, the estimated value of the tokens can be obtained by predicting the cluster center and obtaining the compensation value of each token. This estimated value is used as a substitute for the token in the digital human's action generation. This method can reduce the time spent transmitting tokens, thereby shortening the communication time between GPUs caused by token transmission and improving the efficiency of digital human action generation. Furthermore, this application uses an expert network to predict cluster centers. This method can shorten the time required to calculate cluster centers. For example, by obtaining some tokens from a cluster and analyzing these tokens through an expert network, the cluster center can be predicted, thereby shortening the calculation time and improving the efficiency of digital human generation. The following description, in conjunction with the accompanying drawings and embodiments, will provide a more detailed account of this application.

[0031] This application provides a data processing method, see [link to relevant documentation] Figure 3 The method includes: Step 301: Obtain the token sequence.

[0032] The token sequence was derived from the skeletal animation data of the first digital human.

[0033] It should be noted that the token sequence in this application is an exemplary term and can be replaced with any possible term, such as token set or token.

[0034] For example, a token sequence may include multiple tokens.

[0035] It should be noted that, in a typical application of this application, the skeletal animation data is skeletal animation data of a digital human. The digital human can be multi-dimensional, for example, it can be 2D, 3D, 4D, or 5D. This application does not limit the field of digital humans; for example, a digital human can be a virtual anchor, virtual idol, animated character, game character, intelligent human assistant, virtual teacher, virtual instrument player, augmented reality (AR) digital human, virtual reality (VR) digital human, digital human in the metaverse, digital twin-generated digital human, or digital physiological human, etc. This application does not limit the skeletal animation data to necessarily being skeletal animation data of a digital human; for example, the skeletal animation data in this application can also be skeletal animation data of a digital animal or virtual character. This virtual character does not have to be a human or animal, but rather a newly created character, such as a monster character created in a game. This application does not limit the skeletal animation data to necessarily being skeletal animation data of virtual human beings or animals; the skeletal animation data in this application can also be skeletal animation data of actual human beings or animals.

[0036] For example, the data types of skeletal animation data include: posture data (e.g., gesture data), skeletal animation data during movement, or facial expression data, etc. Skeletal animation data during movement can be skeletal animation data during activities such as walking, running, boxing, ball sports (e.g., basketball games, football games), or swimming. For example, when this application is applied to a game scenario, the skeletal animation data in this application can be skeletal animation data during action interactions between game characters, such as skeletal animation data of game characters during movement, or skeletal animation data corresponding to the gestures or facial expressions of game characters, etc. For example, when this application is applied to the metaverse, the skeletal animation data can be skeletal animation data of a digital human performing musical instrument playing, running, boxing, etc.

[0037] For example, the skeletal animation data can be used for model training. For instance, the model can be a multi-expert network model, which can be used for motion generation of digital humans. Alternatively, the model can be a neural network model other than an expert network model, such as a muscle deformation model. This model can also be used for scenarios other than motion generation of digital humans. For example, when the model is a muscle deformation model, it can be used to calculate muscle contraction and skin elastic deformation.

[0038] For example, the continuous motion representation in skeletal animation data can be discretized to obtain a token sequence. For instance, the skeletal animation data can be processed using a Vector Quantized Variational Autoencoder (VQ-VAE) to obtain the token sequence.

[0039] For example, a frame of skeletal animation data is converted into a token sequence.

[0040] For example, a multi-expert network model includes a gated network and multiple expert networks.

[0041] Step 302: Divide the tokens in the token sequence into multiple clusters.

[0042] Understandably, each of these multiple clusters includes one or more tokens.

[0043] For example, the number of expert networks is equal to the number of clusters in multiple clusters.

[0044] For example, in a gated network, the tokens in the token sequence are divided into multiple clusters.

[0045] In practical applications, the token sequence is divided into multiple clusters using locality sensitive hashing (LSH) based on random projection.

[0046] It should be noted that Locality Sensitive Hashing (LSH) enhances the ability to quickly identify similarities in high-dimensional spaces by hashing similar data points into the same bucket. This application employs a LSH method based on random projection, which is well-suited to the high-dimensional features and sparsity of the token sequences corresponding to skeletal animation data of digital humans, improving the efficiency and accuracy of token sequence clustering. Specifically, firstly, the token sequences corresponding to skeletal animation data of digital humans have high-dimensional features. These token sequences typically contain a large number of features, which can lead to the "curse of dimensionality" in high-dimensional spaces. Random projection is an effective dimensionality reduction technique that projects high-dimensional data to lower dimensions using a random matrix while preserving distance relationships as much as possible, making subsequent calculations more efficient. Secondly, LSH based on random projection enables similarity search. The main goal of LSH is to quickly find similar data points, and the similarity between data points in the token sequences of this application can be evaluated using metrics such as Euclidean distance and cosine similarity between feature vectors. LSH based on random projection can map similar data points into the same hash bucket through a hash function, thereby greatly accelerating the similarity search process. Secondly, locality-sensitive hashing based on random projection can reduce storage requirements. Traditional distance calculation methods require storing and processing large similarity matrices, while reducing data dimensionality through random projection can significantly reduce storage needs and accelerate the retrieval process, especially when dealing with large-scale datasets. Finally, locality-sensitive hashing based on random projection is robust and flexible. Random projection has a certain tolerance for noise and outliers, which is particularly important when dealing with real-world data. Furthermore, random projection methods can be combined with various distance metrics to adapt to different similarity definitions.

[0047] In Locality Sensitive Hashing (LSH) algorithms, each algorithm employs a unique hashing method to map data into buckets. Digital human data typically contains rich feature vectors; therefore, LSH algorithms based on random projection can be used to achieve hashing by mapping high-dimensional feature vectors to a low-dimensional space. For example, for each token, multiple projection vectors are projected onto each token, resulting in multiple projection values ​​for each token. These multiple projection values ​​serve as hash bucket identifiers for each token; the token sequence is then divided into multiple clusters based on the similarity of the hash bucket identifiers. Specifically: For example, suppose the tokens in the input token sequence are Generate k random projection vectors. For example, these random projection vectors can be drawn from a normal distribution with a mean of 0 and a variance of 1, where k is a positive integer. Then x is projected using the following formula:

[0048] in, It is a symbolic function, for >0 returns 1. <0 returns -1, =0 returns 1 or -1, and usually strictly chooses one of them.

[0049] Generate a hash bucket identifier from multiple hash functions: .

[0050] In practical applications, the similarity of hash bucket identifiers of different tokens can be calculated, and tokens with a similarity of not less than a threshold of 1 can be grouped into a cluster.

[0051] To help understand the above example of locality-sensitive hashing based on random projection, consider the following: In a token sequence, token 1 is (1, -2, 0, 0), token 2 is (2, -3, -4, 0), and token 3 is (10, 20, 30, 40). The projection vectors are (1 / 2, 1 / 2, 1 / 2, 1 / 2) and (0, 1, 0, 1). Assume... =0 If the value is -1, then h1(x) = 1 - 1 - 1 - 1 = -2, and h2(x) = -1 - 1 - 1 - 1 = -4. Similarly, h1(x) = 1 - 1 - 1 - 1 = -2, and h2(x) = -1 - 1 - 1 - 1 = -4. Therefore, h1(x) = 1 + 1 + 1 + 1 = 4, and h2(x) = -1 + 1 - 1 + 1 = 0. Assuming threshold 1 is 50%, and since h1(x) and h1(x) of token 1 are the same, and h2(x) of token 1 and h2(x) of token 2 are also the same, the similarity between token 1 and token 2 is 100%, which is not lower than threshold 1. Therefore, token 1 and token 2 can be grouped into the same cluster. Since h1(x) of token 1 is different from h1(x) of token 3, and h2(x) of token 1 is also different from h2(x) of token 3, the similarity between token 1 and token 3 is 0, which is lower than the threshold of 1. Therefore, token 1 and token 3 cannot be classified into the same cluster.

[0052] In practical applications, locality-sensitive hashing based on random projection can be introduced into gated networks to divide the tokens in the token sequence into multiple clusters.

[0053] It should be noted that this application is not limited to dividing the tokens in the token sequence into multiple clusters only by local sensitive hashing based on random projection. Other clustering methods can also be used in this application, such as MinHash or imHash, or traditional clustering algorithms such as K-means.

[0054] Step 303: Predict the cluster center of each cluster in multiple clusters using the expert network in the multi-expert network model, and obtain the first predicted value of each cluster center.

[0055] Understandably, please refer to Figure 4 Each cluster includes a cluster center, i.e. In this application, the cluster center is predicted by an expert network. For example, a portion of the tokens in the cluster can be obtained, and the cluster center of the cluster can be predicted by analyzing these tokens through an expert network. This method can shorten the time for calculating the cluster center and thus improve the efficiency of digital human generation.

[0056] In practical applications, the compensation value of each token is determined based on neighborhood information, which is the association information of each data point in one or more first data points. The distance between one or more data points in the skeletal animation data and the first data points is less than a first threshold.

[0057] It should be noted that the compensation value in this application is an exemplary term and can be replaced with any possible term, such as compensation or residual compensation.

[0058] Understandably, since there may be differences between each token and the cluster center, in order to more accurately determine the estimated value of each token, it is necessary to determine a compensation value for each token relative to the cluster center. That is, in this application, the compensation value is to compensate for the difference between each token and the cluster center.

[0059] To more accurately determine the compensation value, this application introduces neighborhood information, which is the association information of each data point in one or more first data points. The distance between one or more data points in the skeletal animation data and the first data point is less than a first threshold. That is, the neighborhood information is the association information of the neighboring data points of the skeletal data point. In this way, the token can be estimated more accurately by referring to the association information of the neighboring data points.

[0060] It should be noted that the compensation value for each token determined based on neighborhood information in this application is a dynamic compensation scheme based on neighborhood information, which is an implementation method for residual compensation of high-dimensional semantic features. Digital human skeletal animation data is typically high-dimensional data, which usually involves complex feature spaces. The dynamic compensation method based on neighborhood information in this application is well-suited for processing high-dimensional data, thus enabling better processing of digital human skeletal animation data. This method can dynamically adjust compensation by combining features within the local neighborhood, thereby improving the effectiveness of compensation. By applying the dynamic compensation method based on neighborhood information, this application can effectively improve the accuracy and reliability of the data. In digital human applications, such as virtual reality, augmented reality, and human-computer interaction, the real-time performance and accuracy of the data are crucial. During the use of digital human skeletal animation data, the data may be affected by various factors, such as sensor errors, environmental noise, and unpredictable changes in user interaction. The dynamic compensation method based on neighborhood information can adjust the data in real time, reducing the impact of these interferences and thus improving the overall system performance.

[0061] It should be noted that, in this application, the distance between data points can be the distance between data points in the feature space. This means that the position of a data point in the feature space is represented by a vector, and the distance between data points in the feature space is the distance between these vectors.

[0062] It should be noted that the first data point is a data point within the neighborhood of the skeletal animation data, and each data point within this neighborhood is less than the first threshold distance from the skeletal animation data in the feature space.

[0063] In practical applications, neighborhood information includes one or more of the following: information indicating changes in the motion state of the first digital human; information indicating the state of the second digital human, wherein the difference between the feature vectors of the first digital human and the feature vectors of the second digital human is less than a second threshold; information indicating changes in the environment in which the first digital human is located; and human-computer interaction information.

[0064] For example, the information used to indicate changes in the motion state of the first digital human could be information indicating changes in the first digital human's motion speed, motion mode, and motion position.

[0065] For example, the second digital human could be a digital human surrounding the first digital human, and the two could engage in some kind of movement interaction. For instance, the first and second digital humans could be boxers in a match. Since the second digital human might influence the state of the first digital human, incorporating information about the second digital human's state helps to more accurately determine the state of the first digital human.

[0066] For example, the information used to indicate changes in the environment in which the first digital human is located could be changes in the terrain or changes in the spatial location of the first digital human.

[0067] For example, human-computer interaction information can be user interaction behavior information. In a game scenario, a digital human acts as a game character, and the user's interaction behavior information can be the user's instructions to the game character, such as instructing the game character to run faster.

[0068] It should be noted that the process of determining the compensation value of each token based on neighborhood information is dynamic, meaning that the compensation value can be updated. For example, when neighborhood information changes, the compensation value can be updated, or when an error is detected, such as a sensor error, the compensation value can be updated.

[0069] In practical applications, the compensation value of each token is determined based on neighborhood information in gated networks, thereby shortening the time required for expert networks to obtain compensation values ​​and reducing the time for full-to-full communication between different GPUs.

[0070] In practical applications, the compensation value of each token is determined based on neighborhood information, including: for each token, the compensation value of each token is determined based on neighborhood information and the first vector corresponding to each token. The first vector is obtained by randomly projecting each token.

[0071] For example, suppose the tokens in the input token sequence are x∈R d Then the first vector can be the aforementioned .

[0072] In practical applications, the first vector can be obtained by randomly projecting each token, or it can be the vector representation of the token in the feature space, or a vector obtained by reducing the dimensionality of the token through other dimensionality reduction methods.

[0073] In practical applications, the compensation value of each token is determined based on neighborhood information and the first vector corresponding to each token. This includes: determining the K nearest neighbors of each token from the cluster in which each token belongs based on neighborhood information and the first vector; and calculating the compensation value of each token based on the K nearest neighbors, where K is a positive integer.

[0074] For example, for each token, the distance between other tokens in the same cluster as each token and the first vector corresponding to each token can be calculated based on neighborhood information, and the k tokens with the smallest distance can be taken as the K nearest neighbors of each token.

[0075] For example, when calculating the distance between a data point in the neighborhood of the skeletal animation data and the first vector corresponding to each token, this distance can be the Hamming distance.

[0076] For example, let the token in the token sequence be x. i This application calculates each token x i When compensating, it depends not only on the cluster center of the token. Furthermore, the characteristics of other tokens within the same cluster are also considered. This can be achieved by calculating the K nearest neighbors of the token. Specifically: firstly, based on neighborhood information and token x... i The corresponding first vector is determined by the token x. i The K nearest neighbors within the same cluster, assuming the K nearest neighbors are... Then calculate mean Then, the required compensation is determined based on the information of the K nearest neighbors, as shown in the following example: First, calculate the K-nearest neighbor mean: .

[0077] Secondly, calculate x i and The difference: .

[0078] For example, it can be As token x i Compensation, or The product of the weight λ and the token x is used as the token x. i The compensation, wherein λ can be determined in one or more of the following ways: λ is determined based on the user's instruction information, or based on the token x. i λ is determined by the distance to the cluster center, λ is determined based on neighborhood information, λ is determined according to a predefined definition, λ is determined according to system presets, or λ is randomly selected. Wherein, when based on token x... i When the distance λ between the cluster center and the cluster center is determined, the token x i The greater the difference from the cluster center, the larger the value of λ.

[0079] In the aforementioned methods for determining the compensation value of the token, combining information from the K nearest neighbors allows for better capture of local feature changes, enhancing the model's adaptability. Dynamically adjusting the output based on neighborhood information better handles anomalous samples and uneven feature distribution. Experiments show that compensation based on neighborhood information can further improve the model's accuracy and robustness when dealing with complex, high-dimensional data.

[0080] It should be noted that determining the compensation value for each token based on neighborhood information is one example of a dynamic compensation scheme based on neighborhood information. This application can also implement a dynamic compensation scheme based on neighborhood information through one or more of the following examples: Example 1: By monitoring multiple data points in the neighborhood of skeletal animation data in real time, outliers or erroneous data in the skeletal animation data can be identified, and then tokens converted from outliers or erroneous data can be used for compensation.

[0081] Example 2: Calculate a weighted average using valid data within the neighborhood of the skeletal animation data, and then dynamically compensate for erroneous data based on this weighted average. This method can enhance the compensation effect by weighting the valid data within the neighborhood according to their importance.

[0082] Example 3: By combining machine learning algorithms, historical data within the neighborhood is analyzed to learn the relationships and patterns between the data, so that more accurate compensation can be made when new skeletal animation data appears.

[0083] Step 304: For each token in the token sequence, obtain the estimated value of each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token.

[0084] The compensation value is used to compensate for the difference between each token and the cluster center, and the estimated value of each token is used by the expert network to generate the action of the first digital human.

[0085] It should be noted that the estimated value in this application is an exemplary statement and can be any possible statement, such as an approximation or a predicted value.

[0086] It should be noted that this application is not limited to immediately executing step 303 after obtaining the first predicted value of the cluster center corresponding to each token and the compensation value of each token. Step 303 can be executed when the expert network analyzes the estimated value of the token to generate the action of the first digital human. For example, when the expert network needs to analyze token 1, it can obtain the first predicted value of the cluster center of the cluster to which token 1 belongs and the compensation value of token 1, then determine the estimated value of token 1 based on the first predicted value and the compensation value of token 1, and then process the estimated value through the expert network to generate the action of the first digital human.

[0087] For example, there is a one-to-one relationship between the token sequence and the action of the first digital human. For instance, an expert network can process a token sequence to generate an action of the first digital human.

[0088] For example, the estimated value of each token can be a combination of the first predicted value of the cluster center corresponding to each token and the compensation value of each token. For instance, assuming the weight of the compensation value of each token is λ, the estimated value of each token can be the sum of the product of the compensation value of each token and λ and the first predicted value of the cluster center corresponding to each token. For token x i The estimated value, To process token x through an expert network in a multi-expert network model i The first predicted value is obtained by predicting the cluster center of the cluster, and the compensation value for each token is... ,but .

[0089] For example, the first predicted value of the cluster center corresponding to each token can be calculated in the expert network, while the compensation value of each token can be calculated in the gating network. See also Figure 5 In a gating network, the token sequence can be divided into multiple clusters, such as cluster 1, cluster 2, and cluster 3. An expert network predicts the cluster center of each cluster. For example, the expert network can predict the cluster center of cluster 1 to obtain cluster center 1. The gating network calculates the compensation value of each token in cluster 1. Based on cluster center 1 and the compensation value of each token in cluster 1, the estimated value of each token in cluster 1 is obtained. Then, the expert network processes the estimated value of each token in cluster 1 to generate the actions of the digital human. For example, for each token, the expert network predicting the cluster centers of the token and the expert network processing the token estimate may or may not be the same expert network. For instance, when they are not the same expert network, the expert network predicting the cluster centers needs to transmit the first predicted value of the cluster centers to the expert network processing the token estimate. For example, the expert network predicting the cluster centers can transmit the first predicted value of the cluster centers to the expert network processing the token estimate via point-to-point communication. Furthermore, the expert network processing the token estimate also needs to obtain the token's compensation value, then determine the token's estimate based on the first predicted value of the token's cluster centers and the token's compensation value, and then process the estimate to generate the actions of the first digital human.

[0090] For example, please refer to Figure 6 , Figure 6 The dots in the image represent the skeletal points of the digital human. Figure 6In this process, the token sequence converted from skeletal animation data is processed by a multi-expert network model to generate the actions of the digital human. For example, the actions of the digital human can be the gestures of a virtual anchor. The generated actions of the digital human can be sent to the terminal and displayed to the user on the terminal's screen.

[0091] This application can be used to approximate the skeletal animation data input to the model training, obtaining an estimated value for each token. This estimated value can then be used in the subsequent training process of the model, thereby accelerating the training process. For example, the model can be a multi-expert network model, which can be used for digital human motion generation. The expert network can then generate the motion of the first digital human based on the estimated value of each token. However, this application does not limit the model to digital human motion generation; it can also be used for other processes involving digital humans. For instance, the model can be a muscle deformation model, in which case the estimated value of each token in this application can be used to calculate muscle contraction and skin elastic deformation.

[0092] In practical applications, expert networks can be based on Transformer model architectures or Generative Adversarial Networks (GAN) architectures, and can be used to generate digital actions. For example, the multilayer perceptron (MLP) in the Transformer model can be replaced with sparse MoE blocks, each containing 8 experts. Each token activates only two of these experts, meaning that the estimate for each token can be processed by just two experts. As mentioned earlier, in this application, the expert network can also be used to predict the cluster centers of each cluster.

[0093] In practical applications, based on the first motion library, the estimated value of each token is processed through a multi-expert network model to generate the first motion of the first digital human. The first motion is different from the motion of the first digital human in the skeletal animation data.

[0094] For example, the first action library may include one or more action sequences and semantic tags corresponding to each action sequence. For instance, taking BABEL as an example, this dataset contains human geometric models and action sequences, along with semantic tags corresponding to the action sequences, including both sequence-level and action-frame-level levels. For example, the first action can be an action sequence, and the first action library includes the first action and its corresponding semantic tags. For example, the first action can be determined in one or more of the following ways: determining the first action based on user instructions; determining the first action by analyzing user intent; or determining the first action based on task requirements. For example, if the task requirement is to generate gestures for a virtual anchor, then the first action can be a gesture.

[0095] In practical applications, the actions of the first digital human can be generated by combining the results of multiple expert networks. For example, the estimated value of token 3 in the token sequence needs to be processed by expert network 1 and expert network 2. In this case, the processing results of expert network 1 and expert network 2 can be combined to obtain the processing result of token 3. Furthermore, in practical applications, an action of the first digital human can be generated based on the processing results of multiple tokens.

[0096] As shown above, this application can obtain the estimated value of each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. Then, the expert network generates digital human actions based on the estimated value of the token. Since the estimated value of the token is determined based on the cluster center and the compensation value of each token, it is not necessary to transmit the token to the expert network. Instead, the estimated value of the token can be obtained by predicting the cluster center and calculating the compensation value of each token. This estimated value is used as a token substitute in the digital human action generation. This method can reduce the time spent transmitting tokens, thereby shortening the communication time between GPUs caused by transmitting tokens and improving the efficiency of digital human action generation. In addition, this application can accelerate the model training process of digital humans. This application can effectively process skeletal animation data, quickly estimate the estimated value of the token corresponding to the skeletal animation data, and use the estimated value as a token substitute in the model training process, thereby reducing the time required for the expert network to obtain tokens across GPUs, thus shortening the communication time between different GPUs, and therefore shortening the model training time. Furthermore, in this application, the token sequence is clustered using Locality Sensitive Hashing (LSH) based on random projection, then cluster centers are predicted by an expert network, and residual compensation is applied to the cluster centers predicted by the expert network using a dynamic compensation scheme based on neighborhood information to obtain the estimated token values. This application significantly reduces communication time by clustering tokens in the token sequence using LSH based on random projection, and the dynamic compensation scheme based on neighborhood information protects the integrity of the model. While maintaining model quality, it enables faster training on datasets of digital human skeletal animation data.

[0097] The present application will be further described in detail below with reference to application examples.

[0098] This application provides a data processing method through its application embodiments; see [link to relevant documentation]. Figure 7 This data processing method can mainly include the following steps: Step 701: Obtain the skeletal animation data of the first digital human.

[0099] For example, the skeletal animation data can be input data for model training, and the model can be a multi-expert network model or a neural network model other than an expert network model. This application can approximate the skeletal animation data used for model training to obtain an estimate of the token corresponding to the skeletal animation data, wherein the time required to obtain the estimate is shorter than the time required to obtain the skeletal animation data.

[0100] For example, skeletal animation data types include: posture data (e.g., gesture data), skeletal animation data during movement, or facial expression data, etc. Skeletal animation data during movement can be skeletal animation data during activities such as walking, running, boxing, ball sports (e.g., basketball games, football games), or swimming.

[0101] Step 702: Convert the skeletal animation data into a token sequence.

[0102] It should be noted that step 2 is optional. For example, if the skeletal animation data may be converted into a token sequence through other devices or servers, step 2 may not be performed.

[0103] Step 703: Divide the token sequence into multiple clusters using locality-sensitive hashing based on random projection.

[0104] For example, for each token, multiple projection vectors are used to project each token, resulting in multiple projection values ​​for each token. These multiple projection values ​​are then used as hash bucket identifiers for each token. The token sequence is then divided into multiple clusters based on the similarity of the hash bucket identifiers. Specifically: For example, suppose the tokens in the input token sequence are Generate k random projection vectors. For example, these random projection vectors can be drawn from a normal distribution with a mean of 0 and a variance of 1, where k is a positive integer. Then x is projected using the following formula:

[0105] Here, sign(z) is a sign function that returns 1 for z>0, -1 for z<0, and can choose to return 1 or -1 when z equals 0. Usually, one of them is strictly chosen.

[0106] Generate a hash bucket identifier from multiple hash functions: .

[0107] In practical applications, the similarity of hash bucket identifiers of different tokens can be calculated, and tokens with a similarity of not less than a threshold of 1 can be grouped into a cluster.

[0108] It should be noted that step 703 above is an optional step. This application does not limit the token sequence to be divided into multiple clusters by locality-sensitive hashing based on random projection. Other clustering methods can also be used to divide the token sequence into multiple clusters.

[0109] Step 704: Predict the cluster center of each cluster in multiple clusters using the expert network in the multi-expert network model, and obtain the first predicted value of each cluster center.

[0110] In practical applications, an expert network is used to analyze a subset of tokens in each cluster in order to predict the cluster center of the cluster to which that subset of tokens belongs.

[0111] In practical applications, the compensation value of each token is determined based on neighborhood information, which is the association information of each data point in one or more first data points. The distance between one or more data points in the skeletal animation data and the first data points is less than a first threshold.

[0112] Step 705: Determine the compensation value for each token based on neighborhood information.

[0113] For example, let the token in the token sequence be x. i This application calculates each token x i When compensating, it depends not only on the cluster center of the token. Furthermore, the characteristics of other tokens within the same cluster are also considered. This can be achieved by calculating the K nearest neighbors of the token. Specifically: firstly, based on neighborhood information and token x... i The corresponding first vector is determined by the token x. i The K nearest neighbors within the same cluster, assuming the K nearest neighbors are... Then calculate mean Then, the required compensation is determined based on the information of the K nearest neighbors, as shown in the following example: First, calculate the K-nearest neighbor mean: .

[0114] Secondly, calculate x i and The difference: .

[0115] For example, it can be As token x i Compensation, or The product of the weight λ and the token x is used as the token x. i The compensation, wherein λ can be determined in one or more of the following ways: λ is determined based on the user's instruction information, or based on the token x. i λ is determined by the distance to the cluster center, λ is determined based on neighborhood information, λ is determined according to a predefined definition, λ is determined according to system presets, or λ is randomly selected. Wherein, when based on token x... i When the distance λ between the cluster center and the cluster center is determined, the token x i The greater the difference from the cluster center, the larger the value of λ.

[0116] It should be noted that step 705 above is an optional step. This application does not limit the determination of the compensation value of each token to neighborhood information, and other methods of determining the compensation value may also be used.

[0117] Step 706: For each token in the token sequence, obtain an estimate of each token based on the first predicted value of the cluster center corresponding to each token and a compensation value of the token.

[0118] For example, the estimated value of each token can be a combination of the first predicted value of the cluster center corresponding to each token and the compensation value of each token. For instance, assuming the weight of the compensation value of each token is λ, the estimated value of each token can be the sum of the product of the compensation value of each token and λ and the first predicted value of the cluster center corresponding to each token. For token x i The estimated value, To process token x through an expert network in a multi-expert network model i The first predicted value is obtained by predicting the cluster center of the cluster, and the compensation value for each token is... ,but .

[0119] Step 707: Generate the action of the first digital human based on the estimate of each token through an expert network.

[0120] In practical applications, based on the first motion library, the estimated value of each token is processed through a multi-expert network model to generate the first motion of the first digital human. The first motion is different from the motion of the first digital human in the skeletal animation data.

[0121] For example, the first action library may include one or more action sequences and semantic tags corresponding to each action sequence. For instance, the first action may be an action sequence, and the first action library includes the first action and the semantic tags corresponding to the first action.

[0122] In practical applications, the actions of the first digital human can be generated by combining the results of multiple expert networks. For example, the estimated value of token 3 in the token sequence needs to be processed by expert network 1 and expert network 2. In this case, the processing results of expert network 1 and expert network 2 can be combined to obtain the processing result of token 3. Furthermore, in practical applications, an action can be generated based on the processing results of multiple tokens.

[0123] It should be noted that step 707 is optional. This application does not require that step 707 be performed.

[0124] For example, this application can be used to approximate the skeletal animation data input to the model training, obtain an estimated value for each token, and use the estimated value of each token in the subsequent training process of the model, thereby accelerating the training process. For example, the model can be a multi-expert network model, which can be used for digital human motion generation. In this case, the first digital human's motion can be generated by the expert network based on the estimated value of each token. However, this application does not limit the model to digital human motion generation only. The model can also be used for other processes of digital humans. For example, the model can be a muscle deformation model, in which case the estimated value of each token in this application can be used to calculate muscle contraction and skin elastic deformation.

[0125] As shown above, this application can obtain the estimated value of each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. Compared with the method of obtaining tokens through expert networks, the method of obtaining token estimation values ​​through expert networks can shorten the communication time between different GPUs. For example, when it is necessary to process skeletal animation data through a multi-expert network to generate digital human movements, this application can effectively process skeletal animation data and quickly estimate the estimated value of the tokens corresponding to the skeletal animation data, thereby accelerating the generation of digital human movements. For example, when it is necessary to train a multi-expert network model using skeletal animation data, this application can reduce the time required for the expert network to obtain tokens across GPUs, thereby shortening the communication time between different GPUs and thus shortening the model training time. In addition, in this application, the token sequence is clustered by locality-sensitive hashing based on random projection, then the cluster centers are predicted by the expert network, and residual compensation is performed on the cluster centers predicted by the expert network based on a dynamic compensation scheme based on neighborhood information to obtain the estimated value of the tokens. In this application, expert networks can be used to process skeletal animations quickly, and an approximation method is used for acceleration to support downstream intelligent tasks more quickly. This application can also be applied to training datasets of skeletal animation data of digital humans, accelerating the training speed and reducing training time and cost. Specifically, clustering tokens in a token sequence using locality-sensitive hashing based on random projection significantly reduces communication time, and a dynamic compensation scheme based on neighborhood information protects the integrity of the model. While maintaining model quality, this application achieves faster training of datasets of skeletal animation data of digital humans.

[0126] Based on the embodiments described above, this application also provides a data processing apparatus, see [link to previous document]. Figure 8 The data processing device includes: Acquisition unit 801 is used to acquire a token sequence, which is obtained by converting the skeletal animation data of the first digital human; The partitioning unit 802 is used to divide the tokens in the token sequence into multiple clusters; The prediction unit 803 is used to predict the cluster center of each of the multiple clusters through the expert network in the multi-expert network model, and obtain the first predicted value of each cluster center. The first processing unit 804 is configured to obtain an estimated value for each token in the token sequence based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. The compensation value is used to compensate for the difference between each token and the cluster center, and the estimated value of each token is used to generate the action of the first digital human.

[0127] In one embodiment, the data processing apparatus further includes a second processing unit, the second processing unit being configured to: The compensation value for each token is determined based on neighborhood information, which is the association information of each data point in one or more first data points. The distance between one or more data points in the skeletal animation data and the first data point is less than a first threshold.

[0128] In one embodiment, the neighborhood information includes one or more of the following: Information used to indicate changes in the motion state of the first digital human; Information used to indicate the state of the second digital human, wherein the difference between the feature vector of the first digital human and the feature vector of the second digital human is less than a second threshold. Information used to indicate changes in the environment in which the first digital human is located; Human-computer interaction information.

[0129] In one embodiment, the second processing unit determines the compensation value for each token based on neighborhood information, including: For each token, a compensation value is determined based on the neighborhood information and the first vector corresponding to each token. The first vector is obtained by randomly projecting each token.

[0130] In one embodiment, the second processing unit determines the K nearest neighbors of each token from the cluster in which each token is located based on the neighborhood information and the first vector, where K is a positive integer; and calculates the compensation value of each token based on the K nearest neighbors.

[0131] In one embodiment, the partitioning unit 802 divides the token sequence into multiple clusters, including: The token sequence is divided into multiple clusters using locality-sensitive hashing based on random projection.

[0132] In one embodiment, the first processing unit 804 is further configured to process the estimated value of each token based on the first action library through the multi-expert network model to generate a first action of the first digital human, the first action being different from the action of the first digital human in the skeletal animation data.

[0133] In practical applications, the acquisition unit 801, the division unit 802, the prediction unit 803, the first processing unit 804, and the second processing unit can be implemented by the processor in the data processing device.

[0134] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0135] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, see [link to relevant documentation]. Figure 9 The electronic device includes: The first communication interface 1 is capable of exchanging information with other devices; The first processor 2 is configured to acquire a token sequence, which is obtained by converting the skeletal animation data of the first digital human; divide the tokens in the token sequence into multiple clusters; predict the cluster center of each cluster in the multiple clusters through the expert network in the multi-expert network model to obtain a first predicted value for each cluster center; and obtain an estimated value for each token in the token sequence based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token, wherein the compensation value is used to compensate for the difference between each token and the cluster center, and the estimated value of each token is used to generate the action of the first digital human.

[0136] Specifically, the first processor 2 determines the compensation value of each token based on neighborhood information, wherein the neighborhood information is the association information of each data point in one or more first data points, and the distance between one or more data points in the skeletal animation data and the first data points is less than a first threshold.

[0137] In one embodiment, the neighborhood information includes one or more of the following: Information used to indicate changes in the motion state of the first digital human; Information used to indicate the state of the second digital human, wherein the difference between the feature vector of the first digital human and the feature vector of the second digital human is less than a second threshold; Information used to indicate changes in the environment in which the first digital human is located; Human-computer interaction information.

[0138] In one embodiment, the first processor 2 determines the compensation value for each token based on neighborhood information, including: For each token, a compensation value is determined based on the neighborhood information and the first vector corresponding to each token. The first vector is obtained by randomly projecting each token.

[0139] In one embodiment, the first processor 2 determines the K nearest neighbors of each token from the cluster in which each token is located based on the neighborhood information and the first vector, where K is a positive integer; The compensation value for each token is calculated based on the K nearest neighbors.

[0140] In one embodiment, the first processor 2 divides the token sequence into multiple clusters, including: The token sequence is divided into multiple clusters using locality-sensitive hashing based on random projection.

[0141] In one embodiment, the first processor 2 is further configured to process the estimated value of each token based on the first motion library through the multi-expert network model to generate a first motion of the first digital human, which is different from the motion of the first digital human in the skeletal animation data.

[0142] It should be noted that the specific processing procedure of the first communication interface 1 can be understood by referring to the above method.

[0143] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 9 The general will label all buses as Bus System 4.

[0144] The first memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0145] The methods disclosed in the embodiments of this application can be applied to the first processor 2, or implemented by the first processor 2. The first processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 2. The first processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 2 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 3. The first processor 2 reads the information in the first memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0146] In an exemplary embodiment, the electronic device may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0147] It is understood that the first memory 3 in the embodiments of this application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0148] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 3 storing a computer program, which can be executed by a first processor 2 to complete the steps described in the aforementioned method.

[0149] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 2 to perform the steps described in the foregoing method.

[0150] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a token sequence, which is derived from the skeletal animation data of a first digital human; The tokens in the token sequence are divided into multiple clusters; The cluster center of each of the multiple clusters is predicted by the expert network in the multi-expert network model, and a first predicted value of each cluster center is obtained. For each token in the token sequence, an estimated value for each token is obtained based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token. The compensation value is used to compensate for the difference between each token and the cluster center. The estimated value of each token is used by the expert network to generate the action of the first digital human.

2. The method according to claim 1, characterized in that, Before obtaining the estimated value of each token based on the first predicted value of the cluster center corresponding to each token and the compensation value of each token, the method further includes: The compensation value of each token is determined based on neighborhood information, where the neighborhood information is the association information of each data point in one or more first data points, and the distance between one or more data points in the skeletal animation data and the first data points is less than a first threshold.

3. The method according to claim 2, characterized in that, The neighborhood information includes one or more of the following: Information used to indicate changes in the motion state of the first digital human; Information used to indicate the state of the second digital human, wherein the difference between the feature vector of the first digital human and the feature vector of the second digital human is less than a second threshold. Information used to indicate changes in the environment in which the first digital human is located; Human-computer interaction information.

4. The method according to claim 2 or 3, characterized in that, The determination of the compensation value for each token based on neighborhood information includes: For each token, a compensation value is determined based on the neighborhood information and a first vector corresponding to each token, wherein the first vector is obtained by randomly projecting each token.

5. The method according to claim 4, characterized in that, Determining the compensation value for each token based on the neighborhood information and the first vector corresponding to each token includes: Based on the neighborhood information and the first vector, determine the K nearest neighbors of each token from the cluster in which each token belongs, where K is a positive integer; The compensation value for each token is calculated based on the K nearest neighbors.

6. The method according to any one of claims 1 to 5, characterized in that, The step of dividing the token sequence into multiple clusters includes: The token sequence is divided into multiple clusters using locality-sensitive hashing based on random projection.

7. The method according to claim 6, characterized in that, The method further includes: Based on the first motion library, the estimated value of each token is processed by the multi-expert network model to generate the first motion of the first digital human, which is different from the motion of the first digital human in the skeletal animation data.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images