Method and system for interactive motion data processing

Abstract

A method and system for motion data processing using a neural network data encoding model are provided. The method includes: inputting source motion data into the input layer of the encoder part of the data encoding model; processing the source motion data through at least one continuous representation generation layer of the encoder part to convert the source motion data into compressed motion data based on a predefined continuous function; inputting the compressed motion data into the decoder input layer of the decoder part of the data encoding model and guiding it to multiple independent component neural networks of the decoding layer; decoding the compressed motion data by each of the multiple independent component neural networks to generate multiple independent decoded compressed motion data, and fusing the multiple independent decoded compressed motion data to generate decoded compressed motion data; performing decompression processing on the decoded compressed motion data through the decompression layer of the decoder part; and outputting the decoded and decompressed motion data as output motion data.

Description

Technical Field

[0001] This invention relates to methods and systems for interactively processing motion data, and more particularly to methods and systems for processing, reproducing, or synthesizing motion data during user interaction. Background Technology

[0002] Currently, digital interactive applications are dedicated to improving animation effects, making them more natural and realistic. In recent years, various techniques such as animation trees, parametric blending, and motion graphs have been developed to enhance animation quality. However, this comes at the cost of increased resource management complexity and a greater strain on the resources of video game devices, including a significant increase in CPU processing time and memory usage.

[0003] To address the complexities of animation system architecture design related to quality improvement, further technologies such as motion matching have emerged and gained widespread application. However, these technologies still suffer from significant performance overhead, including computational and memory resource overhead. This limits their scalability in the performance-demanding scenarios of modern video games, and consequently, their applicability in various applications, such as cross-device video games. As the required resources increase, the CPU cost of animation compositing and retrieval rises accordingly. Similarly, the more resources required, the more memory is consumed, which increases linearly with the amount of data needed to improve quality.

[0004] Limited resources, which must compete with other systems within modern video game engines, become a limiting factor in achieving high-quality results.

[0005] Significant progress has been made in using machine learning and neural networks to synthesize high-quality animations from production data. Learning-based motion matching is one such technique, which has addressed some of the aforementioned challenges. However, while this technique reduces memory requirements, it leads to increased performance consumption.

[0006] Therefore, current machine learning and neural network techniques used for motion data synthesis still suffer from similar performance issues, limiting their applicability in real-time animation synthesis, such as in games. As the amount of data to be represented increases, the size and computational complexity of these neural networks rise dramatically, making them unsuitable for applications with numerous interactive characters, such as animated characters in video games. Ultimately, only a very few methods have been successfully transferred from research to practical implementations in video game engines. Summary of the Invention

[0007] The purpose of this invention is to solve the above-mentioned defects and problems.

[0008] Specifically, the purpose of this invention is to reduce the memory resource requirements and computational complexity, i.e., CPU complexity, needed for animation data processing.

[0009] The methods and systems disclosed in this paper are further developments based on motion matching and machine learning-based action synthesis methods.

[0010] It has been observed that the method and system disclosed in this invention can significantly reduce the memory usage associated with compressing and reconstructing animation data, while also reducing the computational complexity and resource consumption associated with decompressing and compressing animation data. This effect is achieved through optimized computational load allocation, thereby shortening the time required for animation data processing. At the same time, the compactness of storable data and the robustness of the accuracy of posture output are both improved.

[0011] In summary, the method and system described in this invention reduce the memory capacity and computational complexity required for animation data processing and control, enabling faster and more diverse interaction between the user and the computing system. This allows the user to control the computing system with a faster system response speed and allows the user to select and / or control the system to perform a variety of different operations.

[0012] The above effects are achieved by the methods and systems defined in the appended independent claims.

[0013] The details and technical features of the present invention are set forth in the dependent claims.

[0014] In a first aspect, a method for motion data processing using a neural network data encoding model including an encoder portion and a decoder portion is provided, the method comprising: Motion data is input to the encoder input layer of the encoder section; At least one continuous representation generation layer in the encoder part processes the source motion data by transforming the source motion data into compressed motion data based on a predefined continuous function; The encoder output layer of the encoder section outputs compressed motion data; The compressed motion data is input into the decoder input layer of the decoder section, and then the compressed motion data is directed through the decoder input layer to multiple independent component neural networks of the decoder layer. Each of the multiple independent component neural networks contained in the decoding layer of the decoder part decodes the compressed motion data to generate multiple independent decoded compressed motion data, wherein each independent component neural network is connected to the decoder input layer and merges the multiple independent decoded compressed motion data into one or more decoded compressed motion data. The decompression layer in the decoder section decompresses the decoded compressed motion data; and The decoded and decompressed motion data is output as the output motion data.

[0015] This neural network data encoding model performs motion data encoding and subsequent decoding functions, thereby generating reconstructed motion data from received source motion data frames. It can be a neural network autoencoder model, or simply an autoencoder. Specifically, it can be configured to process pose data contained in the motion data to encode and subsequently reconstruct the pose.

[0016] Motion data can include multiple motion data frames. Each motion data frame contains information about the pose or state of a virtual character or object, as well as features associated with that character and its pose.

[0017] Motion data may include a pose information dataset containing information associated with a single animation frame in three-dimensional space. In embodiments of this disclosure, including those described in the claims and specification (hereinafter referred to as "all embodiments of this disclosure"), the motion data is preferably configured as motion data frames, including pose data and related feature data. After encoding, the motion data is in an encoded format.<X,Z> The data is stored, where X and Z can represent vector data <features, latent variables>. Features can include various attributes that can be used to control the movement of virtual objects, such as current / future speed and position or orientation, and / or different actions that the object will perform.

[0018] The attitude can be encoded offline using an encoder, and the encoding format generated by the attitude encoder can be used.<X,Z> That is, <features, latent variables> are stored. When an application using encoded motion data runs, it can use only the decoder to extract the encoded motion data.<X,Z> Begin regenerating the pose.

[0019] This method can be advantageously used for interactive motion data processing, enabling interactive control of the movement and / or other actions of virtual characters. For example, the motion trajectory can be at least partially controlled by the user through a user interface.

[0020] Therefore, encoded motion data, typically comprising multiple motion data frames, can be stored in a vector format representing attitude data and related feature data. One or more motion data frames from the stored encoded motion data can be input to a decoder, which processes the motion data frame and outputs reconstructed attitude data. The motion data frame to be decoded can be selected based on user-input control data. Furthermore, as will be discussed in more detail later, this method can be used to predict future data frames. This prediction can be based on feature X, which can ultimately be adjusted according to user-input control data.

[0021] This method can be applied to various applications that use interactive control based on motion data, such as for synthesizing future motion data frames (e.g., animation clips), and / or for providing predictions or estimates of future motion and / or posture data based on user input. To this end, as described later, user input control data can be fed into a data encoding model to control or influence virtual characters or objects present in the motion data. Applications can include various educational, training, and / or simulation uses, covering multiple fields. Examples can be found in healthcare, traffic management and / or security, vehicle safety, entertainment, and other areas.

[0022] This method can be applied, for example, to pose reconstruction by encoding and decoding motion data containing the pose of a virtual character or object. Here, the term "pose" refers to the posture or physical state of a virtual character or object (e.g., within a virtual environment) at a given point in time.

[0023] An animation data frame refers to a single sampled pose of a virtual character at a specific point in time within an animation.

[0024] Animation clips refer to subsampled segments of animation, used to display character movements for different purposes in a virtual environment.

[0025] Motion matching refers to selecting animation clips based on physical attributes such as speed, rate, position, and orientation, as well as features such as motion and timing.

[0026] This method may involve predicting the pose of a virtual character or object at future points in time based on input control data.

[0027] A continuous representation generation layer transforms the input motion data, specifically pose data, according to a predefined continuous function. This continuous function is chosen to preserve the temporal relationship between consecutive motion data frames and maintain linearity and consistency between the original space (i.e., the input motion data frames) and the compressed space (also known as the hidden manifold, containing the encoded motion data frames). The transformation according to the predefined continuous function ensures that the internal representation of the motion data (e.g., hidden data) is highly aligned with the source motion data input to the encoder section, while maintaining a compact form. This allows other parts of the data encoding model (e.g., layers) and other components that may be included in the motion data processing system (e.g., neural network stepper models and neural network projector models) to run and process quickly. A second aspect of this disclosure provides such a system.

[0028] In the embodiments of this disclosure, including the embodiments in the claims and specification (hereinafter referred to as "all embodiments of this disclosure"), the predefined continuous function is preferably a Lipshitz continuous function, and the continuous representation generation layer may be referred to as a Lipshitz continuous representation generation layer, or simply a Lipshitz layer.

[0029] One or more continuous representation generation layers can be set in the encoder section.

[0030] Encoded and compressed motion data, also known as hidden data, can be stored in memory or storage units.

[0031] Input control data typically includes user input or is generated at least in part based on user input, such as motion data used to control or influence virtual characters or other objects in a virtual environment for interactive control. When input control data involves controlling the movement and / or orientation changes of virtual characters or objects, additional input control data (such as motion matching features) may be input into the data encoding model.

[0032] In the gameplay example, the input control data includes parameters representing one or more physical attributes of the virtual character, such as speed, direction, and / or position, as well as gameplay features such as actions and timings contained in the interactive user controls.

[0033] The decoding layer is configured as a layer containing multiple independent component neural networks, rather than a single neural network (where each node is connected to every node in the preceding and following layers). This achieves load distribution and reduces computational complexity. It also reduces memory requirements. The independent component neural networks can be feedforward neural networks. The number of independent component neural networks processing motion data frames can be selected based on computational complexity (i.e., the complexity of the motion data). In some embodiments, motion data frames may be decoded by two independent component neural networks.

[0034] In the embodiments of this disclosure, including those in the claims and specification (hereinafter referred to as "all embodiments of this disclosure"), the decoding layer is preferably configured as a sparse layer, more preferably as a sparse hybrid expert layer.

[0035] In embodiments of this disclosure, including those described in the claims and specification (hereinafter referred to as "all embodiments of this disclosure"), the decompression layer can be described as a linear layer with an activation function. The term "linear layer" refers to a layer where all nodes are connected to all nodes in both the preceding and following layers. Decompression can be performed according to computational methods known in the field of data encoding and decoding.

[0036] The predefined continuous function is preferably a Lipshitz continuous function. Configuring one or more coding layers to provide a Lipshitz continuous representation of the input motion data has proven particularly advantageous, offering superior coding characteristics. The temporal relationship between consecutive data frames is preserved, and the linearity and consistency between the original space and the compressed space (also known as the hidden manifold) are maintained. Improvements in the distribution of motion data frames in the hidden manifold have been observed when the Lipshitz technique is applied to the encoder section. These effects enable the use of sparse layers (e.g., sparse hybrid expert SMoE layers) as decoding layers in the decoder section.

[0037] The function f is defined as Lipschitz continuous when the constant c satisfies the following condition:

[0038] The Lipschitz continuous function can be effectively learned by a neural network model. The learning of the constant c is guided to satisfy the above conditions.

[0039] Each continuous representation generation layer can be configured as a Lipshitz continuous representation layer with an associated learnable Lipshitz constant c, which is associated with the weight normalization that the Lipshitz continuous representation layer will use.

[0040] Motion data can be transformed by multiple sequentially connected layers of representation generation.

[0041] The Lipshitz continuous representation layer used in the encoder section provides a Lipshitz continuous representation of the animation data in the data encoding model. In other words, the compressed or implicit form of the animation data is represented by a Lipshitz continuous neural network.

[0042] This has been observed to be particularly advantageous in ensuring a high degree of alignment between the internal representation of motion data (e.g., latent data) and the source motion data input to the data encoding model encoder. The compact form of the internal representation of the motion data allows for rapid operation and processing of other parts of the data encoding model (e.g., layers) and other components that may be included in the interactive motion data processing system according to the second aspect of this disclosure (e.g., stepper and projector models). Lipschitz continuity ensures that the relationships between animations in the original motion data are preserved in the compressed space and subsequently in the reconstructed motion data. This has been observed to enable reconstruction quality comparable to the source motion data when decompressing the data to runtime motion data decoding and motion data frame synthesis. Specifically, this method has been observed to be able to reconstruct pose data with high accuracy and predict future pose data.

[0043] It has been observed that the reconstruction quality using the data encoding method described in this paper, especially through the Lipshitz continuous representation of motion data, surpasses that of currently known methods. The temporal characteristics of the motion data are preserved in a way that conventional methods without the application of Lipshitz representation cannot achieve.

[0044] By including Lipschitz representations in the encoder part of the data encoding model, a smooth transition of poses represented by motion data can be achieved in the hidden manifold, where different samples of motion-matching features related to poses in subsequent motion data frames have good correlations.

[0045] The decoding layer is preferably configured as a sparse hybrid expert (SMoE) layer. The SMoE layer comprises multiple independent component neural networks, of which only a subset is activated to process compressed motion data.

[0046] The SMoE layer can be viewed as replacing a large neural network layer with multiple smaller neural network layers. This significantly reduces the number of connections between neural network nodes in the layer. Consequently, computational complexity and the requirements for memory capacity and load are significantly reduced. Instead of feeding compressed motion data into and processing it through a large neural network layer, a subset of smaller neural network layers is selected to process the compressed motion data. This results in a substantial reduction in processing time. Simultaneously, higher accuracy has been observed. Furthermore, because the processing is distributed across a subset of independent neural network components (also known as experts), the reduced computational complexity allows for interactive motion data processing on a larger number of virtual characters.

[0047] By employing the Sparse Hybridization Expert (SMoE) architecture, large neural network layers are replaced with multiple smaller neural network layers, thereby reducing the number of connections between neural network nodes within that layer. Using the SMoE architecture significantly reduces the memory required to store the neural network model weights, resulting in a substantial decrease in CPU complexity. Simultaneously, the time required for motion data processing is reduced, and the accuracy of the resulting output motion data (i.e., reconstructed motion data) can be improved.

[0048] Preferably, when the decoding layer is configured as an SMoE layer, the decoder section may adopt the following architecture.

[0049] The decoder input layer is a gated layer, and the method also includes: Input the compressed motion data into the gating layer; The gating layer calculates the activation probability of each individual component of the neural network; Select a subset from a plurality of independent component neural networks, the subset corresponding to a predetermined number of independent component neural networks with the highest activation probability; Gating compressed motion data to this subset of independent component neural networks; and Calculate the weighted sum of the outputs of the independent component neural networks of the subset, and use this weighted sum as the output of the SMoE layer, where the weighted sum represents the sum of the outputs of the independent component neural networks of the subset after being weighted by the associated normalized gating value.

[0050] The preset number of independent component neural networks is a trade-off between the required database size (i.e., the amount of motion data to be processed by the data encoding model), the computational load, and the desired accuracy of the reconstructed or output motion data. In some embodiments, the preset number of independent component neural networks in the subset is two. However, other numbers are also possible.

[0051] It has been observed that implementing a sparse hybrid expert neural network architecture in interactive motion data processing methods can significantly reduce the size of independent components and connections in the neural network data encoding model compared to conventional architectures that do not use the SMoE architecture, while maintaining or at least providing substantially equivalent quality.

[0052] As a result, computational complexity or CPU complexity, as well as memory usage, can be significantly reduced. It has been observed that, in terms of memory usage, linear or even quadratic performance improvements are achieved compared to conventional methods that do not apply SMoE technology.

[0053] It has been observed that applying the sparse hybrid expert architecture disclosed in this paper to neural network data encoding models or autoencoders can increase the amount of data that the method can represent, while reducing memory and CPU requirements, compared to motion synthesis methods based on motion matching and neural networks.

[0054] In some embodiments, the method may include decoding data through multiple SMoE layers in the decoder section. While a single SMoE layer has been observed to be sufficient (e.g., to reconstruct motion data that meets a preset quality level), an SMoE architecture may include stacked SMoE layers. The number of layers selected may depend on the complexity of the function to be performed.

[0055] The method also advantageously includes: storing compressed motion data output from the encoder output layer as stored compressed motion data, the compressed motion data including multiple compressed motion data frames; and further includes: Receive input control data; Select a compressed motion data frame from the stored compressed motion data based on the input control data; and The selected compressed motion data frame is input to the decoder input layer, and then the selected motion data frame is guided, decoded, decompressed and output.

[0056] The input control data may include user input control data and information about the virtual object controlled by the user, wherein the user input control data includes information related to at least one of the following: the action to be performed on the virtual object or the action that the virtual object will perform, the movement that the virtual object will perform, and the orientation change that the virtual object will perform.

[0057] The input control data may include user input control data and information about the virtual object controlled by the user. The user input control data includes information related to at least one of the following: the action to be performed on the virtual object or the action that the virtual object will perform, the movement that the virtual object will perform, and the orientation change that the virtual object will perform.

[0058] According to this method, the motion data frame that best matches the input control data, namely the attitude and related features X, can be found.

[0059] It can predict future motion data frames representing motion data at future points in time based on selected compressed motion data and user input control data, and input these future motion data frames into the decoder input layer.

[0060] Compressed motion data or future motion data frames can be concatenated with additional control data, and the concatenated data can be input to the decoder input layer.

[0061] It has been observed that by preserving the linear relationship between consecutive motion data frames using the neural network model described in this paper, future frames of motion data can be predicted over a longer time period, further reducing the need to resample to a known position in this space. This, in turn, reduces the CPU overhead of other components used in this method.

[0062] Those skilled in the art will understand that the different steps and features described above with respect to the first aspect can be combined with each other in various ways.

[0063] The method according to the first aspect can be implemented by the system according to the second aspect as described below.

[0064] In a second aspect, a motion data processing system is provided, the system including a neural network data encoding model for processing motion data, the neural network data encoding model including an encoding part and a decoding part, the system further including: The processor is configured to control the processing of motion data by the neural network data encoding model; Neural network data encoding models include: The encoder section includes multiple encoder layers connected in sequence, and each encoder layer includes: An encoder input layer is configured to receive source motion data; at least one continuous representation generation layer is configured to transform the source motion data into compressed motion data based on a predefined continuous function; and an encoder output layer is configured to output compressed motion data. The decoder section includes multiple decoder layers connected in sequence. Each decoder layer includes: The system comprises a decoder input layer for receiving compressed motion data; a decoding layer configured to decode the compressed motion data into decoded compressed motion data, the decoding layer including multiple independent component neural networks, each of which is connected to the decoder input layer to receive and process the compressed motion data and output the decoded compressed motion data; a decompression layer configured to decompress the decoded compressed motion data; and an output layer configured to output the decompressed decoded motion data as output motion data.

[0065] The system according to the second aspect can be configured to perform the method described above according to the first aspect. The system according to the second aspect can be configured to perform any combination of the steps and features described in the first aspect.

[0066] The features and terms used in connection with the second aspect have the same definitions, meanings, functions, and effects as those described above with respect to the first aspect. Therefore, they will not be repeated below.

[0067] At least one continuous representation generation layer is preferably configured as a Lipshitz continuous representation layer, configured to transform source motion data using Lipshitz continuous transform to form a Lipshitz continuous representation of the motion data.

[0068] Each of at least one Lipshitz continuous representation layer can be configured as a linear layer with an associated learnable Lipshitz constant c for determining the weight normalization to be used by the Lipshitz continuous representation layer.

[0069] Multiple Lipschitz continuum representation layers can be configured.

[0070] The decoding layer is preferably configured as a sparse hybrid expert (SMoE) layer comprising multiple independent component neural networks.

[0071] When the decoding layer is configured as an SMoE layer, the decoder input layer can be a gated layer, and each of the multiple independent component neural networks is connected to the gated layer to receive compressed motion data from the gated layer and is connected to the output layer.

[0072] When using this type of SMoE architecture, the processor can be configured to control the decoder section: Input the compressed motion data into the gating layer; The gating layer calculates the activation probability of each individual component of the neural network; Select a subset from multiple independent component neural networks, which corresponds to a predetermined number of independent component neural networks with the highest activation probability, and have each independent component neural network in the subset process the compressed motion data. Calculate the weighted sum of the outputs of the independent component neural networks of the subset, and use this weighted sum as the output of the SMoE layer, where the output is represented by the sum of the outputs of the independent component neural networks of the subset after being weighted by associated normalized gating values.

[0073] The system may also include an interface for receiving input control data and a storage device (such as a memory) for storing compressed motion data output from the encoder output layer. The processor is also configured to select a compressed motion data frame from the stored compressed motion data and input the selected compressed motion data frame to the decoder input layer, and then control the guiding, decoding, decompression and output steps to be performed based on the selected motion data frame.

[0074] The input control data may include user input control data and information about the virtual object controlled by the user. The user input control data includes information related to at least one of the following: the action to be performed on the virtual object or the virtual object to be briefly performed, the movement to be performed by the virtual object, and the orientation change to be performed by the virtual object.

[0075] The processor can also be configured to receive predicted future motion data frames representing motion data at future points in time, as well as additional input control data.

[0076] The processor can also be configured to concatenate compressed motion data or future motion data with additional control data and input the concatenated data to the encoder input layer.

[0077] The system may also include a neural network prediction model configured to predict future motion data frames at future points in time based on coded compressed motion data and user control input data. The neural network prediction model is also preferably configured to generate additional input control data.

[0078] Those skilled in the art will understand that the steps and features described above can be combined in various ways.

[0079] In a third aspect, a method is provided for generating a neural network data encoding model for processing motion data, the data encoding model including an encoder part and a decoder part, the method comprising the following steps: The encoder section is initialized by sequentially connecting multiple encoder layers, each encoder layer including: an encoder input layer for receiving source motion data; at least one continuous representation generation layer configured to transform the source motion data into compressed motion data based on a predefined continuous function; and an encoder output layer for outputting the compressed motion data. The decoder section is initialized by sequentially connecting multiple decoder layers. Each decoder layer includes: a decoder input layer for receiving compressed motion data; a decoding layer configured to decode the compressed motion data into decoded compressed motion data, the decoding layer including multiple independent component neural networks, each of which is connected to the decoder input layer to receive the compressed motion data and output the decoded compressed motion data; a decompression layer configured to decompress the decoded compressed motion data; and an output layer configured to output the decompressed decoded motion data as output motion data. Train the neural network data encoding model until the matching degree between the source motion data input to the data encoding model and the resulting output motion data (representing the source motion data processed by the data encoding model) meets the preset quality level.

[0080] The features and terms used in connection with the third aspect have the same definitions, meanings, functions, and effects as those described above with respect to the first and second aspects. Therefore, they will not be repeated below.

[0081] The method according to the third aspect can be used to generate neural network data encoding models that are used in the method according to the first aspect and / or included and used in the system according to the second aspect.

[0082] The neural network data encoding model contained in the system according to the second aspect can be generated using the method according to the third aspect.

[0083] The method according to the third aspect may use, include, or incorporate any of the features, steps, and / or details described above with respect to the first and second aspects.

[0084] The method may further include configuring at least one continuous representation generation layer as a Lipshitz continuous representation layer, configured to transform source motion data using Lipshitz continuous transform to form a Lipshitz continuous representation of the motion data.

[0085] The method may also include forming each Lipshitz continuous representation layer by constructing linear layers with associated learnable Lipshitz constants c, and using the Lipshitz constants c to determine the weight normalization used by the Lipshitz continuous representation layer.

[0086] The method may further include configuring the decoding layer as a sparse hybrid expert (SMoE) layer; and wherein the decoder input layer is a gated layer. The method further includes constructing the SMoE layer by forming multiple independent component neural networks, each of which is connected to the gated layer to receive compressed motion data from the gated layer. The method also includes: Input the compressed motion data into the gating layer; The gating layer calculates the activation probability of each individual component of the neural network; Select a subset from a set of independent component neural networks, which corresponds to a predetermined number of independent component neural networks with the highest activation probability; Calculate the weighted sum of the outputs of the independent component neural networks of the subset, and use this weighted sum as the output of the SMoE layer, where the output is represented by the sum of the outputs of each independent component neural network of the subset after being weighted by associated normalized gating values.

[0087] The training process may include: inputting source motion data into the encoder input layer; processing the motion data by the data encoding model; receiving output motion data from the decoder output layer, wherein the output motion data is decompressed and decoded motion data; comparing the output motion data with the source motion data to evaluate the degree of matching between the output motion data and the source motion data, and calculating the quality level of the output motion data; the method further includes repeated iterative training until the quality level of the output data reaches a preset standard.

[0088] The training can be performed using methods well-known in the field of neural networks. Through different iterations, the internal weights of each layer in the neural network model can be updated using a standard deep learning process involving backpropagation, based on the gradients calculated from the "loss value". The weight update method and update criteria can be set according to principles well-known in the field, such as relevant methods in backpropagation theory. The loss value, in numerical form, reflects the quality of a specific iteration or a subset of all source training data.

[0089] The training of the neural network model can be run or iterated repeatedly until a minimum number of steps is reached. This minimum number of steps can be set through testing to ensure that the iteration duration is sufficient to ensure that the average positional error of the features (e.g., bones in the character's pose) of the virtual character or object in the motion data frame is less than a preset value. For example, in applications involving virtual characters, this preset value can be set to achieve a character pose bone reconstruction accuracy of 1 cm or less. This preset value can serve as the primary quality evaluation indicator for end users.

[0090] One or more training steps, such as the calculation and / or evaluation of the matching degree between the source motion data and the motion data output by the neural network data encoding model during training (i.e., reconstructed motion data), may be performed by the same or different processors as the processors that perform the encoding and / or decoding steps.

[0091] The training may further include training the neural network data encoding model with temporally continuous pairs of motion data frames.

[0092] When training an autoencoder with temporally continuous paired pose samples, various well-known methods can be used to calculate the additional loss to minimize the velocity difference between the training data and the prediction results in the local space and the model space. In one example, this loss can be expressed as:

[0093] in, These represent two raw motion data frames, or attitudes, at times t0 and t1, respectively.

[0094] This represents the corresponding motion data frame in the output data, i.e., the output result of the neural network after reconstructing the motion data frame (or posture). Based on this, the velocity "t" observed between t0 and t1 in the original motion data can be calculated. The design idea is to minimize the velocity difference between the original motion data and the output reconstructed from the neural network model. Its core logic lies in guiding the neural network to learn and generate velocities similar to the posture of the original data, achieving smoother motion data reconstruction and a higher degree of matching with the source motion data.

[0095] This allows for the assessment of overall speed by combining predictions of future frames.

[0096] Those skilled in the art should understand that the above steps and technical features can be combined in many different ways.

[0097] Fourthly, a computer-readable medium is provided, the medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8 or claims 19 to 24.

[0098] As can be seen from the above, by combining the encoding of motion data using Lipschitz continuity-based neural network modeling techniques with the decoding using sparse hybrid expert methods, a neural network data encoding model (also known as an autoencoder) is advantageously realized.

[0099] In summary, it has been observed that the methods and systems disclosed herein can significantly reduce CPU computational complexity and required storage space, thereby improving performance in terms of both CPU computation and memory requirements. This invention enables high-speed, high-precision user-system interaction, including dynamic character control with high-quality animated synthesis, such as video game applications involving multiplayer games across different sites and a large number of associated virtual characters.

[0100] Furthermore, it has been observed that predictive neural network models employing the neural network autoencoder disclosed in this invention (e.g., animation data processing systems) can learn and predict future frames of motion data represented in a virtual environment with higher quality compared to currently known methods during interactive motion data processing. Attached Figure Description

[0101] Other technical features and advantages of the present invention will become clearer through the description of non-limiting and non-exclusive embodiments. These embodiments should not be construed as limiting the scope of protection. Those skilled in the art will understand that other alternative and equivalent embodiments can be designed and implemented without departing from the scope of protection of the present invention. The embodiments of the present invention will be described in conjunction with the accompanying drawings, wherein the same or similar reference numerals denote the same, similar, or corresponding parts, as follows: Figure 1 This is a flowchart of the motion data processing method described in an embodiment of the present invention; Figure 2 The embodiments of the present invention can be applied to Figure 1 A schematic diagram of the neural network data encoding model of the method; Figure 3 This is an embodiment of the present invention. Figure 2 The diagram shows the structure of the decoder part in the data encoding model shown. Figure 4 This is a flowchart of a motion data processing method according to another embodiment of the present invention; Figure 5The embodiments of the present invention can be applied to Figure 4 A schematic diagram of the neural network data encoding model of the method; Figure 6 The embodiments of the present invention can be used for Figure 5 A schematic diagram of the neural network architecture of the decoder part of the data encoding model; Figure 7 This is a flowchart of the sparse hybrid expert architecture execution method according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the motion data processing system described in an embodiment of the present invention; Figure 9 This is a schematic diagram of the motion data processing method described in an embodiment of the present invention; Figure 10 This is a flowchart of the neural network data encoding model construction method according to an embodiment of the present invention; Figure 11A , 11B This illustration demonstrates the beneficial effects of using the Lipschitz transform for motion data encoding in the embodiments of the present invention; Figure 12A , 12B This demonstrates the improved accuracy of animation data reconstruction achieved by the method described in the embodiments of the present invention. Detailed Implementation

[0102] In the following description, several exemplary embodiments of the aspects disclosed above will be described. The definitions, descriptions, and details disclosed in the above summary of the invention are applicable by analogy and therefore need not be repeated.

[0103] Figure 1 A flowchart illustrating a method for motion data processing using a neural network data encoding model according to a non-limiting embodiment is shown. Figure 1 The neural network data encoding model corresponding to the method shown is in Figure 2 The diagram is schematic. Figure 2 Details of the decoder used are in Figure 3 It is shown schematically in the middle.

[0104] Figure 1 The method illustrated herein includes the following steps: S101: Input source motion data into the encoder input layer of the encoder part of the data encoding model.

[0105] Motion data processing methods can be executed during application runtime, such as in an interactive application that displays motion data on one or more display devices. This motion data may include, for example, pose and related data, or a pose information dataset referred to as the currently displayed animation data. Users can input user control data to control one or more actions related to the object or virtual character represented in the currently displayed animation data.

[0106] S102: The source animation data is processed by at least one continuous representation generation layer in the encoder section. The continuous representation generation layer transforms the input animation data into compressed animation data based on a predefined continuous function. Through this continuous function, the layer transforms the source animation data into compressed animation data, that is, into animation data in the latent space.

[0107] This transformation can be performed by a single layer or sequentially through multiple continuous representation generation layers, each associated with a different continuous function. One or more continuous representation generation layers can be linear layers, each coupled with or equipped with an activation function, preferably the GELU activation function or a similar function known in the art. Linear layers can also be referred to as fully connected layers. The activation function can be selected from activation functions known in the art. The last continuous representation generation layer can be a purely linear layer, described as a linear layer with an identity activation function. The nature of the final output determines which activation function is used for a specific group of neurons in the last layer. Preferably, a sigmoid activation function is used for specific neurons in the last layer where the expected output probability is in the range of 0-1, while no activation function is applied to the last layer when the expected output is in the real number domain. For example, if a neuron needs to output a probability, the activation function can be a sigmoid function; if the output needs to be in the interval [0, +∞), the activation function can be GELU / ReLU or a similar function. If the expected output is all real numbers, no activation function is set, or equivalently, an identity activation function is used.

[0108] In a preferred embodiment of this disclosure, the predefined continuous function is preferably a Lipshitz continuous function. One or more continuous representation generation layers may then be referred to as Lipshitz continuous representation generation layers, or Lipshitz layers.

[0109] S103: Compressed motion data is output from the encoder output layer of the encoder section. The encoded compressed motion data can also be referred to as latent data and constitutes part of the latent space. As further described above, the motion data typically includes data related to pose or role Z and related features X, and can be implemented using...<X,Z> Formatted storage.

[0110] Encoding can be performed on multiple motion data frames, each typically associated with a character pose and related features. Therefore, motion data can include pose information datasets, where each pose information dataset can include information associated with a single animation frame in 3D space. These can also be referred to as animation frames or animation data. The encoded data for all frames can be stored in memory and / or a database. In this way, motion data can be encoded offline using the encoder portion of the data encoding model before running the application or in a manner independent of application operation. Encoding motion data...<X,Z> Stored in an encoded format that represents <features, latent variables>.

[0111] Steps S101 to S103 can be referred to as encoding steps.

[0112] In a preferred embodiment, the method is an interactive method and further includes steps S104 and S105.

[0113] S104: Receive input control data. As described above, this may include user input data for interactively processing motion data. Input control data typically includes user input, or is at least generated based on user input, for example, for controlling or influencing motion data in a virtual environment so that the system user can interactively control virtual characters or other objects in the virtual environment. For example, the user may input control data indicating one or more of the following: the action the virtual character will perform, the direction the user wants the virtual character to move, the motion the virtual character will perform, the direction of movement of the virtual character, etc.

[0114] S105: Based on the input control data, select a motion data frame from the encoded motion data in the latent space. During application runtime, the pose can be reconstructed from the feature and pose latent data generated by the model encoder portion through the decoder portion. Feature X is preferably related to attributes that can be used to control motion, such as the current / future velocity and position of the virtual character or object to be controlled in a virtual environment. This trajectory can be at least partially controlled by the user running the application. The user can input control data through a user interface known in the art (e.g., a game controller, console, etc.). Therefore, the current feature vector X can be influenced by user input, i.e., changed.

[0115] When no user input is available, motion data output from the encoder output layer can be stored in memory and retrieved, then directly or indirectly passed to the decoder input layer for decoding. In embodiments running applications that display motion data (e.g., animation data) and continuously update the data, the system can use a projector model to automatically estimate the optimal features X and latent variables for the next motion data frame or animation frame, and a stepper model can apply forward kinematics to the data of the selected motion data frame and input it to the decoder section. This will be referred to below. Figure 8 and Figure 9 Further detailed description.

[0116] S106: Input compressed motion data into the decoder input layer of the decoder part of the data encoding model, and guide it through the decoder input layer to the decoder layer containing a neural network with multiple independent components.

[0117] Therefore, the decoding layer can also be referred to as a nonlinear layer or a non-fully connected layer. In all embodiments disclosed herein, the decoding layer may be a sparse layer. In all embodiments, the decoding layer is preferably a sparse hybrid expert layer.

[0118] S107: Decodes compressed motion data from each of multiple independent component neural networks, with each independent component neural network generating its own decoded compressed motion data. Each independent component neural network is connected to the decoder input layer.

[0119] S108: The independently decoded compressed motion data generated by each independent component neural network is merged into decoded compressed motion data. Although Figure 1 The motion data is illustrated by processing two independent component neural networks, but depending on the complexity of the motion data to be decoded, other numbers of independent component neural networks may be used.

[0120] S109: The decoded compressed motion data is decompressed by the decompression layer of the decoder section. Decompression can be performed by a single decompression layer or by multiple sequentially connected decompression layers. The decompression layer is preferably a linear layer with an activation function, which can be similar to the activation function described above for the continuous representation generation layer.

[0121] S110: Outputs decoded and decompressed motion data as output motion data. The output motion data can also be referred to as reconstructed motion data or reconstructed pose data.

[0122] Steps S106 to S110 can be referred to as decoding steps or decoding methods.

[0123] Therefore, using this method, source animation data frames can be input to the encoder section, encoded into implicit form, and then decoded into reconstructed animation data frames. For example, raw pose data can be encoded into implicit pose data, and then decoded to obtain reconstructed pose data. As described, the implicit data may include multiple animation data frames, i.e., multiple pose data related to different poses. Based on input control data (e.g., user control data related to the expected motion and / or action to be performed on the pose), one of the multiple implicit animation data frames can be selected for decoding. The animation or motion data retrieved from the implicit space and input to the decoder section can be concatenated or combined with additional data.

[0124] Figure 2 This schematically illustrates a neural network data encoding model 200 according to an embodiment of the present invention, which can be executed... Figure 1 The method is applied when the data encoding model 200 is also referred to as an autoencoder, which includes an encoder part 210 and a decoder part 220. The encoder part 210 can be configured to execute steps S103 to S105, and the decoder part can be configured to execute reference... Figure 1 Steps S106 to S110 are described above.

[0125] The encoder and decoder parts of the data encoding model each include at least the following layers.

[0126] The encoder section 210 includes at least the following layers.

[0127] The encoder input layer 211 is configured to receive source motion data. The motion data may include pose data (i.e., data related to the pose or state of a virtual character or object) and feature data related to the pose data.

[0128] The continuous representation generation layer 212 is configured to transform source motion data into compressed motion data based on a predefined continuous function. The continuous representation generation layer transforms the input motion data according to the predefined continuous function. At least the attitude data can undergo this transformation.

[0129] Although Figure 2 The illustration shows a single continuous representation generation layer, but this is only an example. Multiple sequentially connected continuous representation layers can be included.

[0130] The encoder output layer 213 is used to output compressed motion data. This compressed motion data is output to the latent space and can be referred to as latent motion data. The latent motion data includes attitude data and related feature data.

[0131] The decoder section 220 includes at least the following layers.

[0132] The decoder input layer 221 is used to receive compressed motion data. Specifically, compressed motion data frames can be input to the decoder, including compressed attitude data and related feature data.

[0133] In some embodiments, as discussed in this disclosure, compressed motion data may be concatenated with additional input control data (e.g., motion matching features M) before, during, or after input to the decoder input layer. Alternatively, future motion data frames may be predicted based on the compressed motion data and optionally concatenated with additional input control data.

[0134] Motion matching features are typically derived from source motion data. Motion matching features may include the position of several features or parts of a virtual character or object to be controlled, i.e., the object whose movement, orientation, and / or other actions will be affected by user input data. Additionally, control parameters may be included, which control how the character or object will perform actions at future moments without being interrupted or affected by user control data. In the example of a virtual character, motion matching features may include the position and velocity of a selected number of bones (e.g., feet) in model space or forward kinematic space.

[0135] Decoding layer 222 is configured to decode compressed motion data into decoded compressed motion data. The decoding layer comprises multiple independent neural network components, each connected to the decoder input layer to receive and process the compressed motion data and output decoded compressed motion data. The decoding layer in... Figure 3The diagram is schematic.

[0136] Decompression layer 223 is configured to decompress the decoded compressed motion data. Although only one decompression layer 223 is shown in the figure, multiple decompression layers 223 connected in sequence can be provided. These layers are preferably configured as linear layers with activation functions as described above.

[0137] Output layer 224 is configured to output decompressed and decoded motion data as output motion data.

[0138] exist Figure 2 In the embodiments described, the decoding layer 222 may employ, for example... Figure 3 The architecture implementation is shown.

[0139] like Figure 3 As schematically shown, the decoding layer 222 is configured as layer 322, which includes multiple independent component neural networks 322a and 322b, and is configured to process compressed motion data input to the decoder section via the decoder input layer 321. In the summing layer 330, the outputs of the individual component neural networks 322a and 322b are combined, for example, using the method described in the following reference. Figure 6 The weighted summation form described in summation layer 623.

[0140] Figure 4 A flowchart illustrating a method for interactive motion data processing using a neural network data encoding model, according to another non-limiting embodiment, is shown. This method can be used... Figure 4 The method of neural network data encoding model in Figure 5 The diagram is schematic. Figure 4 Details of the decoder used are in Figure 6 It is shown schematically in the middle.

[0141] according to Figure 4 The method steps largely correspond to the reference Figure 1 The methods discussed here do not need to be discussed in detail again for those directly corresponding to them.

[0142] Figure 4 The method illustrated herein includes the following steps: S401: Input source motion data into the encoder input layer of the encoder part of the data encoding model; S402: The source motion data is processed by at least one Lipshitz continuous representation generation layer, thereby transforming the input motion data into compressed motion data based on the Lipshitz continuous function. In all embodiments of this disclosure, the motion data can be encoded through successive transformations by multiple Lipshitz continuous representation generation layers.

[0143] S405: The encoder output layer of the encoder section outputs compressed motion data.

[0144] and Figure 1 The method is similar, and may optionally include steps S404 and S405: S404: Receive input control data.

[0145] S405: Based on the input control data, select coded motion data from the hidden data (i.e., the coded motion data previously generated by the encoder portion), specifically coded motion data frames.

[0146] S406: The compressed motion data is input to the decoder input layer of the decoder section and directed through the decoder input layer to a subset selected from multiple independent component neural networks of the decoder layer, which is configured as a sparse hybrid expert (SMoE) layer.

[0147] S407: Decode compressed motion data from each of a subset of multiple independent component neural networks to generate multiple independent decoded compressed motion data, wherein each independent component neural network is connected to the decoder input layer; and merge the multiple independent decoded compressed motion data into a single decoded compressed motion data. Steps S406 and S407 can be performed by the Sparse Hybrid Expert (SMoE) layer, which will refer to Figure 5 and Figure 6 Further detailed discussion is needed.

[0148] S408: The decoded compressed motion data is decompressed by the decompression layer of the decoder section. In all embodiments of this disclosure, decompression can be performed by multiple sequentially connected decompression layers.

[0149] S409: Output decoded and decompressed motion data as output motion data.

[0150] Figure 5 This schematically illustrates a neural network data encoding model 500 according to an embodiment of the present invention, which can be executed... Figure 4 The method is applied when needed. The data encoding model 500, also known as an autoencoder, includes an encoder section 510 and a decoder section 520. The encoder section can be configured to perform... Figure 4 In steps S403 to S405, the decoder section can be configured to perform... Figure 4 Steps S406 to S410.

[0151] Data encoding model 500 includes at least the following layers.

[0152] The encoder section 510 includes at least the following layers.

[0153] The encoder input layer 511 is configured to receive source motion data.

[0154] One or more Lipshitz continuous representation generation layers 512 are configured to transform source motion data into compressed motion data based on a Lipshitz continuous function. Preferably, multiple sequentially connected Lipshitz continuous representation generation layers 512 are provided, wherein each layer 512 further includes an activation function for activating subsequent layers 512. Each layer 512 may be associated with its own Lipshitz constant c. Furthermore, except for the last layer 513, each Lipshitz layer 512 is provided with an activation function, as described above.

[0155] Although Figure 5 The diagram shows five Lipschitz layers, but this is only an example. More or fewer Lipschitz successive layers 512 can be set.

[0156] The encoder output layer 513 is used to output compressed motion data. In the example shown, the last Lipschitz layer is configured as the output layer.

[0157] The decoder section 520 includes at least the following layers.

[0158] The decoder input layer 521 is used to receive compressed motion data.

[0159] In some embodiments, as discussed herein, compressed motion data may be concatenated with additional input control data (e.g., motion matching features M) before, during, or after input to the decoder input layer. Alternatively or additionally, future motion data frames may be predicted based on the compressed motion data and optionally concatenated with additional input control data before input to the decoder input layer.

[0160] Decoding layer 522, configured to decode compressed motion data into decoded compressed motion data, is configured as a Sparse Hybrid Expert Layer (SMoE layer). The SMoE layer and its architecture are described in... Figure 6 The diagram illustrates this. The SMoE layer comprises multiple independent component neural networks, where only a subset is activated to process compressed motion data.

[0161] Decompression layer 523 is configured to decompress the decoded compressed motion data. Although two decompression layers 523 and 524 are shown in the figure, different numbers of sequentially connected decompression layers 223 can be configured. These layers can be configured as linear layers and activation functions, as described above.

[0162] Output layer 524 is configured to output decompressed and decoded motion data as output motion data. In the example shown, the last layer 524 can act as both a decompression layer and an output layer.

[0163] As discussed, configuring the coding layer to provide a Lipshitz continuous representation of the input motion data has proven particularly advantageous and offers improved coding properties. The temporal relationships between consecutive data frames are preserved, and the linearity and consistency between the original space and the compressed space (also known as the hidden manifold) are maintained. Improved distribution of data frames in the hidden manifold has been observed after applying the Lipshitz technique to the encoder section. These effects enable the use of sparse layers (e.g., Sparse Hybrid Expert (SMoE) layers) as decoding layers in the decoder section.

[0164] Each Lipshitz continuous representation layer 512 is advantageously configured with an associated learnable Lipshitz constant c, which is associated with the weight normalization to be used by the Lipshitz continuous representation layer.

[0165] The encoder portion 510 of the data encoding model 500 can be configured to receive input motion data (X, FK(X), C) and encode it into a compressed representation of the motion data, i.e., latent variables z. Here, X may represent the pose of a virtual character or the state of an object in local space, i.e., in the source motion data; FK(X) represents the kinematic data associated with that pose, typically including components related to the rotation, translation, and position of joints involved in the virtual character or object; and C represents parameters associated with several labels or logical markers accompanying the motion data. For example, these may be markers indicating one or more effects associated with the motion data. For instance, when a virtual character's footstep is detected, and / or secondary effects such as sound or logical actions should be able to be triggered according to the motion data processing of this disclosure.

[0166] The SMoE layer is configured to receive a latent variable z generated by the encoder portion, which can be combined with additional input control data, including motion matching features M. As described above, these features can be extracted from training data and / or received along with user input for interactive control of the reproduction of motion or animation data on a display device, such as in a virtual environment, like during a simulation or game. The SMoE layer processes the input data and generates a decoded but still compressed representation of the motion or animation data, which is then fed into the first linear layer. The linear layers process the data to decompress it sequentially into a decompressed representation of the motion data ( ).

[0167] Therefore, the data input to the neural network data encoding model can be a combination of poses in local space X, model space evaluations after applying positive kinematics EK(X), and any applicable labels (such as foot contact of a virtual character or events or effects associated with motion data).

[0168] The output of the data encoding model can include pose predictions in the local space C, including root motion velocities, and for each pose... The arbitrary probability of any related marker.

[0169] Each Lipshitz continuum preferably has its own associated Lipshitz constant c, which is used to determine the weight normalization to be used in the Lipshitz continuum. That is, the constant c is used to normalize the weight matrix.

[0170] The constant c is learned through the training process. During the operation of the trained network model, the constant c remains unchanged.

[0171] By enforcing Lipshitz continuity to neural network layers, the similarity of a given pair of input data can be preserved in the reconstructed data output from the data encoding model.

[0172] This paper introduces some general concepts in the theory behind Lipshitz layers. Those skilled in the art will understand that these algorithms can be implemented using different computational methods and procedures. It should be understood that these concepts and definitions apply to all embodiments of this disclosure.

[0173] The function f is defined as Lipschitz continuous when the constant c satisfies the following condition:

[0174] As mentioned above, the Lipschitz constant c is learned through training the data encoding model. This constant is used to normalize the weight matrix W with rows R, for example, using a scaling vector created by the function Scale:

[0175]

[0176] Softplus is a known activation function in the field of neural networks. Softplus activates a floating-point input. It forces the output to be positive, but its smoothing effect produces an initial non-linear output before reaching a threshold input value, after which the output becomes linear. For example, it can be represented as: or

[0177] Subsequently, when interchangeing the Lipschitz layer with a linear model with bias b, weight normalization can be applied:

[0178] During runtime, the constant c remains unchanged, so when preparing the model, the normalization of the weight matrix within the model can be evaluated as a constant.

[0179] Figure 6A schematic diagram of the architecture relating to the sparse hybrid expert layer 522 and related layers according to an embodiment of the present disclosure is shown. This architecture can be advantageously applied to Figure 4 Methods and Figure 5 Data encoding model.

[0180] Figure 6 The sparse hybrid expert architecture 600 shown is preferably applied to the decoder portion 520 of the neural network autoencoder 500, for example, referencing Figure 5 The decoder portion described above can also be applied to components of neural network prediction models, such as the neural network stepper model and neural network projector model also described in this disclosure.

[0181] According to the Sparse Hybrid Expert (SMoE) architecture 600 disclosed herein, applied to various neural network models, it can be conceptually viewed as replacing a large neural network layer with multiple smaller, independent neural networks (called experts). In embodiments, these networks behave as feedforward networks (FFNs). Due to the significant reduction in the number of neural network node connections, the application of the sparse hybrid expert layer 622 achieves the distribution of processing load, i.e., the processing of queries is distributed to a relatively small subset of the multiple experts. This reduces processing complexity and shortens processing time.

[0182] The SMoE architecture 600 includes a gating layer 621, an SMoE layer 622, and a summing layer 623.

[0183] The SMoE layer 622 includes multiple independent component neural networks 622-1, 622-2, ..., 622-n. These can also be referred to as experts and can be considered as functionally parallel arrangements. For ease of illustration, Figure 6 The diagram shows five independent component neural networks. However, this number can typically be much larger. In one embodiment, the SMoE layer contains 128 independent component neural networks. During training, these networks can be trained to decode various types of implicit motion data, allowing parameters (such as weights and activation functions) from different experts to be optimized for different input motion data. This enables efficient load balancing.

[0184] The gating layer 621 receives input x, calculates the gating value for each expert, and routes input x to the expert with the best matching degree selected from multiple experts for processing. Figure 6 In one embodiment, two experts are selected. In an exemplary embodiment, two are selected from a pool of 128 experts. This number has been observed to provide an acceptable trade-off between model accuracy and runtime performance. However, other numbers may also be used.

[0185] Each expert calculates their respective normalized gating values ​​y2 and y4. The final result y output from the SMoE layer is calculated as a weighted sum of the outputs of the selected experts.

[0186] When used in an autoencoder, the output y is then processed by linear layers 224a and 224b to reconstruct the original pose dimension in layer 623.

[0187] This section introduces some general concepts in the theory behind Sparse Hybrid Experts (SMoE) techniques. Those skilled in the art will understand that these algorithms can be implemented using different computational methods and procedures.

[0188] The gating value can be calculated as a probability distribution over N experts set in the SMoE layer. The expert with the highest gating value is selected to receive and process the input x.

[0189] In mathematical terms, gate layer 621 takes motion data x as input and passes it to the expert set. The selected number of experts with the best matching degree.

[0190] The gating logic value h(x) = LINEAR(x) is obtained by processing the input through a linear layer. The final gating value is obtained through a softmax operation, which generates a probability distribution on the expert. Here, y = Linear(x) can be expressed as follows: Where A and b are learnable parameters (weights). The probability or gate value of each expert i can be represented by the softmax function:

[0191] The input x is then passed to the expert with the highest gating value. In the illustrated embodiment, the input is routed to the top two gating values. The final result is calculated as a weighted sum of the outputs of the two selected experts, with the weights being their associated normalized gating values:

[0192] Here, index 1st and index 2nd refer to the indices of the highest and second highest gating values ​​calculated for each of the multiple experts according to the above probability equation.

[0193] Figure 7 The following is illustrated according to an embodiment of the present disclosure: Figure 6 The flowchart illustrates the method executed by the SMoE architecture 600. This method is applicable to all embodiments of this disclosure that use SMoE layers. It can be applied not only to neural network data encoding models, but also to neural network stepper models and neural network projector models that can jointly constitute a prediction model. Regarding... Figure 7This method is described as being applied to the decoder portions 520 and 620 of the data encoding model. The SMoE layers contained in the stepper and projector models can operate in a similar manner. These steps can be performed based on the mathematical concepts explained above regarding the SMoE technique.

[0194] In step S701, compressed motion data (also known as latent variables) is input to the gating layer 310.

[0195] In step S702, the gating layer calculates a gating value or probability for each independent expert, and each corresponding expert processes the data based on this gating value or probability; then in S703, the expert with the highest gating value is selected. That is, the decoder performs a query using latent variables to find the weights and indices of a subset of independent component neural networks in the SMoE layer, to which the input should be forwarded. For example, this subset may include two independent component neural networks.

[0196] In step S704, the input is processed by the selected expert. That is, the component is executed.

[0197] In step S705, the outputs of the selected experts are weighted. The network calculates a weighted sum of the outputs of the independent component neural networks within the subset and uses this weighted sum as the output of the sparse hybrid expert layer. Therefore, the output of the SMoE layer represents the weighted sum of the outputs of the independent component neural networks within the subset, weighted by their associated normalized gating values.

[0198] Figure 8 An interactive motion data processing system 800 is schematically shown.

[0199] System 800 includes a neural network data encoding model for processing motion data, such as a reference Figure 2 or Figure 5 The data encoding models 200 and 500 are described above. The system also includes an interface 810 for receiving input control data (e.g., user input data), and a processor 820 configured to control the data encoding model, for example, according to a reference... Figure 1-3 or Figure 4-6The method described herein operates. Optionally, the system may further include a neural network prediction model 830, configured to compute predicted future motion data frames based on implicit data generated by the encoder portion of the neural network data encoding model, user input data, and motion matching features, and input these as a future representation of the implicit motion data to the decoder portion of the data encoding model. Motion matching features may be determined during neural network training and / or separately predetermined standard motion matching features, and may be stored in a database. Prediction model 830 may include a stepper model 850, configured to predict future motion data frames at future moments by applying motion matching features in consecutive time steps. Projector model 840 may be configured to project input control data into the latent space to select or acquire motion data frames that best match the input control data (e.g., user intent represented or derived from the input control data), such as including character pose and related features.

[0200] Figure 9 This illustration schematically depicts a method for real-time motion data processing using the system or model described in embodiments of the present invention. For example, Figure 9 The method shown can be obtained from the reference. Figure 8 The system is executed. For ease of explanation, the method is described in conjunction with a game, but it should be understood that this system and method can also be applied to other applications of real-time controlled motion data processing that predict future motion data frames based on user input.

[0201] This method can be adopted as follows: Figure 8 The system execution shown includes a prediction model 830 comprising a neural network stepping model 850 and a neural network projection model 840.

[0202] In steps 910-a to 910-e, the user queries appropriate motion data for the desired action using their user-input control data. In other words, the user inputs control data into the system, instructing the desired motion and / or action to be performed or applied to a virtual character or other object within a virtual environment visualized to the user and which the user wishes to control. Through this method, the user can continuously interact with the virtual environment and trigger various actions and events within it.

[0203] 910-a. A motion data processing system, particularly a neural network projector model, executes by means of queries representing user input control data. In a preferred embodiment, the projector model includes an SMoE architecture, which is consistent with a reference... Figure 6 The described architectures are similar.

[0204] 910-b. The query is converted into additional weights for a subset of components in the SMoE layer of the neural network projector to execute.

[0205] 910-c. Each independent neural network layer in this component subset is executed based on this query.

[0206] 910-d. Multiply the two results by their respective weights and then combine them.

[0207] 910-e. The resulting output is passed to the stepper model, which selects from the compressed motion data (or "latent variables") as the motion data frame that best matches the query, which was previously generated as the output of the encoder part of the data encoding model.

[0208] The neural network stepper model 830 is configured to perform relevant steps when the "latent variables" output by the encoder are used to obtain the final motion data after a future time X seconds.

[0209] 920-a. In this stepper model, the prediction results are called through several steps in an autoregressive manner until time X is reached; the stepper model preferably adopts an architecture similar to that of the projector model that performs steps 910a to 910e.

[0210] 920-b. For any remaining time less than a full frame, the result can be interpolated.

[0211] 920-c. Output the obtained prediction results, which are passed to the decoder section to reconstruct the motion data and fuse it into the data in the virtual environment.

[0212] The decoder section 520 of the data encoding model 500 performs relevant steps when the "latent variables" determined by the projector and the stepper are passed into the decoder section of the data encoding model to obtain the final motion data frame in the virtual environment, such as the final animation in the game, i.e., the animation frame at a certain future moment.

[0213] 930-a. The decoder performs the query on this latent variable to find the weights and indices of a subset (e.g., two components) of independent neural network components in the SMoE layer.

[0214] 930-b. Execute two components.

[0215] 930-c. Weight the outputs of the two separately.

[0216] 930-d. Return the merged result to the user.

[0217] 940. At this point, an animation frame can be played on any designated character for which the data was created.

[0218] Before use at runtime, i.e., before use in the interactive motion data processing described in the above embodiments, the neural network model described herein is trained, for example using backpropagation techniques known in the field of neural networks. This trains the model to learn to encode and decode source motion data into reconstructed motion data. During training, the weights of each independent layer are determined and optimized.

[0219] Figure 10 The present invention illustrates a non-limiting implementation of a method for constructing a neural network data encoding model according to embodiments of the present disclosure. References above are made to each other. Figure 2 and Figure 5 The neural network data encoding models 200 and 500 described can be generated using this method.

[0220] In step S1001, the encoder section is initialized by sequentially connecting multiple encoder layers, the multiple encoder layers including: an encoder input layer for receiving source motion data; at least one continuous representation generation layer configured to convert source motion data into compressed motion data based on a predefined continuous function; and an encoder output layer for outputting compressed motion data.

[0221] In step S1002, the decoder part is initialized by sequentially connecting multiple decoder layers. The multiple decoder layers include: a decoder input layer for receiving compressed motion data; a decoding layer configured to decode the compressed motion data into decoded compressed motion data, the decoding layer containing multiple independent component neural networks, each connected to the decoder input layer to receive the compressed motion data and output the decoded compressed motion data; a decompression layer configured to decompress the decoded compressed motion data; and an output layer configured to output the decompressed decoded motion data as output motion data.

[0222] In step S1003, the neural network data encoding model is trained until the matching degree between the source motion data input to the data encoding model and the output motion data reaches a preset quality level, wherein the output motion data represents the source motion data processed by the data encoding model.

[0223] In step S1001, at least one continuous representation generation layer can be configured as a Lipshitz continuous representation layer, which is configured to transform the source motion data using a Lipshitz continuous transformation to form a Lipshitz continuous representation of the motion data. Each Lipshitz continuous representation layer can be formed by constructing a linear layer with an associated learnable Lipshitz constant c, and using the Lipshitz constant c to determine the weight normalization method used by the Lipshitz continuous representation layer.

[0224] In step S1002, the decoding layer can be configured as a Sparse Hybrid Expert (SMoE) layer. The relevant architecture and functions of this layer have been referenced above. Figure 6 Describe it.

[0225] The following describes the details related to training neural network data encoding models.

[0226] During the training of the neural network data encoding model, the source motion data (X, FK(X), C) is input to the encoder, i.e., to the first Lipschitz continuous layer. This motion data is processed sequentially through each layer of the data encoding model, and then the reconstructed motion data is output. The reconstructed motion data is compared with the source motion data to assess or determine the degree of matching between the two, and the quality level of the reconstructed motion data is determined based on this assessment. This step can be performed in a variety of ways known in the art.

[0227] The training continues iteratively until the quality of the reconstructed motion data reaches a preset quality level. In other words, the training continues iteratively until the loss is minimized. This loss can be calculated using a loss function known in the art.

[0228] Through multiple iterations, the internal "weights" of each individual layer in a neural network model can be updated using standard deep learning procedures (e.g., backpropagation involving calculating the gradient of a cost function that is minimized relative to the model's trainable or learnable internal weights). This gradient is then fed into a minimization algorithm. Various algorithms known in the art can be employed, such as stochastic gradient descent, gradient descent, Adan, Adam, etc., which update the weights using the current gradient plus additional information including the learning rate. This additional information could be, for example, a weighted average of past gradients or solutions.

[0229] The methods and criteria for updating the weights can be set according to principles known in the field. These losses are numerically represented to indicate the performance of a specific iteration or a subset of the entire source training data.

[0230] The training of the neural network model can run or iterate repeatedly for a minimum number of steps, which can be set based on the test, ensuring that the iteration time is sufficient to keep the average positional error of each individual feature position (e.g., bones in a character pose) of the virtual character or object in the motion data frame less than a preset value. For example, in applications involving virtual characters, this preset value can be set to a bone reconstruction accuracy of 1 cm or less in the character pose. This preset value can serve as a primary indicator of the final quality for the end user.

[0231] One or more training steps, such as the calculation and / or evaluation of the matching degree between the source motion data and the motion data output by the neural network data encoding model (i.e., reconstructed motion or pose data) during training, may be performed by the same or different processors as the processors that perform the encoding and / or decoding steps.

[0232] The neural network is trained to reconstruct input motion data (e.g., pose) by first compressing it to a hidden representation z, then "decoding" the compressed representation to reconstruct the original pose from the hidden representation, and finally evaluating how well the two match. The neural network is trained continuously until the quality reaches an acceptable level for ideal quality when used in video games.

[0233] This neural network autoencoder model can be trained using pairs of temporally continuous motion data frames, which represent temporally continuous poses.

[0234] When training an autoencoder using time-continuous pairwise poses, the additional loss can be represented in several ways. This additional loss is used to minimize the difference between the local spatial velocity and the model spatial velocity in the training data and the prediction results.

[0235] For example, it can be represented as:

[0236] in, These represent two raw motion data frames (e.g., including pose) at times t0 and t1, respectively. This represents the corresponding motion data frame in the output data, i.e., the output result of the neural network after reconstructing the motion data frame (or posture). Based on this, the velocity "t" observed between t0 and t1 in the original motion data can be calculated. The design idea is to minimize the velocity difference between the original motion data and the output reconstructed from the neural network model. The basic idea is to guide the neural network to learn and generate velocities similar to the posture of the original data, achieving smoother motion data reconstruction and a higher degree of matching with the source motion data.

[0237] Therefore, the overall speed of combining the prediction results of future frames can be evaluated.

[0238] The training of this neural network can incorporate an additional loss term into the loss function to minimize the difference between the local spatial velocity and the model spatial velocity in the training data and the prediction results. This calculation can serve as a load balancer for the SMoE layer to ensure efficient utilization of the expert components.

[0239] For the Lipshitz layers in the encoder, the loss during training can be calculated as follows. Where N is the number of Lipshitz layers, and the cumulative product of the Lipshitz constants c of each layer can be calculated, expressed as softplus(c...). i This is then combined with the L1 and L2 indices and the velocity minimization loss on the latent variable values ​​z generated for each batch of B (weighted by λ1, λ2, and λ3 respectively) to generate the final regularized loss:

[0240]

[0241]

[0242] Regarding the SMoE architecture, during the training of the neural network autoencoder model, an auxiliary load balancing loss term can be optionally added to further improve the load balancing effect.

[0243] The following provides an example of this type of auxiliary load balancing loss term. However, it should be understood that other load balancing loss terms may also be used.

[0244] Given N experts, indices i=1 to N, and a batch B containing T batch entries, the auxiliary loss can be computed as a scaled dot product between vector f and P:

[0245] Where α is the relative weight of loss within the overall model cost function, fi is the proportion of batch entries assigned to expert i, and Pi is the probability proportion assigned to expert i by the gating layer.

[0246] This auxiliary loss promotes uniform routing of batch entries because the loss is minimized under uniform distribution when both vectors fi and Pi take values ​​of 1 / N.

[0247] In some embodiments, α can be set to 0.01 for all models.

[0248] The percentage of batch entries allocated to each expert i can be calculated as follows:

[0249] The percentage Pi can be calculated as follows:

[0250] Alternatively, a "soft constraint" method known to those skilled in the art can be used:

[0251] Where I(X) / |X| is the same as Pi in the above auxiliary load balancing description formula, there is no need to use fi again.

[0252] It should be noted that these two methods of representing auxiliary load balancing are merely examples of load balancing methods among experts in the SMoE layer, and various other functions may also be used.

[0253] During the training process of a neural network model, the model learns to efficiently distribute input data to a subset of independent component neural networks for a given query, and then merges the results of each independent component neural network within that subset.

[0254] By training neural networks to learn to specialize each independent component neural network (i.e., expert) in processing specific parts of the dataset and promoting their even distribution across the dataset, this approach can constrain the size of the network required to synthesize a single data sample at runtime, compared to other methods.

[0255] Therefore, the neural network as a whole can be smaller because this method allows for the specialization of smaller components without using a larger model with more redundant components, and it can be constrained to ensure that the neural network can learn sufficiently high-quality representations without consuming redundant memory and CPU resources compared to other methods.

[0256] This neural network stepper model can be trained in an autoregressive manner to predict the increment of each time step in the original value output, that is, to predict the increment value δz of the latent variable z in the encoder part of the output, as well as the output increment of the motion matching features M and δM.

[0257] Each step in the stepper increment calculation corresponds to a time step δt:

[0258] FPS stands for frames per second.

[0259] Training from spliced ​​input Starting with the sampled animation data frames, the μ-step is then predicted using an autoregressive approach until... .

[0260]

[0261] n Since the animation data is generally smooth, data can be sampled between data frames. The sampling method is similar to that of the original source animation, that is, sampling is performed at the following intervals:

[0262] The prediction loss and velocity similarity loss can be accumulated across all predictions n. The total loss can be estimated using the mean squared error calculated from all losses. This step can be performed using methods known in the art.

[0263] After the autoencoder model is trained, the stepper model and the projector model can be trained, preferably simultaneously. This allows for accurate and efficient training of the functional relationships between the two models.

[0264] Figure 11A , 11B This is an illustrative illustration of the use of the reference above. Figures 4 to 6 The beneficial effects achieved by the Lipschitz layer generating encoded data. Figure 11AThe distribution of latent variables is shown using the conventional encoder portion (without using the Lipshitz transform, i.e., without using the Lipshitz continuous representation generation layer). Figure 11B The diagram shows the latent space distribution obtained using the data encoding model and system of this disclosure, namely the distribution obtained by the Lipschitz continuous representation of the input motion data generated by the encoder layer.

[0265] As can be seen, the Lipshitz continuous transform can improve the distribution of latent variables, for example, by making the latent space denser. This effect is advantageous because it allows prediction of future motion data frames with a step size smaller than the time scale between consecutive motion data frames, meaning that future motion frames can be predicted with higher accuracy. The increased latent space density also makes the SMoE architecture of this disclosure applicable to data encoding models, stepper models, and projector models.

[0266] Figure 12A , 12B This illustration shows the application of the present invention (see reference). Figures 4 to 6 as well as Figures 8 to 9 The improved accuracy of the reconstructed motion data obtained in the above embodiment.

[0267] In each figure, the middle character represents the character pose in the source motion data. The character on the left represents the character pose obtained using a conventional model for encoding and decoding, which does not employ the Lipshitz Continuity and Sparse Hybrid Expert Architecture. The character pose on the right is obtained using the method disclosed herein, i.e., the character pose obtained through Lipshitz transform encoding and sparse hybrid expert architecture decoding. The lighter the grayscale color, the closer the reconstructed character is to the source data, i.e., the higher the reconstruction accuracy.

[0268] Figure 12A Indicates the currently displayed posture. Figure 12B This shows user input data based on instructions for the desired movement of the virtual character. Figure 12A The prediction result of the future attitude is calculated from motion data frames or attitude data.

[0269] It is evident that the method described in this disclosure provides reconstructed pose data with higher accuracy. This is in... Figure 12B This is particularly evident in the predicted postures shown.

[0270] Therefore, in summary, according to this disclosure, a neural network (also called a projector) is employed to acquire samples based on user input and the character (e.g., a desired movement direction deviating from the current movement direction at a future moment). Subsequently, another neural network in the form of an autoencoder is used to reconstruct the original animation pose from the compressed samples. A third neural network is used to predict what the animation will continue to play from the compressed samples, thereby allowing the user to combine new choices with existing ones. The internal architecture of these neural network models has been described in the embodiments above.

[0271] Those skilled in the art will understand that the scope of protection of this invention is not limited to the examples discussed above, and various modifications and variations can be made without departing from the scope of protection defined by the appended claims. Although the invention has been illustrated and described in detail in the accompanying drawings and specification, such illustrations and descriptions are considered illustrative or exemplary only, and not restrictive. The invention is not limited to the disclosed embodiments, but includes any combination of the disclosed embodiments that can achieve beneficial effects.

[0272] Those skilled in the art, when implementing the claimed invention, can understand and implement variations of the disclosed embodiments by studying the drawings, specification, and appended claims. In the specification and claims, the word "comprising" does not exclude other elements, and the indefinite articles "a" or "an" do not exclude a plurality. In fact, they should be understood to mean "at least one." The fact that only certain features are recited in mutually different dependent claims does not mean that a combination of these features cannot achieve beneficial effects. Any reference numerals in the claims should not be construed as limiting the scope of protection of the invention. Features of the above embodiments and aspects can be combined with each other unless the combination would result in an obvious technical conflict.

Claims

1. A method for motion data processing using a neural network data encoding model, the neural network data encoding model comprising an encoder part and a decoder part, the method comprising: The source motion data is input to the encoder input layer of the encoder section; At least one continuous representation generation layer of the encoder portion processes the source motion data by converting the source motion data into compressed motion data based on a preset continuous function. The encoder output layer of the encoder section outputs the compressed motion data; The compressed motion data is input to the decoder input layer of the decoder section, and the decoder input layer directs the compressed motion data to multiple independent component neural networks of the decoder layer; Each of the multiple independent component neural networks included in the decoding layer of the decoder part decodes the compressed motion data to generate multiple independent decoded compressed motion data, wherein each independent component neural network is connected to the input layer of the decoder and merges the multiple independent decoded compressed motion data to generate decoded compressed motion data. The decompression layer of the decoder part decompresses the decoded compressed motion data; as well as The decoded and decompressed motion data is output as the output motion data.

2. The method according to claim 1, wherein, The preset continuous function is the Lipschitz continuous function.

3. The method according to claim 2, wherein, Each of the continuous representation generation layers is configured as a Lipshitz continuous representation layer with an associated learnable Lipshitz constant c, which is associated with the weight normalization method to be adopted by the Lipshitz continuous representation layer.

4. The method according to any one of the preceding claims, wherein, The motion data is transformed by multiple sequentially connected layers of representation generation.

5. The method according to any one of the preceding claims, wherein, The decoding layer is configured as a Sparse Hybrid Expert (SMoE) layer.

6. The method according to claim 5, wherein, The decoder input layer is a gated layer, and the method further includes: inputting the compressed motion data into the gated layer; The gating layer calculates the activation probability of each individual component neural network; Select a subset of the plurality of independent component neural networks, the subset corresponding to a preset number of independent component neural networks with the highest activation probability among the plurality of independent component neural networks; Gating and routing the compressed motion data to independent component neural networks within the subset; and Calculate the weighted sum of the outputs of the independent component neural networks in the subset, and provide the weighted sum as the output of the SMoE layer, wherein the weighted sum represents the sum of the outputs of the independent component neural networks in the subset after being weighted by associated normalized gating values.

7. The method according to any one of the preceding claims further includes storing the compressed motion data output by the encoder output layer as stored compressed motion data, the compressed motion data including a plurality of compressed motion data frames; the method further includes: Receive input control data; Based on the input control data, a compressed motion data frame is selected from the stored compressed motion data; as well as The selected compressed motion data frame is input to the decoder input layer, and then the selected motion data frame is guided, decoded, decompressed and output.

8. The method according to claim 7, wherein, The input control data includes user input control data and information about the virtual object controlled by the user. The user input control data includes information related to at least one of the following: the action to be performed on the virtual object or the action that the virtual object will perform, the movement that the virtual object will perform, and the orientation change that the virtual object will perform.

9. The method according to claim 7 or 8, further comprising: Based on the compressed motion data and the user input control data, predict future motion data frames representing future moments, and The future motion data frame is input to the encoder input layer.

10. The method according to any one of claims 7 to 9, further comprising concatenating the compressed motion data or the future motion data with additional control data, and inputting the concatenated data into the decoder input layer.

11. A motion data processing system, the system comprising a neural network data encoding model for processing motion data, the neural network data encoding model comprising an encoder part and a decoder part, the system further comprising: The processor is configured as follows: Controls the neural network data encoding model to process motion data; The neural network data encoding model includes: The encoder portion includes multiple encoder layers connected in sequence. Each encoder layer includes: an encoder input layer configured to receive source motion data; at least one continuous representation generation layer configured to convert the source motion data into compressed motion data based on a preset continuous function; and an encoder output layer for outputting the compressed motion data. The decoder portion includes multiple decoder layers connected in sequence. Each decoder layer includes: a decoder input layer for receiving the compressed motion data; a decoding layer configured to decode the compressed motion data into decoded compressed motion data, the decoding layer including multiple independent component neural networks, each independent component neural network being connected to the decoder input layer to receive and process the compressed motion data and output the decoded compressed motion data; a decompression layer configured to decompress the decoded compressed motion data; and an output layer configured to output the decompressed decoded motion data as output motion data.

12. The system according to claim 11, wherein, The at least one continuous representation generation layer is configured as a Lipshitz continuous representation layer, configured to transform the source motion data through a Lipshitz continuous transformation to form a Lipshitz continuous representation of the motion data.

13. The system according to claim 12, wherein, Each of the at least one Lipshitz continuous representation layer is generated as a linear layer with an associated learnable Lipshitz constant c, which is used to determine the weight normalization method to be used by the Lipshitz continuous representation layer.

14. The system according to claim 12 or 13, wherein, Multiple Lipschitz continuous characterization layers are provided.

15. The system according to any one of claims 11 to 14, wherein, The decoding layer is configured as a Sparse Hybrid Expert (SMoE) layer, comprising multiple independent component neural networks.

16. The system according to claim 15, wherein, The decoder input layer is a gated layer, and each of the plurality of independent component neural networks is connected to the gated layer to receive the compressed motion data from the gated layer and is connected to the output layer.

17. The system according to claim 16, wherein, The processor is configured to control the decoder section to: The compressed motion data is input into the gating layer; The activation probability of each independent component neural network is calculated through the gated layer; A subset is selected from the plurality of independent component neural networks, the subset corresponding to a preset number of independent component neural networks with the highest activation probability among the plurality of independent component neural networks, and the compressed motion data is processed through each independent component neural network in the subset; Calculate the weighted sum of the outputs of the independent component neural networks in the subset, and provide the weighted sum as the output of the SMoE layer, wherein the output is represented by the weighted sum obtained by weighting the outputs of the independent component neural networks in the subset with associated normalized gating values.

18. The system according to any one of claims 11 to 17, further comprising an interface for receiving input control data, and the processor further configured to: The compressed motion data output by the encoder output layer is stored as stored compressed motion data, which includes multiple compressed motion data frames; Based on the input control data, a compressed motion data frame is selected from the stored compressed motion data; as well as The selected compressed motion data frame is input into the decoder input layer, and then the guiding, decoding, decompression and output steps to be performed based on the selected motion data frame are controlled.

19. The system according to any one of claims 11 to 18, wherein, The input control data includes user input control data and information about the virtual object controlled by the user. The user input control data includes information related to at least one of the following: the action to be performed on the virtual object or the action that the virtual object will perform, the movement that the virtual object will perform, and the orientation change that the virtual object will perform.

20. The system according to claim 18 or 19, wherein, The processor is also configured to receive a predicted future motion data frame representing motion data at a future moment, as well as additional input control data, and to concatenate the compressed motion data or the future motion data with the additional control data, and input the concatenated data to the encoder input layer.

21. The system according to claim 20, wherein, The system further includes a neural network prediction model configured to predict future motion data frames at future times based on the user control input data and coded compressed motion data obtained from the source motion data, wherein the neural network prediction model is preferably configured to generate the additional input control data.

22. A method for generating a neural network data encoding model for motion data processing, the data encoding model comprising an encoder part and a decoder part, the method comprising the following steps: The encoder portion is initialized by sequentially connecting multiple encoder layers, wherein each encoder layer includes: an encoder input layer for receiving source motion data; at least one continuous representation generation layer configured to convert the source motion data into compressed motion data based on a preset continuous function; and an encoder output layer for outputting the compressed motion data. The decoder section is initialized by sequentially connecting multiple decoder layers. Each decoder layer includes: a decoder input layer for receiving the compressed motion data; a decoding layer configured to decode the compressed motion data into decoded compressed motion data, the decoding layer comprising multiple independent component neural networks, each of which is connected to the decoder input layer to receive the compressed motion data and provide the compressed motion data as output decoded compressed motion data; and... A decompression layer is configured to decompress the decoded compressed motion data, and an output layer is configured to output the decompressed decoded motion data as output motion data. The neural network data encoding model is trained until the degree of matching between the source motion data input to the data encoding model and the output motion data representing the source motion data obtained after processing by the data encoding model reaches a preset quality level.

23. The method according to claim 22, wherein, The at least one continuous representation generation layer is configured as a Lipshitz continuous representation layer, configured to transform the source motion data through a Lipshitz continuous transformation to form a Lipshitz continuous representation of the motion data.

24. The method of claim 23, further comprising: Each of the Lipschitz continuous characterization layers is formed in the following manner: Construct a linear layer with an associated learnable Lipshitz constant c, wherein the Lipshitz constant c is used to determine the weight normalization method to be used in the Lipshitz continuous representation layer, and The Lipschitz constant c is learned through model training.

25. The method according to any one of claims 22 to 24, further comprising configuring the decoding layer as a Sparse Hybrid Expert (SMoE) layer; and The decoder input layer is a gated layer, and the method further includes: constructing the SMoE layer by forming multiple independent component neural networks, each of which is connected to the gated layer to receive the compressed motion data from the gated layer; The method further includes: The compressed motion data is input into the gating layer; The gating layer calculates the activation probability of each individual component neural network; Select a subset from the plurality of independent component neural networks, wherein the subset corresponds to a preset number of independent component neural networks with the highest activation probability among the plurality of independent component neural networks; Calculate the weighted sum of the outputs of the independent component neural networks in the subset, and provide the weighted sum as the output of the SMoE layer, wherein the output is represented by the weighted sum obtained by weighting the outputs of the independent component neural networks in the subset with associated normalized gating values.

26. The method according to any one of claims 22 to 25, wherein, The training includes: The source motion data is input to the encoder input layer; The data encoding model processes the motion data; Receive output motion data representing decompressed and decoded motion data from the decoder output layer; The output motion data is compared with the source motion data, and the degree of matching between the output motion data and the source motion data is evaluated by calculating the quality level of the output motion data; and The method includes repeated iterative training until the quality level of the output motion data reaches a preset quality level.

27. The method of claim 26 further includes training the neural network data encoding model using time-continuous pairs of motion data frames.

28. A computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 10 or the method of any one of claims 22 to 27.

Images