Object control method and apparatus, device, and storage medium

Abstract

An object control method, which is executed by a computer device. The method comprises: acquiring a historical motion parameter and a control parameter, the historical motion parameter representing a historical motion executed by a controlled object prior to a current motion prediction round, and the control parameter representing a motion expectation of the current motion prediction round for the controlled object (201); on the basis of the historical motion parameter and the control parameter, by means of a motion diffusion model, denoising a noise motion parameter obtained from Gaussian noise sampling to obtain an expected motion parameter of the current motion prediction round (202); and controlling the controlled object on the basis of the expected motion parameter (203).

Description

Object control method, device, equipment and storage medium

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 2024104371540, filed on April 11, 2024, and entitled "Object control method, device, equipment and storage medium", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] Embodiments of the present application relate to the field of human-computer interaction, and in particular to an object control method, device, equipment and storage medium. BACKGROUND

[0004] Diffusion Model as a generative model, its principle is to learn the noise distribution of adding noise on the original data in the forward noise adding process, and to denoise the noise data in the reverse diffusion process to obtain the denoised data. Motion Diffusion Model (MDM) is a diffusion model that generates motion parameters for controlling the controlled object by denoising noise motion parameters. The motion parameters can be used to control game characters, robots, etc.

[0005] In related technologies, in order to generate motion parameters using the motion diffusion model to control the controlled object, the control target needs to be input into the motion diffusion model as a diffusion condition, so as to output the expected motion parameters through the motion diffusion model, wherein the control target includes motion type, motion direction, etc., and the motion parameters include joint position coordinates and joint rotation amounts of the controlled object. For example, NPCs (Non-Player Characters) in games usually need to perform specific tasks, that is, motion parameters for controlling the NPCs to perform the tasks can be output using the motion diffusion model, and in the case of long task execution time, the motion diffusion model needs to output long-time motion parameters at one time.

[0006] However, outputting long-time motion parameters at one time requires using motion diffusion models with large parameter amounts and high complexity, and the requirements for model deployment conditions are higher. SUMMARY

[0007] Embodiments of the present application provide an object control method, device, equipment and storage medium:

[0008] In one aspect, the present application provides an object control method, the method comprising:

[0009] acquire a history action parameter and a control parameter, the history action parameter representing a history action performed by the controlled object before a current action prediction round, and the control parameter representing an action expectation of the controlled object in the current action prediction round;

[0010] de-noise, by an action diffusion model, a noise action parameter sampled from Gaussian noise based on the history action parameter and the control parameter, to obtain an expected action parameter of the current action prediction round; and

[0011] control the controlled object according to the expected action parameter.

[0012] In another aspect, an embodiment of the present application provides an object control device, which comprises:

[0013] a parameter acquisition module, configured to acquire a history action parameter and a control parameter, the history action parameter representing a history action performed by the controlled object before a current action prediction round, and the control parameter representing an action expectation of the controlled object in the current action prediction round;

[0014] an action prediction module, configured to de-noise, by an action diffusion model, a noise action parameter sampled from Gaussian noise based on the history action parameter and the control parameter, to obtain an expected action parameter of the current action prediction round; and

[0015] a control module, configured to control the controlled object according to the expected action parameter.

[0016] In another aspect, an embodiment of the present application provides a computer device, which comprises a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the object control method in the above aspect.

[0017] In another aspect, an embodiment of the present application provides a computer readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the object control method in the above aspect.

[0018] In another aspect, an embodiment of the present application provides a computer program product, which comprises at least one instruction stored in a computer readable storage medium. A processor of a computer device reads the at least one instruction from the computer readable storage medium, and the processor executes the at least one instruction, so that the computer device executes the object control method in the above aspect.

[0019] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used for the description of the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of the disclosed drawings.

[0021] FIG. 1 shows a structural block diagram of a computer system according to an example embodiment of the present application;

[0022] FIG. 2 shows a flowchart of an object control method according to an example embodiment of the present application;

[0023] FIG. 3 shows a flowchart of an object control method according to another example embodiment of the present application;

[0024] FIG. 4 shows a structural schematic diagram of an action diffusion model according to an example embodiment of the present application;

[0025] FIG. 5 shows a flowchart of a trajectory expansion process according to an example embodiment of the present application;

[0026] FIG. 6 shows a schematic diagram of trajectory expansion according to an example embodiment of the present application;

[0027] FIG. 7 shows a flowchart of controlling a controlled object based on a desired action parameter according to an example embodiment of the present application;

[0028] FIG. 8 shows a flowchart of a training method of an action diffusion model according to an example embodiment of the present application;

[0029] FIG. 9 shows a structural schematic diagram of an action diffusion model according to another example embodiment of the present application;

[0030] FIG. 10 shows a schematic diagram of applying an object control method to a game interaction scenario according to an example embodiment of the present application;

[0031] FIG. 11 shows a schematic diagram of applying an object control method to a robot control scenario according to an example embodiment of the present application;

[0032] FIG. 12 shows a comparison diagram of effects of applying an object control method to control an object according to an example embodiment of the present application;

[0033] FIG. 13 shows a comparison diagram of effects of applying the object control method to control an object according to another example embodiment of the present application;

[0034] FIG. 14 shows a structural block diagram of an object control apparatus according to an example embodiment of the present application;

[0035] FIG. 15 shows a structural schematic diagram of a computer device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0037] The example embodiments will be described in detail below with reference to the drawings. Unless otherwise indicated, the same numbers on different drawings represent the same or similar elements. The following example embodiments described in the example embodiments are not representative of all the embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0038] The terms used in the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc. can be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one piece of information from another piece of information. For example, without departing from the scope of the present application, the first parameter can also be referred to as the second parameter, and similarly, the second parameter can also be referred to as the first parameter. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0040] First, the terms involved in the embodiments of the present application are briefly introduced:

[0041] Past Motion: represents the historical motion of the controlled object before the current motion prediction round. Optionally, the Past Motion can be the motion parameter of the controlled object based on the last round of motion prediction results, or the motion parameter of the controlled object based on all historical motion prediction results. In the first round of motion prediction, the Past Motion is zero.

[0042] Control Parameter: represents the motion expectation of the controlled object in the current motion prediction round. Optionally, the control parameter can be determined according to the control instruction set by the user, and the control parameter can include displacement trajectory, motion style, motion direction, etc.

[0043] Style Label: used to control the motion style of the controlled object, such as running, left foot jumping, right foot jumping, etc.

[0044] Future Root Position: used to control the movement trajectory of the controlled object, the Future Root Position is a trajectory composed of coordinate points on the horizontal coordinate plane, used to indicate the projection displacement expectation of the root joint coordinate of the controlled object on the horizontal coordinate plane.

[0045] Future Root Orient: used to control the movement direction of the controlled object, such as forward, right, southeast, etc.

[0046] Noisy Motion: when applying the motion diffusion model to implement motion prediction, the Noisy Motion is obtained by sampling Gaussian noise; when training the motion diffusion model, the Noisy Motion is obtained by performing noise addition processing on the motion parameter ground truth.

[0047] Motion Parameter Ground Truth: refers to the true motion parameter data used to train the motion diffusion model, a single Motion Parameter Ground Truth can include S-frame sample root joint coordinates and sample joint rotation parameters, and part of the frame motion parameter ground truth can be used to generate sample noisy motion parameters, and part of the frame can be used for model training loss calculation.

[0048] Sample Noisy Motion Parameter: based on the Motion Parameter Ground Truth, generated by performing noise addition processing on part of the frame motion parameter ground truth, used in the motion diffusion model training process as model input for denoising processing to obtain sample motion parameters.

[0049] Action diffusion model: a deep learning framework that combines the principles of diffusion model and action generation task. Diffusion model is a generative model whose core idea is to learn the noise distribution of adding noise to the original data in the forward noise-adding process, and to denoise the noisy data in the reverse diffusion process to obtain the denoised data. In the context of action generation, action diffusion model uses this principle to generate human action sequences that match given text descriptions.

[0050] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions.

[0052] Machine learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a special study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning technologies.

[0053] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned driving, autonomous driving, unmanned aerial vehicle, digital twin, virtual human, robot, artificial intelligence generative content (AIGC), conversational interaction, smart medical treatment, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0054] The scheme provided by the embodiments of the present application relates to machine learning and other technologies of artificial intelligence, which is specifically explained through the following embodiments.

[0055] Please refer to FIG. 1, which shows a structural block diagram of a computer system provided by an example embodiment of the present application, which can include a terminal 110 and a server 120. The terminal 110 and the server 120 communicate data through a communication network, which can be a wired network or a wireless network, and can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0056] The terminal 110 is an electronic device installed with an application program having an object control function. The object control function can be a function of a native application in the terminal, or a function of a third-party application. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart television, a wearable device, or a vehicle-mounted terminal, etc. FIG. 1 only takes the terminal 110 as an example of a desktop computer, but this is not limited.

[0057] The server 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. In the embodiments of the present application, the server 120 can be a background server of an application program having an object control function.

[0058] In some embodiments, there is data interaction between the server and the terminal. Illustratively, as shown in FIG. 1, the terminal 110, in the case of receiving a control instruction input by a user, acquires a control parameter and a historical action parameter of the controlled object, and sends the historical action parameter and the control parameter to the server 120. The server 120, according to the historical action parameter and the control parameter, denoises a noise action parameter sampled from Gaussian noise through an action diffusion model, thereby obtaining an expected action parameter of a current action prediction round, and then the server 120 returns the expected action parameter to the terminal 110, which controls the controlled object according to the expected action parameter.

[0059] In combination with the foregoing, the object control method provided by the present application is described. The method can be executed by a server or a terminal, or by both the server and the terminal.

[0060] Optionally, in the case where the method is executed by the terminal, the terminal, in the case of receiving a control instruction, acquires a control parameter and a historical action parameter of the controlled object, and denoises a noise action parameter sampled from Gaussian noise through an action diffusion model based on the control parameter and the historical action parameter, thereby obtaining an expected action parameter of a current action prediction round, and controls the controlled object according to the expected action parameter.

[0061] Optionally, in the case where the method is executed by both the server and the terminal, the server, in the case of receiving a control instruction sent by the terminal, acquires a control parameter and a historical action parameter of the controlled object, and denoises a noise action parameter sampled from Gaussian noise through an action diffusion model based on the control parameter and the historical action parameter, thereby obtaining an expected action parameter of a current action prediction round, and returns the expected action parameter to the terminal, which controls the controlled object according to the expected action parameter.

[0062] Referring to FIG. 2, a flowchart of an object control method provided by an exemplary embodiment of the present application is shown. The embodiment takes the method used in a computer device (including a terminal and / or a server) as an example for description, and the method includes the following steps:

[0063] Step 201: Acquire a historical action parameter and a control parameter. The historical action parameter represents a historical action performed by a controlled object before a current action prediction round, and the control parameter represents an action expectation of the controlled object in the current action prediction round.

[0064] Optionally, the controlled object can be a virtual character in a game, or a robot, etc., and the embodiment of the present application does not limit this.

[0065] The control parameter represents an action expectation of the controlled object in the current action prediction round. Optionally, the control parameter can include a displacement trajectory, an action style, a motion direction, etc.

[0066] In a possible implementation, in a case where a control instruction input by a user is received, the computer device determines a control parameter based on the control instruction. For example, in a game scenario, in a case where a control instruction "run forward" input by a user is received, the computer device can determine that the action type is running and the movement direction is forward.

[0067] In some embodiments, in the process of implementing segmented control of the controlled object, in addition to the need to perform action prediction according to the control parameter, in order to improve the continuity of the action of the controlled object before and after each round of action prediction, the computer device also needs to obtain a historical action parameter of the controlled object.

[0068] The historical action parameter represents a historical action performed by the controlled object before the current round of action prediction. Optionally, the historical action parameter can be an action parameter performed by the controlled object based on a previous round of action prediction result, or an action parameter performed by the controlled object based on all historical action prediction results, and the historical action parameter is zero at the first action prediction.

[0069] Optionally, according to the properties of each joint of the controlled object and the connection between the joints, the skeleton joints of the controlled object can be divided into a root joint and other joints, the other joints including arm joints, leg joints, and the like. The action parameter includes the root joint coordinates and the joint rotation parameters, and the historical action parameter is the root joint coordinates and the joint rotation parameters of the controlled object when performing an action before the current round of action prediction.

[0070] In a possible implementation, before controlling the controlled object, the computer device first constructs a joint coordinate system corresponding to the controlled object according to the initial root joint coordinates of the controlled object, with the initial root joint coordinates as the coordinate origin and the front direction of the controlled object as the longitudinal axis direction, so as to determine the joint coordinates and the joint rotation of each joint of the controlled object based on the joint coordinate system.

[0071] Joint Coordinate System: refers to a coordinate system constructed by the computer device according to the initial root joint coordinates of the controlled object, with the initial root joint coordinates as the coordinate origin and the front direction of the controlled object as the longitudinal axis direction, and used to determine the joint coordinates and the joint rotation of each joint of the controlled object.

[0072] In step 202, based on the historical action parameter and the control parameter, the noise action parameter sampled from the Gaussian noise is denoised by the action diffusion model to obtain the expected action parameter of the current round of action prediction.

[0073] In some embodiments, after obtaining the historical action parameters and the control parameters, the computer device can denoise the noise action parameters obtained by sampling Gaussian noise according to the historical action parameters and the control parameters through the action diffusion model, so as to obtain the expected action parameters of the current action prediction round.

[0074] The noise action parameters can be obtained by sampling Gaussian noise. In a possible implementation, the computer device determines the feature dimension and the frame number of the noise sampling based on the parameter features of the historical action parameters and the action prediction frame number of the current action prediction round, so as to sample the noise action parameters with the same feature dimension and corresponding frame number from the Gaussian noise according to the feature dimension and the frame number.

[0075] In some embodiments, in order to optimize the denoising effect, the computer device also needs to determine the diffusion step length and perform denoising processing on the noise action parameters through the action diffusion model according to the diffusion step length, so as to obtain the expected action parameters of the current action prediction round. For example, if the diffusion step length is 8, the computer device needs to perform denoising processing on the noise action parameters through the action diffusion model for 8 diffusion step lengths based on the historical action parameters and the control parameters, so as to obtain the expected action parameters. Diffusion step length (Diffusion Timestep): refers to the number of steps of the action diffusion model when performing denoising processing on the noise action parameters.

[0076] Optionally, each action prediction round can predict multiple frames of expected action parameters, and the expected action parameters include joint displacement parameters of each frame and joint rotation parameters of each frame, wherein the joint displacement parameters represent the joint position coordinates of the current frame, and the joint rotation parameters represent the joint rotation angles of the current frame.

[0077] Step 203: controlling the controlled object according to the expected action parameters.

[0078] In some embodiments, after obtaining the expected action parameters of the current action prediction round, the computer device can control the controlled object according to the expected action parameters.

[0079] Optionally, in the case where the expected action parameters include multiple frames of joint displacement parameters and joint rotation parameters, the computer device can control the controlled object to perform actions of the corresponding frames according to the joint displacement parameters and the joint rotation parameters of each frame.

[0080] To sum up, in the process of predicting the action parameter by using the action diffusion model in the embodiment of the application, the single output long-time action parameter is converted into the multi-round output short-time action parameter, the required generation time of the action parameter is shortened each time, the requirement for the model parameter quantity of the action diffusion model is reduced, and thus the requirement for the model deployment condition is reduced. In addition to the control parameter, the historical action parameter is added as the diffusion condition, the historical action performed by the controlled object before each round of action prediction is referred to, the continuity between the actions in the phased output is improved, and the output quality of the long-time action parameter is ensured.

[0081] In some embodiments, the control parameter can include an action style label, a root joint displacement trajectory, and a movement direction. For the parameter characteristics of different control parameters, the computer device can perform corresponding processing on the model parameters in the action prediction process to improve the accuracy of the action diffusion model in realizing action prediction based on the control parameter.

[0082] Please refer to FIG. 3, which shows a flowchart of an object control method provided by another exemplary embodiment of the application. The embodiment takes the method used in a computer device (including a terminal and / or a server) as an example for illustration, and the method includes the following steps:

[0083] In step 301, historical action parameters and control parameters are obtained. The historical action parameters represent the historical action performed by the controlled object before the current round of action prediction, and the control parameters represent the action expectation of the controlled object in the current round of action prediction.

[0084] The specific implementation of this step can refer to step 201, and the embodiment will not be described here.

[0085] Optionally, the control parameter can include an action style label, a root joint displacement trajectory, and a movement direction. The action style label is used to control the action style of the controlled object, such as running, left-foot jumping, right-foot jumping, etc. The root joint displacement trajectory is used to control the movement trajectory of the controlled object. The root joint displacement trajectory is a trajectory composed of coordinate points on a horizontal coordinate plane, which is used to indicate the projection displacement expectation of the root joint coordinate of the controlled object on the horizontal coordinate plane. The movement direction is used to control the movement direction of the controlled object, such as moving forward, moving right, etc.

[0086] In a possible implementation, in the first round of action prediction, the movement direction can be determined according to the control instruction input by the user, and the root joint displacement trajectory can be generated according to the movement direction. For example, if the movement direction is to move right, the root joint displacement trajectory can be a straight line to the right.

[0087] At step 302, the historical motion parameters are respectively encoded by a feature mapping layer of the action diffusion model to obtain historical action features, the action style label is encoded to obtain action style features, the root joint displacement trajectory is encoded to obtain root joint displacement features, and the motion direction is encoded to obtain motion direction features.

[0088] Optionally, the action diffusion model comprises a feature mapping layer and an action prediction network. In some embodiments, after the historical motion parameters, the action style label, the root joint displacement trajectory, and the motion direction are input into the action diffusion model, the computer device can encode the historical motion parameters, the action style label, the root joint displacement trajectory, and the motion direction by the feature mapping layer of the action diffusion model, so as to obtain the historical action features, the action style features, the root joint displacement features, and the motion direction features.

[0089] The feature mapping layer (Feature Mapping Layer) is a component in the action diffusion model, which can encode the input historical motion parameters, action style label, root joint displacement trajectory, and motion direction, respectively, to obtain corresponding historical action features, action style features, root joint displacement features, and motion direction features. Optionally, it is a linear layer. In order to avoid mutual coupling between different features or some feature components occupying a dominant position, multiple feature mapping layers can be set in the action diffusion model, and different inputs correspond to different feature mapping layers.

[0090] The action prediction network (Motion Prediction Network) is a component of the action diffusion model, which receives the historical action features, action style features, root joint displacement features, and motion direction features obtained by encoding the historical motion parameters, action style label, root joint displacement trajectory, and motion direction by the feature mapping layer, and uses these features as diffusion conditions to denoise the noise motion parameters, thereby obtaining the expected motion parameters.

[0091] Optionally, the feature mapping layer can be a linear layer (Linear). In order to avoid mutual coupling between different features or some feature components occupying a dominant position during feature mapping, multiple feature mapping layers can be included in the action diffusion model, and the historical motion parameters, action style label, root joint displacement trajectory, and motion direction correspond to different feature mapping layers, respectively.

[0092] In one possible implementation, the computer device inputs the historical motion parameters, the action style label, the root joint displacement trajectory, and the motion direction into the corresponding feature mapping layers, respectively, so as to perform feature encoding by the respective corresponding feature mapping layers to obtain the historical action features, the action style features, the root joint displacement features, and the motion direction features, respectively.

[0093] As shown in FIG. 4, the action diffusion model includes a plurality of linear layers and an action prediction network. The computer device inputs the historical action parameter p, the action style label c l , the root joint displacement trajectory c rv , and the movement direction c ro into corresponding feature mapping layers respectively, so as to perform feature encoding through the respective corresponding feature mapping layers, and obtain the historical action feature, the action style feature, the root joint displacement feature, and the movement direction feature respectively.

[0094] In step 303, based on the historical action feature, the action style feature, the root joint displacement feature, and the movement direction feature, the computer device denoises the noise action parameter sampled from the Gaussian noise through the action diffusion model to obtain the expected action parameter of the current action prediction round.

[0095] In some embodiments, after completing the feature encoding, the computer device can perform denoising processing on the noise action parameter through the action diffusion model according to the historical action feature, the action style feature, the root joint displacement feature, and the movement direction feature, so as to obtain the expected action parameter of the current action prediction round.

[0096] In one possible implementation, the computer device inputs the historical action feature, the action style feature, the root joint displacement feature, and the movement direction feature into the action prediction network in the action diffusion model as diffusion conditions, and denoises the noise action parameter through the action prediction network to obtain the expected action parameter.

[0097] In one possible implementation, the action diffusion model further includes an MLP (Multi-Layer Perceptron). In the case of setting a diffusion step length (Diffusion Timestep), the computer device can perform feature encoding on the diffusion step length through the MLP to obtain a step length feature, so as to denoise the noise action parameter based on the step length feature. The MLP is part of the action diffusion model. In the case of setting a diffusion step length, the MLP can perform feature encoding on the diffusion step length to obtain a step length feature, and then denoise the noise action parameter based on the step length feature. Step length feature (Timestep Feature): refers to the feature obtained by performing feature encoding on the diffusion step length through the MLP. This feature can be used by the action diffusion model to denoise the noise action parameter based on the historical action feature, the action style feature, the root joint displacement feature, and the movement direction feature, so as to obtain the expected action parameter.

[0098] As shown in FIG. 4, the computer device performs feature encoding on the diffusion step length t through the MLP network to obtain a step length feature, and denoises the F-frame noise action parameter de-noise, to obtain the expected action parameter of the current action prediction round

[0099] In some embodiments, in the process of action prediction and control, both continuity between each round of action prediction is ensured by historical action parameters and action style or action direction is adjusted by control parameters, that is, control parameters between adjacent action prediction rounds can change greatly, such as action style switching, switching from left foot jump to right foot jump, etc. Therefore, in order to realize smooth transition of action style, in the case of action style switching, the computer device can reduce the influence of historical action parameters in the action prediction process according to the action sampling principle of classifier-free guidance (CFG). Classifier-free guidance is a principle for action sampling. In this application, when the action style label changes between adjacent action prediction rounds, in order to realize smooth transition of action style, the computer device can reduce the influence of historical action parameters in the action prediction process according to this principle.

[0100] In a possible implementation, before performing the action prediction of the current round, the computer device can first compare the action style label of the current action prediction round and the action style label of the last action prediction round. Further, in the case that the action style label of the current action prediction round is consistent with the action style label of the last action prediction round, the computer device can directly de-noise the noise action parameter sampled from Gaussian noise based on the historical action feature, the action style feature, the root joint displacement feature and the motion direction feature through the action diffusion model, to obtain the expected action parameter of the current action prediction round.

[0101] Optionally, the historical action parameter can be represented as p, the action style label can be represented as c l , the root joint displacement trajectory can be represented as c rv , the motion direction can be represented as c ro , the diffusion step can be represented as t, and the noise action parameter can be represented as x t , the process of de-noising the noise action parameter sampled from Gaussian noise through the action diffusion model to obtain the expected action parameter can be represented as

[0102] In a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, in order to reduce the influence of the historical action parameter in the action prediction process, the computer device can perform twice denoising on the noise action parameter through the action diffusion model according to the historical action feature, the action style feature, the root joint displacement feature and the motion direction feature, and obtain twice denoising results, and the historical action feature is set to zero in one of the twice denoising processes, and then the expected action parameter of the current action prediction round is determined according to the twice denoising results.

[0103] In a possible implementation, in a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, the computer device first performs denoising on the noise action parameter sampled from the Gaussian noise through the action diffusion model according to the historical action feature, the action style feature, the root joint displacement feature and the motion direction feature to obtain a first candidate action parameter, and then sets the historical action feature to zero, and performs denoising on the noise action parameter sampled from the Gaussian noise through the action diffusion model only according to the action style feature, the root joint displacement feature and the motion direction feature to obtain a second candidate action parameter, and finally the expected action parameter of the current action prediction round can be obtained by fusing the first candidate action parameter and the second candidate action parameter.

[0104] Optionally, after obtaining the first candidate action parameter and the second candidate action parameter, the computer device can determine an action parameter difference value according to the first candidate action parameter and the second candidate action parameter, and then determine an action style guidance amount of the action style label of the current action prediction round according to the action parameter difference value and a guidance proportion, and determine the expected action parameter of the current action prediction round according to the second candidate action parameter and the action style guidance amount.

[0105] The action parameter difference value is the difference between the first candidate action parameter and the second candidate action parameter obtained by performing twice denoising on the noise action parameter through the action diffusion model in a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round. The difference value can be used to determine the action style guidance amount of the action style label of the current action prediction round.

[0106] The guidance proportion is used to adjust the influence degree of the first candidate action parameter on the expected action parameter; and the action style guidance amount is the influence amount of the first candidate action parameter on the expected action parameter.

[0107] Motion Style Guidance: an influence amount of the first candidate motion parameter on the expected motion parameter, determined according to the motion parameter difference and a guidance ratio when the motion style label of the current motion prediction round is inconsistent with the motion style label of the last motion prediction round, used to determine the expected motion parameter of the current motion prediction round together with the second candidate motion parameter.

[0108] In a possible implementation, after obtaining the first candidate motion parameter and the second candidate motion parameter, the computer device subtracts the second candidate motion parameter from the first candidate motion parameter to obtain a motion parameter difference, and then multiplies the motion parameter difference by a guidance ratio to adjust the influence degree of the first candidate motion parameter on the expected motion parameter, that is, the motion style guidance can be obtained, and the expected motion parameter can be obtained by adding the second candidate motion parameter to the motion style guidance.

[0109] The lower the guidance ratio is, the smaller the influence degree of the first candidate motion parameter on the expected motion parameter is, the smaller the motion style guidance is, and the closer the expected motion parameter is to the second candidate motion parameter, and the smaller the influence of the historical motion parameter on the expected motion parameter is.

[0110] Optionally, in the case that the motion style label of the current motion prediction round is inconsistent with the motion style label of the last motion prediction round, the process of determining the expected motion parameter can be represented as wherein γ is the guidance ratio, which can be obtained by pre-setting; is an empty set, represents setting the historical motion parameter p to zero.

[0111] The guidance ratio γ can be pre-set according to actual application scenarios and experimental results, and is usually in the range of 0-1. The lower the γ value is, the smaller the influence degree of the first candidate motion parameter on the expected motion parameter is, the smaller the motion style guidance is, and the closer the expected motion parameter is to the second candidate motion parameter, and the smaller the influence of the historical motion parameter on the expected motion parameter is. The guidance ratio can also be dynamically determined based on model training, for example, a larger guidance ratio is set in the early stage of model training, and the guidance ratio is gradually reduced as the training proceeds.

[0112] Step 304: controlling the controlled object according to the expected motion parameter.

[0113] In a possible implementation, the computer device predicts N frames of expected action parameters in each round of action prediction, i.e., 1 to N frames of expected action parameters in the first round of action prediction, N+1 to 2N frames of expected action parameters in the second round of action prediction, and so on, to obtain the expected action parameters corresponding to each round of action prediction. Moreover, after each round of action prediction, the computer device controls the controlled object to perform N frames of actions based on the N frames of expected action parameters.

[0114] Optionally, the N frames of expected action parameters are predicted in the current round of action prediction, and the expected action parameters include N frames of target root joint coordinates and N frames of joint rotation parameters, so that the computer device controls the controlled object to perform N frames of actions according to the N frames of target root joint coordinates and the N frames of joint rotation parameters.

[0115] Optionally, the target root joint coordinates (Target Root Joint Coordinates) are a component of the expected action parameters, and can be represented as O∈R F×3 , where F represents the number of action prediction frames, O represents the target root joint coordinates, and R represents a real number set. The target root joint coordinates represent target coordinate positions of a root joint of the controlled object, and can be used together with the joint rotation parameters to control the controlled object to perform corresponding actions. F×JQ , where F represents the number of action prediction frames, J represents the number of joints, Q represents the number of rotation features, P represents the joint rotation parameters, and R represents a real number set. The joint rotation parameters can be used together with the target root joint coordinates to control the controlled object to perform corresponding actions.

[0116] In the above embodiments, by setting multiple feature mapping layers in the action diffusion model, independent feature coding is performed on the historical action parameters, the action style label, the root joint displacement trajectory, and the motion direction in the action prediction process, respectively, to avoid coupling between features or cause feature imbalance problems, improve the effectiveness of each feature, and thus improve the stability and accuracy of action prediction.

[0117] Moreover, by using segmented action prediction, the control parameters can be adjusted in a timely manner according to control requirements in each round of action prediction, the action style label can be changed or the motion direction can be adjusted, the flexibility of object control is improved, and the style diversity of action prediction is increased.

[0118] In addition, in the case that the action style label changes between two adjacent action predictions, the noise action parameters are subjected to two times of denoising processing by the action diffusion model, and the historical action parameters are set to zero in one of the two times of denoising processes, so that the expected action parameters of the current action prediction round are determined according to the two times of denoising results, which can reduce the influence of the historical action parameters on the current action prediction result, make the action prediction closer to the action style indicated by the current action style label, and further improve the accuracy of the action prediction.

[0119] In some embodiments, considering that the control parameters may differ greatly between adjacent action prediction rounds in the process of controlling the object through segmented action prediction, including action style switching, motion direction mutation, etc., and in the case that N frames of expected action parameters are obtained in each prediction round, if all the expected action parameters of the frames are used to control the controlled object, the action of the controlled object between adjacent action prediction rounds may be jittered, resulting in poor control effect. Therefore, in order to improve the prediction continuity of the action parameters between adjacent action prediction rounds, the computer device can control the controlled object only according to the first M (M is less than N) frames of target root joint coordinates and the first M frames of joint rotation parameters in the N frames of expected action parameters, and determine the root joint displacement trajectory of the next action prediction round according to the last N-M frames of target root joint coordinates.

[0120] Optionally, as shown in FIG. 5, the process of determining the root joint displacement trajectory of the next action prediction round according to the last N-M frames of target root joint coordinates can include the following steps:

[0121] Step 501, project the N frames of target root joint coordinates to a coordinate plane to obtain a target root joint displacement trajectory.

[0122] In a possible implementation, after obtaining the expected action parameters of the current action prediction round, the computer device can project the N frames of target root joint coordinates to a horizontal coordinate plane to obtain the target root joint displacement trajectory of the current action prediction round.

[0123] Coordinate projection: refers to the operation of projecting the N frames of target root joint coordinates to a coordinate plane (such as a horizontal coordinate plane), by which the target root joint displacement trajectory can be obtained, for example, setting the coordinate value in the Z-axis direction of the target root joint coordinates to zero, and obtaining the target root joint displacement trajectory according to the projection displacement points of each frame of target root joint coordinates in the XY plane.

[0124] Optionally, the computer device can set the coordinate value in the Z-axis direction of the target root joint coordinates to zero, so as to obtain the target root joint displacement trajectory according to the projection displacement points of each frame of target root joint coordinates in the XY plane.

[0125] As shown in FIG. 6, the current action prediction round obtains 9 frames of target root joint coordinates. By projecting the 9 frames of target root joint coordinates to a horizontal coordinate plane, a target root joint displacement trajectory can be obtained.

[0126] At step 502, based on the last N-M frames of target root joint displacement points in the target root joint displacement trajectory, a root joint expansion trajectory is obtained through trajectory expansion. The root joint expansion trajectory includes M frames of root joint expansion displacement points.

[0127] In a possible implementation, since the first M frames of target root joint coordinates have been used to control the controlled object, in order to improve the diversity of action prediction, the computer device can determine the root joint expansion trajectory through trajectory expansion only according to the last N-M frames of target root joint displacement points in the target root joint displacement trajectory, and the root joint expansion trajectory includes M frames of root joint expansion displacement points.

[0128] Trajectory expansion: refers to a process of generating a root joint expansion trajectory according to the last N-M frames of target root joint displacement points in the target root joint displacement trajectory. For example, taking the N-th target root joint displacement point in the last N-M frames of target root joint displacement points as a center symmetry point, symmetry points of the N-1-th target root joint displacement point to the N-L-th target root joint displacement point are determined, N+1-th to N+L-th root joint expansion displacement points (L is less than N-M) are obtained, and then a root joint expansion trajectory is generated. If necessary, the expansion can be performed multiple times to obtain a sufficient number of root joint expansion displacement points.

[0129] Root joint expansion trajectory: a trajectory obtained through trajectory expansion, including M frames of root joint expansion displacement points, which can be used together with the last N-M frames of target root joint displacement points to determine a root joint displacement trajectory of a next action prediction round.

[0130] Optionally, in the last N-M frames of target root joint displacement points, the computer device takes the N-th target root joint displacement point as a center symmetry point, and determines symmetry points of the N-1-th target root joint displacement point to the N-L-th target root joint displacement point, so as to obtain N+1-th to N+L-th root joint expansion displacement points, where L is less than N-M, and then a root joint expansion trajectory can be generated according to the N+1-th to N+L-th root joint expansion displacement points.

[0131] Regarding the process of determining the extended displacement points, in one possible implementation, the computer device can construct a local coordinate system with the Nth target root joint displacement point as the coordinate origin, and the tangent direction and the tangent vertical direction of the Nth to (N-1)th target root joint displacement trajectory as the coordinate axes, so as to determine, in the local coordinate system, the symmetric points of the (N-1)th target root joint displacement point to the (N-L)th target root joint displacement point with the coordinate origin as the center of symmetry, thereby obtaining the (N+1)th to (N+L)th root joint extended displacement points.

[0132] In one possible implementation, considering that in the case where L is less than M, it is insufficient to obtain the root joint extended trajectory containing M root joint extended displacement points only according to the (N+1)th to (N+L)th root joint extended displacement points, i.e., single trajectory expansion is insufficient, therefore, in order to obtain as many root joint extended displacement points as possible, after completing the trajectory expansion once, the computer device can further determine, with the (N+L)th root joint extended displacement point as the center of symmetry, the symmetric points of the (N+L-1)th root joint extended displacement point to the Nth target root joint displacement point, thereby obtaining the (N+L+1)th to (N+2L)th root joint extended displacement points.

[0133] Further, in the case where 2L is greater than M, the computer device can generate the root joint extended trajectory containing M root joint extended displacement points according to the (N+1)th to (N+2L)th root joint extended displacement points. In the case where 2L is less than M, the computer device needs to perform trajectory expansion again according to the above-mentioned extended displacement points until the number of root joint extended displacement points is greater than M.

[0134] Illustratively, as shown in FIG. 6, the target root joint displacement trajectory generated based on the expected action parameters of the current action prediction round includes 1-9 target root joint displacement points, of which the last 5 target root joint displacement points can be used for trajectory expansion, therefore, the computer device determines the root joint extended displacement points corresponding to the 5th to 8th target root joint displacement points with the 9th target root joint displacement point as the center of symmetry, thereby completing the first trajectory expansion, further, the computer device determines the secondary root joint extended displacement points corresponding to the root joint extended displacement points corresponding to the 9th target root joint displacement point, the 6th to 8th target root joint displacement points, with the root joint extended displacement point corresponding to the 5th target root joint displacement point as the center of symmetry, thereby completing the second trajectory expansion.

[0135] In step 503, the root joint displacement trajectory of the next action prediction round is determined based on the last N-M target root joint displacement points and the M root joint extended displacement points.

[0136] In some embodiments, after obtaining the root joint extension trajectory, the computer device can determine the root joint displacement trajectory of the next action prediction round according to the target root joint displacement points of the last N-M frames and the M frame root joint extension displacement points according to the frame number corresponding to each displacement point.

[0137] Illustratively, as shown in FIG. 6, after the computer device controls the controlled object based on the target root joint coordinates of the 1st-4th frames, the computer device can perform trajectory extension using the target root joint displacement points of the 5th-9th frames, determine the root joint extension displacement points corresponding to the target root joint displacement points of the 5th-8th frames with the target root joint displacement point of the 9th frame as the center symmetry point, and then obtain 4 root joint extension displacement points (10th-13th frames). Thus, the computer device can generate the root joint displacement trajectory of the next action prediction round according to the target root joint displacement points of the 5th-9th frames and the root joint extension displacement points of the 10th-13th frames. Optionally, considering that the motion direction in the next action prediction round can change and the motion direction will affect the root joint displacement trajectory, the computer device can further adjust the root joint displacement trajectory in combination with the motion direction of the next action prediction round to optimize the root joint displacement trajectory of the next action prediction round.

[0138] In one possible implementation, after obtaining the root joint extension trajectory containing M frame root joint extension displacement points, the computer device first generates a first candidate root joint displacement trajectory of the next action prediction round according to the target root joint displacement points of the last N-M frames and the M frame root joint extension displacement points, generates a second candidate root joint displacement trajectory of the next action prediction round according to the motion direction of the next action prediction round, and finally obtains the root joint displacement trajectory of the next action prediction round by trajectory fusion in combination with the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory.

[0139] Trajectory Fusion: refers to an operation of generating the root joint displacement trajectory of the next action prediction round in combination with the first candidate root joint displacement trajectory generated according to the target root joint displacement points of the last N-M frames and the M frame root joint extension displacement points, and the second candidate root joint displacement trajectory generated according to the motion direction of the next action prediction round. For example, the root joint displacement trajectory of the next action prediction round is generated by calculating the average of the displacement point coordinates of each frame.

[0140] Regarding the process of trajectory fusion, in one possible implementation, the computer device can calculate the average of the displacement point coordinates of each frame according to the first displacement point coordinates in the first candidate root joint displacement trajectory and the second displacement point coordinates in the second candidate root joint displacement trajectory, and then generate the root joint displacement trajectory of the next action prediction round according to the average of the displacement point coordinates.

[0141] In the above embodiments, after obtaining the expected motion parameters of N frames through one motion prediction, the expected motion parameters of all frames are not directly applied to the object control, but the controlled object is controlled based on the expected motion parameters of the first M frames, so that the root joint coordinates in the target root joint coordinates in the expected motion parameters of the last N-M frames are used to generate the root joint displacement trajectory of the next motion prediction round through coordinate projection and trajectory expansion, and the root joint displacement trajectory is used as the input of the motion diffusion model in the next round of motion prediction, which can reduce the motion jitter problem existing in the motion execution process and improve the motion continuity of the controlled object between two motion predictions.

[0142] In combination with the above embodiments, refer to FIG. 7, which shows a flowchart of controlling the controlled object based on the expected motion parameters of multiple rounds of motion prediction according to an example embodiment of the present application.

[0143] In the first round of motion prediction, the computer device determines the motion style label, the motion direction and the root joint displacement trajectory according to the first round of control instruction, and obtains N frames of noise motion parameters through sampling Gaussian noise, so that the N frames of expected motion parameters output by the motion diffusion model 701 can be obtained by inputting the motion style label, the motion direction, the root joint displacement trajectory, the diffusion step and the N frames of noise motion parameters into the motion diffusion model 701. Further, the computer device controls the controlled object 702 by using the first M frames of expected motion parameters, and uses the last N-M frames of expected motion parameters for trajectory expansion.

[0144] For the trajectory expansion process, the computer device projects the target root joint coordinates in the last N-M frames of expected motion parameters to the horizontal coordinate plane to obtain the last N-M frames of target root joint displacement points, so that the first N+1 to N+M frames of root joint expansion displacement points are obtained through trajectory expansion according to the last N-M frames of target root joint displacement points, and the first candidate root joint displacement trajectory corresponding to the second round of motion prediction is generated according to the last N-M frames of target root joint displacement points and the first N+1 to N+M frames of root joint expansion displacement points.

[0145] In the case of receiving the second round of control instruction, the computer device first determines the motion direction and the motion style label according to the second round of control instruction, and generates the second candidate root joint displacement trajectory according to the motion direction, so that the root joint displacement trajectory corresponding to the second round of motion prediction is obtained through trajectory fusion according to the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory. At the same time, the computer device determines the historical motion parameters according to the executed motion of the controlled object 702 in the first round, so that the N frames of expected motion parameters output by the motion diffusion model 701 can be obtained by inputting the historical motion parameters, the root joint displacement trajectory, the motion direction, the motion style label, the diffusion step and the N frames of noise motion parameters into the motion diffusion model 701.

[0146] In some embodiments, in order to improve the action prediction quality of the action diffusion model, the computer device also needs to train the action diffusion model using a large number of action parameter true values before performing action prediction through the action diffusion model.

[0147] Please refer to FIG. 8, which shows a flowchart of a method for training an action diffusion model according to an example embodiment of the present application. The present embodiment takes the method for a computer device (including a terminal and / or a server) as an example for illustration, and the method includes the following steps:

[0148] Step 801: generating sample noise action parameters and sample control parameters based on action parameter true values.

[0149] Sample control parameters: generated based on action parameter true values, used for training the action diffusion model, including sample action style labels, sample root joint displacement trajectories, and sample motion directions, which can be determined according to different frames of action parameter true values.

[0150] Before model training, the computer device first needs to generate sample noise action parameters and sample control parameters through data preprocessing according to action parameter true values.

[0151] Optionally, a single action parameter true value includes S frames of sample root joint coordinates and sample joint rotation parameters. In a possible implementation, the computer device can take the first T frames of action parameter true values as model input, i.e., perform noise addition processing on the first T frames of action parameter true values to obtain sample noise action parameters; and use the last S-T frames of action parameter true values for model training loss calculation, i.e., determine sample control parameters according to the last S-T frames of sample root joint coordinates and sample joint rotation parameters.

[0152] For example, a single action parameter true value includes 45 frames of sample root joint coordinates and sample joint rotation parameters. The computer device can perform noise addition processing on the first 1-10 frames of action parameter true values to obtain sample noise action parameters, and use the 11-45 frames of action parameter true values for model training loss calculation.

[0153] Optionally, the sample control parameters include sample action style labels, sample root joint displacement trajectories, and sample motion directions. The sample action style labels can be determined according to the action style represented by each frame of action parameter true value, the sample root joint displacement can be determined according to the root joint coordinates of each frame of action parameter true value, and the sample motion direction can be determined according to the action projection direction of the controlled object between consecutive frames.

[0154] Step 802: denoising the sample noise action parameters through the action diffusion model based on the sample historical action parameters and the sample control parameters to obtain sample action parameters.

[0155] In some embodiments, after generating the sample noise motion parameter and the sample control parameter, the computer device can denoise the sample noise motion parameter through the motion diffusion model according to the sample past motion parameter and the sample control parameter to obtain a sample motion parameter.

[0156] Sample motion parameter: a motion parameter obtained by denoising the sample noise motion parameter through the motion diffusion model based on the sample past motion parameter and the sample control parameter, used to determine a model prediction loss by comparing with a motion parameter true value to train the motion diffusion model. In the denoising process, the denoising result obtained through a single diffusion step can also be used as a sample motion parameter for a motion prediction.

[0157] Sample past motion parameter: used in the training process of the motion diffusion model, used together with the sample control parameter as model input to denoise the sample noise motion parameter to obtain the sample motion parameter, which can be set to zero according to the parameter zero probability in the model training.

[0158] In a possible implementation, in the process of denoising the sample noise motion parameter through the motion diffusion model, the computer device can also use the denoising result obtained through a single diffusion step as a sample motion parameter for a motion prediction.

[0159] As shown in FIG. 9, in the process of denoising the sample noise motion parameter based on a diffusion step t, the motion diffusion model (MDM) denoises the sample noise motion parameter x t After each denoising, a sample motion parameter x After denoising the sample noise motion parameter x t After one diffusion, a sample noise motion parameter x t-1 After one more denoising and diffusion, a sample motion parameter x After denoising the sample noise motion parameter x t-2 Thus, the computer device can train the motion diffusion model according to the sample motion parameter obtained after single denoising, thereby improving the model training efficiency.

[0160] In some embodiments, considering that each action parameter true value may only contain a single action style label during the model training process, i.e., there may be no action style switching during the model training process, in order to reduce the influence of the sample historical action parameters on the model output during the model training process, the computer device can set a parameter zeroing probability, zero the sample historical action parameters through the parameter zeroing probability, so that in the absence of historical action parameters, the action diffusion model can also output more accurate action parameters.

[0161] Optionally, the parameter zeroing probability can be pre-set or determined based on the model training condition, such as 15%. The parameter zeroing probability can be pre-set, and the value range is usually between 0-1, for example, set to 15%. The parameter zeroing probability can also be dynamically determined based on the model training condition, such as setting a higher parameter zeroing probability at the beginning of model training to promote the model to learn action features with less dependence on historical action parameters, and gradually reducing the parameter zeroing probability as the training progresses to allow the model to learn more by combining historical action parameters.

[0162] In one possible implementation, during the model training process, the computer device sets the sample historical action parameters to zero according to the parameter zeroing probability. Then, in the case where the sample historical action parameters are not zeroed, the computer device denoises the sample noise action parameters according to the sample historical action parameters and the sample control parameters through the action diffusion model to obtain the sample action parameters; in the case where the sample historical action parameters are zeroed, the computer device denoises the sample noise action parameters according to the sample control parameters through the action diffusion model to obtain the sample action parameters.

[0163] Step 803, determine the model prediction loss based on the sample action parameters and the action parameter true value.

[0164] After obtaining the sample action parameters output by the action diffusion model, the computer device can determine the model prediction loss based on the sample action parameters and the action parameter true value.

[0165] Model prediction loss: determined based on the sample action parameters and the action parameter true value, used for training the action diffusion model, and can be determined based on at least one of the action prediction loss, the speed loss, the joint position loss, and the foot contact loss. Different losses can be set with corresponding loss weights.

[0166] In a possible implementation, to improve the accuracy of the motion prediction, the computer device can calculate the motion prediction loss according to the sample motion parameters and the motion parameter true value by using the mean square error loss.

[0167] Optionally, the sample motion parameters can be represented as The motion parameter true value can be represented as x0, and the motion prediction loss can be represented as

[0168] In another possible implementation, to improve the smoothness between the front and rear motions, the computer device can further calculate the velocity loss according to the sample motion parameter difference between adjacent frames in the sample motion parameters and the true value motion parameter difference between adjacent frames in the motion parameter true value by using the mean square error loss.

[0169] Optionally, the sample motion parameter difference between adjacent frames in the sample motion parameters can be represented as The true value motion parameter difference between adjacent frames in the motion parameter true value can be represented as The velocity loss can be represented as

[0170] In another possible implementation, to improve the accuracy of the joint position prediction, the computer device can first determine the sample joint position and the joint position true value by using the forward kinematics function according to the sample motion parameters and the motion parameter true value, and then calculate the joint position loss according to the sample joint position and the joint position true value by using the mean square error loss. The forward kinematics function is used in the process of determining the loss in the model training, and according to the sample motion parameters and the motion parameter true value, the function can be used to determine the sample joint position and the joint position true value, so as to calculate the joint position loss subsequently.

[0171] Optionally, the sample joint position can be represented as The joint position true value can be represented as FK(x0, S), where S is a static skeleton used in the training data, and S remains unchanged in the training process.

[0172] Optionally, the joint position loss can be represented as

[0173] In another possible implementation, to improve the authenticity of the action prediction, the computer device can further calculate a foot contact loss according to the foot contact data in the joint position ground truth and the sample foot joint displacement difference between adjacent frames in the sample joint position, by a mean square error loss calculation.

[0174] Optionally, the sample foot joint displacement difference between adjacent frames in the sample joint position can be represented as The foot contact loss can be represented as Wherein, f i is a binary foot contact mask, used when calculating the foot contact loss, represented as f i , f i , the value range of f i is {0, 1}, f i = 0 means that the foot does not contact the ground, and f sample = 1 means that the foot contacts the ground.

[0175] Further, after determining the various losses, the computer device can determine a model prediction loss according to at least one of the action prediction loss, the speed loss, the joint position loss, and the foot contact loss.

[0176] In one possible implementation, the computer device sets corresponding loss weights for different losses according to the importance of the model training of different losses, and thus determines the model prediction loss according to the action prediction loss and the corresponding first loss weight, the speed loss and the corresponding second loss weight, the joint position loss and the corresponding third loss weight, and the foot contact loss and the corresponding fourth loss weight.

[0177] Optionally, the model prediction loss can be represented as L = λ sample L sample + λ foot L foot + λ pos L pos + λ vel L vel The loss weights corresponding to different losses can be the same or different. In one possible implementation, λ sample = λ foot = λ pos = λ vel = 1.

[0178] Step 804, training the action diffusion model based on the model prediction loss.

[0179] In some embodiments, after determining the model prediction loss, the computer device can train the action diffusion model according to the model prediction loss, update the model parameters of the action diffusion model, and continue to perform sample action prediction through the updated action diffusion model until the model prediction loss reaches a minimum.

[0180] In the above embodiments, in the process of training the action diffusion model, different dimensions of loss can be determined based on different training targets, including action prediction loss, velocity loss, joint position loss, and foot contact loss. Then, according to different losses and corresponding loss weights, the model training loss is determined, and the action diffusion model is trained using the model training loss, which can realize the optimization of the action diffusion model and improve the action prediction accuracy of the action diffusion model.

[0181] Optionally, the object control method provided by the embodiments of the present application can be applied to various scenes, such as game interaction scenes, robot control scenes, etc.

[0182] For game interaction scenes:

[0183] In an optional example, the object control method provided by the embodiments of the present application is applied to a game interaction scene. Please refer to FIG. 10, which shows a schematic diagram of applying the object control method to a game interaction scene according to an example embodiment of the present application. As shown in FIG. 10, in the game interaction scene, the terminal 1002 receives the control instruction input by the player, and determines the control parameter according to the control instruction. Then, the terminal 1002 sends the control parameter and the historical action parameter to the server 1001. Based on the control parameter and the historical action parameter, the server 1001 denoises the noise action parameter sampled from the Gaussian noise through the action diffusion model to obtain the expected action parameter of the current action prediction round, and returns the expected action parameter to the terminal 1002. The terminal 1002 controls the virtual object 1003 in the game scene according to the expected action parameter.

[0184] In some embodiments, the control parameter of the virtual object 1003 can include an action style label, a root joint displacement trajectory, and a motion direction, wherein the action style label is used to control the action style of the virtual object 1003, the root joint displacement trajectory is used to control the movement trajectory of the virtual object 1003, and the motion direction is used to control the movement direction of the virtual object 1003.

[0185] In a possible implementation, the computer device encodes the historical action parameters, the action style label, the root joint displacement trajectory, and the motion direction respectively through a feature mapping layer of the action diffusion model to obtain historical action features, action style features, root joint displacement features, and motion direction features, where the action diffusion model includes multiple feature mapping layers, and the historical action parameters, the action style label, the root joint displacement trajectory, and the motion direction correspond to different feature mapping layers respectively.

[0186] In a possible implementation, in a case where the action style label of the current action prediction round is consistent with the action style label of the previous action prediction round, the computer device denoises the noise action parameters sampled from the Gaussian noise based on the historical action features, the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain the expected action parameters of the current action prediction round.

[0187] In another possible implementation, in a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, the computer device first denoises the noise action parameters sampled from the Gaussian noise based on the historical action features, the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain first candidate action parameters; then denoises the noise action parameters sampled from the Gaussian noise based on the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain second candidate action parameters; and finally obtains the expected action parameters of the current action prediction round by fusing the first candidate action parameters and the second candidate action parameters.

[0188] Regarding the process of fusing the first candidate action parameters and the second candidate action parameters, in a possible implementation, the computer device determines an action parameter difference value based on the first candidate action parameters and the second candidate action parameters; determines an action style guidance amount of the action style label of the current action prediction round based on the action parameter difference value and a guidance ratio; and determines the expected action parameters of the current action prediction round based on the second candidate action parameters and the action style guidance amount.

[0189] In some embodiments, the expected action parameters include N frames of target root joint coordinates and N frames of joint rotation parameters; and the computer device can control the virtual object 1003 based on the first M frames of target root joint coordinates and the first M frames of joint rotation parameters in the expected action parameters, where M is less than N.

[0190] In a possible implementation, to determine the root joint displacement trajectory of the next action prediction round, the computer device can project the N target root joint coordinates to the coordinate plane to obtain a target root joint displacement trajectory; in the last N-M target root joint displacement points, taking the Nth target root joint displacement point as a center symmetry point, symmetry points of the N-1th target root joint displacement point to the N-Lth target root joint displacement point are determined to obtain N+1th to N+Lth root joint extended displacement points, where L is less than N-M; based on the N+1th to N+Lth root joint extended displacement points, a root joint extended trajectory is generated. Based on the last N-M target root joint displacement points and the M root joint extended displacement points, a first candidate root joint displacement trajectory of the next action prediction round is generated; based on the motion direction of the next action prediction round, a second candidate root joint displacement trajectory of the next action prediction round is generated; based on the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory, a root joint displacement trajectory of the next action prediction round is obtained through trajectory fusion.

[0191] For a robot control scenario:

[0192] In an optional example, the object control method provided by the embodiments of the present application is applied to a robot control scenario. Please refer to FIG. 11, which shows a schematic diagram of applying the object control method to a robot control scenario according to an example embodiment of the present application. As shown in FIG. 11, in the robot control scenario, in response to a control instruction, the terminal 1101 obtains a control parameter from the control instruction, and based on the control parameter and a historical action parameter of the robot 1102, denoises a noise action parameter sampled from Gaussian noise through an action diffusion model to obtain an expected action parameter of a current action prediction round, and then controls the robot 1102 with the expected action parameter.

[0193] In some embodiments, the control parameter for the robot 1102 can include an action style label, a root joint displacement trajectory, and a motion direction, where the action style label is used to control the action style of the robot 1102, the root joint displacement trajectory is used to control the movement trajectory of the robot 1102, and the motion direction is used to control the movement direction of the robot 1102.

[0194] In a possible implementation, the computer device encodes the historical action parameter, the action style label, the root joint displacement trajectory, and the motion direction through a feature mapping layer of the action diffusion model to obtain a historical action feature, an action style feature, a root joint displacement feature, and a motion direction feature, where the action diffusion model includes multiple feature mapping layers, and the historical action parameter, the action style label, the root joint displacement trajectory, and the motion direction correspond to different feature mapping layers respectively.

[0195] In a possible implementation, in a case where the action style label of the current action prediction round is consistent with the action style label of the previous action prediction round, the computer device denoises the noise action parameters sampled from the Gaussian noise based on the historical action features, the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain the expected action parameters of the current action prediction round.

[0196] In another possible implementation, in a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, the computer device first denoises the noise action parameters sampled from the Gaussian noise based on the historical action features, the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain the first candidate action parameters; then denoises the noise action parameters sampled from the Gaussian noise based on the action style features, the root joint displacement features, and the motion direction features through the action diffusion model to obtain the second candidate action parameters; and finally obtains the expected action parameters of the current action prediction round by fusing the first candidate action parameters and the second candidate action parameters.

[0197] Regarding the process of fusing the first candidate action parameters and the second candidate action parameters, in a possible implementation, the computer device determines an action parameter difference value based on the first candidate action parameters and the second candidate action parameters; determines an action style guidance amount of the action style label of the current action prediction round based on the action parameter difference value and a guidance ratio; and determines the expected action parameters of the current action prediction round based on the second candidate action parameters and the action style guidance amount.

[0198] In some embodiments, the expected action parameters include N frames of target root joint coordinates and N frames of joint rotation parameters; and the computer device can control the robot 1102 based on the first M frames of target root joint coordinates and the first M frames of joint rotation parameters in the expected action parameters, where M is less than N.

[0199] In a possible implementation, to determine the root joint displacement trajectory of the next action prediction round, the computer device can project the N target root joint coordinates to the coordinate plane to obtain a target root joint displacement trajectory; in the last N-M target root joint displacement points, taking the Nth target root joint displacement point as a center of symmetry, the symmetric points of the N-1th target root joint displacement point to the N-Lth target root joint displacement point are determined to obtain the N+1th to N+Lth root joint extended displacement points, where L is less than N-M; based on the N+1th to N+Lth root joint extended displacement points, a root joint extended trajectory is generated. Based on the last N-M target root joint displacement points and the M root joint extended displacement points, a first candidate root joint displacement trajectory of the next action prediction round is generated; based on the motion direction of the next action prediction round, a second candidate root joint displacement trajectory of the next action prediction round is generated; based on the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory, a root joint displacement trajectory of the next action prediction round is obtained through trajectory fusion.

[0200] In summary of the above embodiments, the object control method provided by the embodiments has at least the following beneficial effects:

[0201] 1. The object control method provided by the embodiments is an action prediction process based on an action diffusion model, and after the model training is completed, the method can generate diverse and high-quality actions at a speed of 60 fps, thereby improving the practicability of the method.

[0202] 2. The embodiments combine the action sampling principle without a classifier guide, can realize the diversity of action prediction styles, and the smooth transition of actions when switching between different styles. For example, as shown in FIG. 12, a comparison diagram of the object control method and related technologies provided by the embodiments when the action style is switched from “jumping with the right foot” to “jumping with the left foot” is shown. The corresponding controlled object 1201 of the embodiments can sequentially complete the switching from jumping with the right foot to jumping with the left foot, while the controlled object 1202 based on LMP (Local Motion Phases), the controlled object 1203 based on MANN-DP (Memory Augmented Neural Network-Deep Phase), and the controlled object 1204 based on MM-DP (Motion Matching-Deep Phase) cannot successfully complete the switching from jumping with the right foot to jumping with the left foot.

[0203] 3. The embodiment of the present application realizes action prediction by adopting the action diffusion model, and can also realize action diversification in a single style. Illustratively, as shown in FIG. 13, taking the action style as a drunken dizzy state and the moving direction as right as an example, the corresponding controlled object 1301 of the present application has a stable root node movement and good left and right hand action transformation effect in the process of moving to the right; while the controlled object 1302 based on LMP, the controlled object 1303 based on MANN-DP and the controlled object 1304 based on MM-DP have a larger root node movement curve and a poorer left and right hand action transformation effect in the process of moving to the right.

[0204] 4. The object control method provided by the embodiment of the present application can produce better control quality and more diverse action styles.

[0205] Illustratively, as shown in Table 1, quantitative results of a single-style character control experiment using the object control method provided by the embodiments of the present application are shown. As can be seen from Table 1, the action quality and condition matching capability of the object control method are relatively good, wherein the indicators for measuring the action quality can include the distance (Frechet Inception Distance, FID) between the generated motion distribution and the training distribution, the foot sliding distance (Foot Sliding Distance, Ft.Slid), and the smoothness (Acceleration, Accel), and the indicators for measuring the condition matching capability can include the trajectory error (Trajectory Error, Traj.Err), the orientation error (Orientation Error, Orient.Err), and the style generation rate (Style Acceleration, Styl.Acc). Frechet Inception Distance (FID) is one of the indicators for measuring the action quality, used to represent the distance between the generated motion distribution and the training distribution, and the smaller the value, the better the action quality. Foot Sliding Distance (Ft.Slid) is one of the indicators for measuring the action quality, indicating the distance of foot sliding, and the smaller the value, the better the action quality. Acceleration (Accel) is one of the indicators for measuring the action quality, used to represent the smoothness of the action, and the smaller the value, the smoother the action, and the better the action quality. Trajectory Error (Traj.Err) is one of the indicators for measuring the condition matching capability, indicating the trajectory error, and the smaller the value, the better the condition matching capability. Orientation Error (Orient.Err) is one of the indicators for measuring the condition matching capability, indicating the orientation error, and the smaller the value, the better the condition matching capability. Style Acceleration (Styl.Acc) is one of the indicators for measuring the condition matching capability, indicating the style generation rate, and the larger the value, the better the condition matching capability. Transition Duration (Trans.Dura) is one of the indicators for measuring the style transition, indicating the number of frames required for style transition, and the smaller the value, the more efficient the style transition. Transition Success Rate (Succ.Rate) is one of the indicators for measuring the style transition, indicating the success rate of style transition, and the larger the value, the more reliable the style transition.

[0206] Table 1

[0207] Illustratively, as shown in Table 2, quantitative results of a multi-style conversion experiment using the object control method provided in the embodiments of the present application are shown. As can be seen from Table 2, the object control method can reliably and efficiently convert to a new target style in the multi-style conversion experiment, and requires fewer transition frames and a very high success rate, wherein the indicators for measuring style conversion can include the number of frames required for style conversion (Transition Duration, Trans. Dura) and the success rate of style conversion (Transition Success Rate, Succ. Rate).

[0208] Table 2

[0209] Referring to FIG. 14, a structural block diagram of an object control device provided in an example embodiment of the present application is shown, which includes:

[0210] The parameter acquisition module 1401 is configured to acquire historical action parameters and control parameters, wherein the historical action parameters represent historical actions performed by the controlled object before the current action prediction round, and the control parameters represent the action expectation of the controlled object in the current action prediction round;

[0211] The action prediction module 1402 is configured to denoise a noise action parameter sampled from Gaussian noise based on the historical action parameters and the control parameters, through an action diffusion model, to obtain an expected action parameter in the current action prediction round.

[0212] The control module 1403 is configured to control the controlled object according to the expected action parameter.

[0213] Optionally, the control parameters include an action style label, a root joint displacement trajectory, and a motion direction, wherein the action style label is used to control the action style of the controlled object, the root joint displacement trajectory is used to control the movement trajectory of the controlled object, and the motion direction is used to control the movement direction of the controlled object.

[0214] The action prediction module 1402 includes:

[0215] The feature encoding unit is configured to respectively encode the historical action parameters through a feature mapping layer of the action diffusion model to obtain historical action features, encode the action style label to obtain an action style feature, encode the root joint displacement trajectory to obtain a root joint displacement feature, and encode the motion direction to obtain a motion direction feature.

[0216] The action prediction unit is configured to denoise the noise action parameter based on the historical action features, the action style feature, the root joint displacement feature, and the motion direction feature, through the action diffusion model, to obtain the expected action parameter in the current action prediction round.

[0217] Optionally, the action diffusion model comprises a plurality of feature mapping layers, and the historical action parameters, the action style label, the root joint displacement trajectory, and the motion direction correspond to different feature mapping layers respectively.

[0218] Optionally, the action prediction unit is configured to:

[0219] In a case where the action style label of the current action prediction round is consistent with the action style label of the previous action prediction round, the action diffusion model is used to denoise the noise action parameters based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, to obtain the expected action parameters of the current action prediction round.

[0220] Optionally, the action prediction unit is configured to:

[0221] In a case where the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, the action diffusion model is used to perform twice denoising on the noise action parameters based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, to obtain twice denoising results, wherein the historical action features are set to zero in one denoising process.

[0222] The expected action parameters of the current action prediction round are determined based on the twice denoising results.

[0223] Optionally, the action prediction unit is configured to:

[0224] The action diffusion model is used to denoise the noise action parameters based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, to obtain the first candidate action parameters.

[0225] The action diffusion model is used to denoise the noise action parameters based on the action style features, the root joint displacement features, and the motion direction features, to obtain the second candidate action parameters.

[0226] The action prediction unit is further configured to:

[0227] The expected action parameters of the current action prediction round are obtained by fusing the first candidate action parameters and the second candidate action parameters.

[0228] Optionally, the action prediction unit is further configured to:

[0229] The action parameter difference value is determined based on the first candidate action parameters and the second candidate action parameters.

[0230] determine an action style guidance amount of the action style label of the current action prediction round based on the action parameter difference value and a guidance ratio, the guidance ratio being used to adjust an influence degree of the first candidate action parameter on the expected action parameter, and the action style guidance amount being an influence amount of the first candidate action parameter on the expected action parameter;

[0231] determine the expected action parameter of the current action prediction round based on the second candidate action parameter and the action style guidance amount.

[0232] Optionally, the expected action parameter comprises N frames of target root joint coordinates and N frames of joint rotation parameters.

[0233] The control module 1403 is configured to:

[0234] control the controlled object based on the first M frames of target root joint coordinates and the first M frames of joint rotation parameters in the expected action parameter, where M is less than N.

[0235] The device further comprises:

[0236] The coordinate projection module is configured to project the N frames of target root joint coordinates to a coordinate plane to obtain a target root joint displacement trajectory.

[0237] The trajectory expansion module is configured to obtain a root joint expansion trajectory based on the last N-M frames of target root joint displacement points in the target root joint displacement trajectory through trajectory expansion, the root joint expansion trajectory comprising M frames of root joint expansion displacement points.

[0238] The trajectory determination module is configured to determine a root joint displacement trajectory of a next action prediction round based on the last N-M frames of target root joint displacement points and the M frames of root joint expansion displacement points.

[0239] Optionally, the trajectory expansion module is configured to:

[0240] determine a symmetric point of the N-1th target root joint displacement point to the N-Lth target root joint displacement point in the last N-M frames of target root joint displacement points, with the Nth target root joint displacement point as a center symmetric point, to obtain N+1th to N+Lth root joint expansion displacement points, where L is less than N-M.

[0241] generate the root joint expansion trajectory based on the N+1th to N+Lth root joint expansion displacement points.

[0242] Optionally, the trajectory determination module is configured to:

[0243] generate a first candidate root joint displacement trajectory of the next action prediction round based on the last N-M frames of target root joint displacement points and the M frames of root joint expansion displacement points.

[0244] generate a second candidate root joint displacement trajectory of the next action prediction round based on the motion direction of the next action prediction round;

[0245] obtain a root joint displacement trajectory of the next action prediction round through trajectory fusion based on the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory.

[0246] Optionally, the device further comprises:

[0247] a sample generation module configured to generate a sample noise action parameter and a sample control parameter based on an action parameter true value;

[0248] a sample action prediction module configured to denoise the sample noise action parameter through an action diffusion model based on a sample historical action parameter and the sample control parameter to obtain a sample action parameter;

[0249] a loss determination module configured to determine a model prediction loss based on the sample action parameter and the action parameter true value;

[0250] a model training module configured to train the action diffusion model based on the model prediction loss.

[0251] Optionally, the loss determination module is configured to:

[0252] obtain an action prediction loss through a mean squared error loss calculation based on the sample action parameter and the action parameter true value;

[0253] obtain a velocity loss through a mean squared error loss calculation based on a sample action parameter difference between adjacent frames in the sample action parameter and a true value action parameter difference between adjacent frames in the action parameter true value;

[0254] determine a sample joint position and a joint position true value through a forward kinematics function based on the sample action parameter and the action parameter true value;

[0255] obtain a joint position loss through a mean squared error loss calculation based on the sample joint position and the joint position true value;

[0256] obtain a foot contact loss through a mean squared error loss calculation based on foot contact data in the joint position true value and a sample foot joint displacement difference between adjacent frames in the sample joint position;

[0257] determine the model prediction loss based on at least one of the action prediction loss, the velocity loss, the joint position loss, and the foot contact loss.

[0258] Optionally, the sample action prediction module is configured to:

[0259] zero the sample historical action parameter based on a parameter zeroing probability;

[0260] In the case that the sample historical action parameter is not zeroed, the sample noise action parameter is denoised by the action diffusion model based on the sample historical action parameter and the sample control parameter to obtain a sample action parameter.

[0261] In the case that the sample historical action parameter is zeroed, the sample noise action parameter is denoised by the action diffusion model based on the sample control parameter to obtain a sample action parameter.

[0262] To sum up, in the embodiment of the present application, in the process of predicting the action parameter by using the action diffusion model, the long-time action parameter output at a time is converted into the short-time action parameter output in multiple rounds, the required time length of the action parameter generated each time is shortened, the requirement for the model parameter quantity of the action diffusion model is reduced, and the requirement for the model deployment condition is reduced. In addition to the control parameter, the historical action parameter is added as a diffusion condition, the historical action performed by the controlled object before each round of action prediction is referred to, the continuity between actions in the phased output is improved, and the output quality of the long-time action parameter is ensured.

[0263] It should be noted that the device provided in the above embodiment is only exemplified by the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be described here.

[0264] It should be noted that before obtaining the relevant user data and during the process of obtaining the relevant user data, the prompt interface, pop-up window or output voice prompt information can be displayed. The prompt interface, pop-up window or voice prompt information is used to prompt the user that the relevant data of the user is currently being collected, so that the application only starts to perform the related steps of obtaining the relevant data of the user after obtaining the confirmation operation of the user to the prompt interface or pop-up window, otherwise (i.e. without obtaining the confirmation operation of the user to the prompt interface or pop-up window), ending the related steps of obtaining the relevant data of the user, i.e. not obtaining the relevant data of the user. In other words, the information (including but not limited to user device information, user personal information, etc., and operation data corresponding to the user) and data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the country and region.

[0265] Referring to FIG. 15, a structural diagram of a computer device is shown according to an example embodiment of the present application. Specifically, the computer device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including a random access memory 1502 and a read-only memory 1503, and a system bus 1505 connecting the system memory 1504 and the central processing unit 1501. The computer device 1500 can also include a basic input / output system (I / O) 1506 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1507 for storing an operating system 1513, application programs 1514, and other program modules 1515.

[0266] In some embodiments, the basic input / output system 1506 includes a display 1508 for displaying information and an input device 1509, such as a mouse, keyboard, or the like, for inputting information into the computer. The display 1508 and input device 1509 are connected to the central processing unit 1501 through the input / output controller 1510, which is connected to the system bus 1505. The basic input / output system 1506 can also include the input / output controller 1510 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1510 provides output to a display screen, printer, or other type of output device.

[0267] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 and its associated computer-readable media provide non-volatile storage for the computer device 1500. That is, the mass storage device 1507 can include a computer-readable medium (not shown) such as a hard disk or drive.

[0268] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes Random Access Memory (RAM), Read Only Memory (ROM), flash memory or other solid state memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that the computer storage media is not limited to the above-mentioned several. The system memory 1504 and the mass storage device 1507 mentioned above can be collectively referred to as memory.

[0269] The memory stores one or more programs configured to be executed by the one or more central processing units 1501, and the one or more programs contain instructions for implementing the above method. The central processing unit 1501 executes the one or more programs to implement the object control method provided by each method embodiment.

[0270] According to various embodiments of the present application, the computer device 1500 can also be connected to a remote computer running on a network such as the Internet through a network. That is, the computer device 1500 can be connected to the network 1511 through the network interface unit 1512 connected to the system bus 1505, or the network interface unit 1512 can also be used to connect to other types of networks or remote computer systems (not shown).

[0271] The embodiments of the present application also provide a computer readable storage medium, which stores at least one instruction loaded and executed by a processor to implement the object control method described in the above embodiments.

[0272] Optionally, the computer readable storage medium can include ROM, RAM, solid state disk (SSD, Solid State Drives) or optical disk, etc. Among them, the RAM can include resistance random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).

[0273] The embodiment of the present application provides a computer program product, which comprises at least one instruction stored in a computer readable storage medium. The processor of the computer device reads the at least one instruction from the computer readable storage medium, and the processor executes the at least one instruction, so that the computer device executes the object control method described in the above embodiment.

[0274] A person skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0275] To sum up, in the scheme in the embodiment of the present application, the computer device acquires a historical action parameter representing a historical action of the controlled object before a current action prediction round and a control parameter representing an action expectation of the controlled object in the current action prediction round, then denoises a noise action parameter sampled from Gaussian noise based on the two parameters through an action diffusion model to obtain an expected action parameter in the current action prediction round, and finally controls the controlled object according to the expected action parameter. The scheme converts single-time output long-time action parameters into multi-round output short-time action parameters, reduces the requirement for the model parameter quantity of the action diffusion model and the model deployment condition requirement, simultaneously increases the historical action parameter as a diffusion condition, improves the continuity between actions in the stage-by-stage output, and guarantees the output quality of the long-time action parameters.

[0276] When the control parameter comprises an action style label for controlling the action style of the controlled object, a root joint displacement trajectory for controlling a movement trajectory, and a motion direction for controlling a movement direction, the computer device first encodes the historical action parameter, the action style label, the root joint displacement trajectory, and the motion direction through a feature mapping layer of the action diffusion model to obtain a historical action feature, an action style feature, a root joint displacement feature, and a motion direction feature, and then denoises the noise action parameter through the action diffusion model based on the features to obtain the expected action parameter in the current action prediction round. This scheme encodes different parameters independently by setting the feature mapping layer in the action diffusion model, avoids the coupling or imbalance problem between features, improves the effectiveness of each feature, and further improves the stability and accuracy of action prediction.

[0277] The action diffusion model comprises a plurality of feature mapping layers, and the historical action parameter, the action style label, the root joint displacement trajectory, and the motion direction correspond to different feature mapping layers respectively. This further ensures the independence of the feature encoding of each parameter, more effectively avoids the mutual interference between features, and improves the stability and accuracy of action prediction.

[0278] If the action style label of the current action prediction round is consistent with the action style label of the last action prediction round, the computer device directly denoises the noise action parameters through the action diffusion model based on the historical action features, the action style features, the root joint displacement features, and the motion direction features to obtain the expected action parameters of the current action prediction round. In this way, the continuity and stability of action prediction are ensured when the action style does not change.

[0279] When the action style label of the current action prediction round is inconsistent with the action style label of the last action prediction round, the computer device performs twice denoising on the noise action parameters through the action diffusion model based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, wherein the historical action features are set to zero in the first denoising process, and the twice denoising results are obtained, and then the expected action parameters of the current action prediction round are determined based on the twice denoising results. This method can reduce the influence of historical action parameters on the current action prediction result, make the action prediction closer to the action style indicated by the current action style label, and improve the action prediction accuracy.

[0280] The twice denoising processes are respectively based on the historical action features, the action style features, the root joint displacement features, and the motion direction features to denoise the noise action parameters through the action diffusion model to obtain the first candidate action parameters, and based on the action style features, the root joint displacement features, and the motion direction features to denoise the noise action parameters through the action diffusion model to obtain the second candidate action parameters. Then, the first candidate action parameters and the second candidate action parameters are fused to obtain the expected action parameters of the current action prediction round. This scheme clearly specifies the specific way of determining the expected action parameters through twice denoising and fusion results when the action style is switched, and guarantees the action prediction accuracy.

[0281] The process of fusing the first candidate action parameters and the second candidate action parameters to obtain the expected action parameters of the current action prediction round is that the action parameter difference value is determined based on the first candidate action parameters and the second candidate action parameters, the action style guiding amount of the action style label of the current action prediction round is determined based on the action parameter difference value and the guiding proportion, the guiding proportion is used to adjust the influence degree of the first candidate action parameters on the expected action parameters, the action style guiding amount is the influence amount of the first candidate action parameters on the expected action parameters, and finally the expected action parameters of the current action prediction round are determined based on the second candidate action parameters and the action style guiding amount. This scheme details the quantization process of determining the expected action parameters when the action style is switched, and improves the accuracy and operability of action prediction.

[0282] If the desired motion parameters include N frames of target root joint coordinates and N frames of joint rotation parameters, the computer device controls the controlled object based on the first M frames of target root joint coordinates and the first M frames of joint rotation parameters in the desired motion parameters (M is less than N), then projects the N frames of target root joint coordinates to the coordinate plane to obtain a target root joint displacement trajectory, then obtains a root joint extension trajectory including M frames of root joint extension displacement points based on the last N-M frames of target root joint displacement points in the target root joint displacement trajectory through trajectory extension, and finally determines the root joint displacement trajectory of the next motion prediction round based on the last N-M frames of target root joint displacement points and the M frames of root joint extension displacement points. This scheme reduces the jitter problem in the motion execution process through segmented use of the desired motion parameters and trajectory processing, and improves the motion continuity of the controlled object between two motion predictions.

[0283] The process of obtaining a root joint extension trajectory through trajectory extension based on the last N-M frames of target root joint displacement points in the target root joint displacement trajectory is as follows: in the last N-M frames of target root joint displacement points, the N-1th target root joint displacement point to the N-Lth target root joint displacement point is determined as a symmetric point with the Nth target root joint displacement point as the center (L is less than N-M), to obtain the N+1th to N+Lth root joint extension displacement points, and a root joint extension trajectory is generated based on these root joint extension displacement points. This scheme details the specific way of trajectory extension, which helps to generate a reasonable root joint displacement trajectory for the next motion prediction round and further improves motion continuity.

[0284] The process of determining the root joint displacement trajectory of the next motion prediction round based on the last N-M frames of target root joint displacement points and the M frames of root joint extension displacement points is as follows: first, a first candidate root joint displacement trajectory of the next motion prediction round is generated based on the last N-M frames of target root joint displacement points and the M frames of root joint extension displacement points, then a second candidate root joint displacement trajectory of the next motion prediction round is generated based on the motion direction of the next motion prediction round, and finally the root joint displacement trajectory of the next motion prediction round is obtained through trajectory fusion based on the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory. This scheme considers multiple factors to determine the root joint displacement trajectory of the next motion prediction round, optimizes trajectory generation, and improves motion continuity and control effect.

[0285] The computer device generates a sample noise action parameter and a sample control parameter based on the true value of the action parameter, then denoises the sample noise action parameter based on the sample historical action parameter and the sample control parameter through the action diffusion model to obtain a sample action parameter, determines a model prediction loss based on the sample action parameter and the true value of the action parameter, and finally trains the action diffusion model based on the model prediction loss. Through training of the action diffusion model, different dimensional losses are determined based on different training targets, the model is trained using the losses, the action diffusion model is optimized, and the action prediction accuracy is improved.

[0286] The process of determining the model prediction loss based on the sample action parameter and the true value of the action parameter includes: calculating an action prediction loss through mean square error loss based on the sample action parameter and the true value of the action parameter; calculating a speed loss through mean square error loss based on the sample action parameter difference between adjacent frames in the sample action parameter and the true value action parameter difference between adjacent frames in the true value of the action parameter; determining a sample joint position and a joint position true value through forward kinematics based on the sample action parameter and the true value of the action parameter, and then calculating a joint position loss through mean square error loss based on the sample joint position and the joint position true value; calculating a foot landing loss through mean square error loss based on the foot landing data in the joint position true value and the sample foot joint displacement difference between adjacent frames in the sample joint position; and finally determining the model prediction loss based on at least one of the action prediction loss, the speed loss, the joint position loss, and the foot landing loss. This scheme describes in detail the specific calculation methods for determining the model prediction loss, thereby ensuring the action prediction accuracy from multiple dimensions.

[0287] The process of denoising the sample noise action parameter based on the sample historical action parameter and the sample control parameter through the action diffusion model to obtain the sample action parameter includes: setting the sample historical action parameter to zero based on a parameter zero setting probability; in the case where the sample historical action parameter is not set to zero, denoising the sample noise action parameter based on the sample historical action parameter and the sample control parameter through the action diffusion model to obtain the sample action parameter; and in the case where the sample historical action parameter is set to zero, denoising the sample noise action parameter based on the sample control parameter through the action diffusion model to obtain the sample action parameter. This scheme sets the parameter zero setting probability to reduce the influence of the sample historical action parameter on the model output during model training, so that the model can output more accurate action parameters in different situations.

[0288] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0289] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for controlling an object, executed by a computer device, the method comprising: Obtaining historical action parameters and control parameters, wherein the historical action parameters represent historical actions performed by the controlled object before the current action prediction round, and the control parameters represent the action expectations of the controlled object in the current action prediction round; Based on the historical action parameters and the control parameters, denoising the noisy action parameters obtained from Gaussian noise sampling using a motion diffusion model to obtain expected action parameters for the current action prediction round; and The controlled object is controlled according to the desired action parameter.

2. The method according to claim 1, wherein the control parameters include a motion style label, a root joint displacement trajectory, and a motion direction; the motion style label is used to control the motion style of the controlled object; the root joint displacement trajectory is used to control the movement trajectory of the controlled object; and the motion direction is used to control the movement direction of the controlled object; The method of denoising the noise action parameters obtained by sampling Gaussian noise based on the historical action parameters and the control parameters by using an action diffusion model to obtain the expected action parameters of the current action prediction round includes: Through the feature mapping layer of the motion diffusion model, the historical motion parameters are respectively encoded to obtain historical motion features, the motion style label is encoded to obtain motion style features, the root joint displacement trajectory is encoded to obtain root joint displacement features, and the motion direction is encoded to obtain motion direction features; Based on the historical action features, the action style features, the root joint displacement features and the motion direction features, the noise action parameters are denoised using the action diffusion model to obtain the expected action parameters of the current action prediction round.

3. According to the method of claim 2, the motion diffusion model includes multiple feature mapping layers, and the historical motion parameters, the motion style label, the root joint displacement trajectory and the motion direction correspond to different feature mapping layers respectively.

4. The method according to claim 2 or 3, wherein the step of denoising the noisy action parameters using the action diffusion model based on the historical action features, the action style features, the root joint displacement features, and the motion direction features to obtain the expected action parameters for the current action prediction round comprises: When the action style label of the current action prediction round is consistent with the action style label of the previous action prediction round, based on the historical action features, the action style features, the root joint displacement features and the motion direction features, the noise action parameters are denoised through the action diffusion model to obtain the expected action parameters of the current action prediction round.

5. The method according to any one of claims 2 to 4, wherein the step of denoising the noisy action parameters using the action diffusion model based on the historical action features, the action style features, the root joint displacement features, and the motion direction features to obtain the expected action parameters for the current action prediction round comprises: When the action style label of the current action prediction round is inconsistent with the action style label of the previous action prediction round, performing denoising twice on the noisy action parameter using the action diffusion model based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, to obtain two denoising results, wherein the historical action features are reset to zero during one denoising process; The expected action parameters of the current action prediction round are determined based on the two denoising results.

6. The method according to claim 5, wherein the denoising is performed twice on the noisy motion parameters using the motion diffusion model based on the historical motion features, the motion style features, the root joint displacement features, and the motion direction features to obtain the two denoising results, comprising: Based on the historical action features, the action style features, the root joint displacement features, and the motion direction features, denoising the noise action parameters using the action diffusion model to obtain first candidate action parameters; Based on the action style feature, the root joint displacement feature, and the motion direction feature, denoising the noise action parameter using the action diffusion model to obtain a second candidate action parameter; The determining the expected action parameter of the current action prediction round based on the two denoising results includes: The first candidate action parameters and the second candidate action parameters are fused to obtain the expected action parameters of the current action prediction round.

7. The method according to claim 6, wherein fusing the first candidate action parameter and the second candidate action parameter to obtain the expected action parameter of the current action prediction round comprises: Determining an action parameter difference based on the first candidate action parameter and the second candidate action parameter; Determining an action style guidance amount of the action style label for the current action prediction round based on the action parameter difference and the guidance ratio, wherein the guidance ratio is used to adjust the degree of influence of the first candidate action parameter on the expected action parameter, and the action style guidance amount is the influence of the first candidate action parameter on the expected action parameter; Based on the second candidate action parameters and the action style guidance amount, the expected action parameters of the current action prediction round are determined.

8. The method according to any one of claims 2 to 7, wherein the desired motion parameters include N frames of target root joint coordinates and N frames of joint rotation parameters; The controlling the controlled object according to the expected action parameter includes: Controlling the controlled object based on the target root joint coordinates of the first M frames and the joint rotation parameters of the first M frames in the desired motion parameters, where M is less than N; The method further comprises: Projecting the target root joint coordinates of the N frames onto a coordinate plane to obtain a target root joint displacement trajectory; Based on the target root joint displacement points of the subsequent NM frames in the target root joint displacement trajectory, a root joint extended trajectory is obtained by trajectory extension, wherein the root joint extended trajectory includes M frames of root joint extended displacement points; Based on the target root joint displacement points of the subsequent NM frames and the extended root joint displacement points of the M frames, the root joint displacement trajectory of the next action prediction round is determined.

9. The method according to claim 8, wherein the step of obtaining a root joint extension trajectory by trajectory extension based on the target root joint displacement point in the subsequent NM frames in the target root joint displacement trajectory comprises: Among the target root joint displacement points of the subsequent NM frames, taking the target root joint displacement point of the Nth frame as the central symmetric point, determine the symmetric points from the target root joint displacement point of the N-1th frame to the target root joint displacement point of the NLth frame, and obtain the root joint extended displacement points of the N+1th to N+Lth frames, where L is less than NM; The root joint extension trajectory is generated based on the root joint extension displacement points of the N+1th to N+Lth frames.

10. The method according to claim 8 or 9, wherein determining the root joint displacement trajectory of the next action prediction round based on the target root joint displacement points of the subsequent NM frames and the extended root joint displacement points of the M frames comprises: Generate a first candidate root joint displacement trajectory for the next action prediction round based on the target root joint displacement points of the subsequent NM frames and the extended root joint displacement points of the M frames; generating a second candidate root joint displacement trajectory for the next action prediction round based on the motion direction of the next action prediction round; Based on the first candidate root joint displacement trajectory and the second candidate root joint displacement trajectory, the root joint displacement trajectory of the next action prediction round is obtained through trajectory fusion.

11. The method according to any one of claims 1 to 10, further comprising: Based on the true value of the action parameter, generate sample noise action parameters and sample control parameters; Based on the sample historical action parameters and the sample control parameters, the sample noise action parameters are denoised by the action diffusion model to obtain sample action parameters; Determining a model prediction loss based on the sample action parameter and the true value of the action parameter; The action diffusion model is trained based on the model prediction loss.

12. The method according to claim 11, wherein determining the model prediction loss based on the sample action parameter and the true value of the action parameter comprises: Based on the sample action parameters and the true values ​​of the action parameters, the action prediction loss is calculated by using the mean square error loss; Based on the sample action parameter differences between adjacent frames in the sample action parameter and the true action parameter differences between adjacent frames in the true action parameter, a speed loss is calculated by a mean square error loss; Based on the sample motion parameters and the motion parameter true values, determining the sample joint positions and the joint position true values ​​through a forward kinematics function; Based on the sample joint position and the true value of the joint position, a joint position loss is calculated by using a mean square error loss; Calculating a foot contact loss by using a mean square error loss based on the foot contact data in the true value of the joint position and the sample foot joint displacement difference between adjacent frames in the sample joint position; The model prediction loss is determined based on at least one of the motion prediction loss, the speed loss, the joint position loss, and the foot contact loss.

13. The method according to claim 11 or 12, wherein the denoising of the sample noise action parameters by the action diffusion model based on the sample historical action parameters and the sample control parameters to obtain the sample action parameters comprises: Based on the parameter zeroing probability, setting the sample historical action parameter to zero; In a case where the sample historical action parameter is not set to zero, based on the sample historical action parameter and the sample control parameter, the sample noise action parameter is denoised by the action diffusion model to obtain the sample action parameter; When the sample historical action parameter is set to zero, the sample noise action parameter is denoised by the action diffusion model based on the sample control parameter to obtain the sample action parameter.

14. An object control device, comprising: A parameter acquisition module, configured to acquire historical action parameters and control parameters, wherein the historical action parameters represent historical actions performed by the controlled object before the current action prediction round, and the control parameters represent the action expectations of the controlled object in the current action prediction round; An action prediction module is configured to denoise the noisy action parameters obtained by sampling Gaussian noise using a motion diffusion model based on the historical action parameters and the control parameters to obtain expected action parameters for the current action prediction round; and A control module is used to control the controlled object according to the desired action parameter.

15. A computer device, comprising a processor and a memory; the memory storing at least one instruction, wherein the at least one instruction is configured to be executed by the processor to implement the object control method according to any one of claims 1 to 13. 16 . A computer-readable storage medium storing at least one instruction, wherein the at least one instruction is configured to be executed by a processor to implement the object control method according to claim 1 .

17. A computer program product, comprising at least one instruction, wherein the at least one instruction is stored in a computer-readable storage medium; a processor of a computer device reads the at least one instruction from the computer-readable storage medium, and the processor executes the at least one instruction, so that the computer device implements the object control method according to any one of claims 1 to 13.

Images