A control method and device of a robot, a medium, and a robot

Abstract

This application discloses a robot control method, device, and medium, and a robot, relating to the field of artificial intelligence. The method includes: acquiring a set of reference states for the robot, where the reference states include the robot's reference body posture and reference motor angles; acquiring the robot's current state, where the current state includes the robot's current body posture and current motor angles in the current cycle; calling a machine learning model to predict the current state using the set of reference states as a reference, and outputting target motor angles for m joint motors; converting the target motor angles for the m joint motors into torque commands for the m joint motors, where the torque commands are used to control the operation of the m joint motors; and controlling the robot using the torque commands for the m joint motors. This application enables the robot to operate with a set of reference states as a reference, and the robot's operation is smooth and realistic.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus and medium for controlling robots, and a robot. Background Technology

[0002] In AI (Artificial Intelligence) technology, robots can simulate the movements of real animals and make realistic and fluid movements.

[0003] The relevant technology first models the environment in which the robot is located, and then, during the operation of the robot, it solves the optimization problem at each time step of the robot based on the kinematic equations to obtain the optimal control signal at the current time step, and controls the robot's actions according to the optimal control signal.

[0004] However, the relevant technologies require a large amount of computing power to calculate the optimal control signals, which is not suitable for scenarios where robots perform complex actions. Summary of the Invention

[0005] This application provides a robot control method, device, and medium, as well as a robot. The method uses a machine learning model to determine the robot's next action, resulting in smooth and realistic robot operation. The technical solution is as follows:

[0006] According to one aspect of this application, a method for controlling a robot is provided. The robot includes a body and m joint motors, where m is a positive integer greater than 1. The method includes:

[0007] Obtain a reference state set for the robot, wherein the reference states in the reference state set include the robot's reference body posture and reference motor rotation angle;

[0008] Obtain the current state of the robot, which includes the robot's current body posture and current joint motor rotation angle in the current cycle;

[0009] The machine learning model is invoked to predict the current state using the reference state set as a reference object, and the target motor rotation angle of the m joint motors is output.

[0010] The target motor rotation angle of the m joint motors is converted into a torque command for the m joint motors, and the torque command is used to control the operation of the m joint motors;

[0011] The robot is controlled by torque commands from the m joint motors.

[0012] According to another aspect of this application, a control device for a robot is provided. The robot includes a body and m joint motors, where m is a positive integer greater than 1. The device includes:

[0013] The acquisition module is used to acquire the robot's reference state set and current state. The reference states in the reference state set include the robot's reference body posture and reference motor rotation angle.

[0014] The acquisition module is further configured to acquire the current state of the robot, which includes the robot's current body posture and current motor rotation angle in the current cycle;

[0015] The control module is used to call the machine learning model, using the reference state set as a reference object, to predict the current state and output the target motor rotation angle of the m joint motors;

[0016] The control module is also used to convert the target motor rotation angle of the m joint motors into torque commands for the m joint motors, and the torque commands are used to control the operation of the m joint motors;

[0017] The control module is also used to control the robot through torque commands of the m joint motors.

[0018] According to another aspect of this application, a method for training a robot control model is provided. The robot includes a body and m joint motors, where m is a positive integer greater than 1. The method includes:

[0019] Obtain a sample reference state set for the robot, wherein the sample reference states in the sample reference state set include the sample reference body posture and sample reference motor rotation angle of the robot;

[0020] Obtain the current state of the robot sample, which includes the current body posture and current motor rotation angle of the robot sample;

[0021] The robot control model is invoked, and the current state of the sample is predicted using the sample reference state set as a reference object. The sample target motor rotation angle of the m joint motors is then output.

[0022] Calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample joint motor rotation angle of the m joint motors;

[0023] Calculate the reward value corresponding to the rotation angle of the sample reference motor for the m joint motors;

[0024] The robot control model is trained based on the difference, with the goal of maximizing the reward value.

[0025] According to another aspect of this application, a training device for a robot control model is provided. The robot includes a body and m joint motors, where m is a positive integer greater than 1. The device includes:

[0026] The sample acquisition module is used to acquire a sample reference state set of the robot, wherein the sample reference state in the sample reference state set includes the sample reference body posture and the sample reference motor rotation angle of the robot;

[0027] The sample acquisition module is also used to acquire the current sample state of the robot, which includes the current body posture of the robot and the current motor rotation angle of the robot.

[0028] The sample determination module is used to call the robot control model, use the sample reference state set as a reference object, predict the current state of the sample, and output the sample target motor rotation angle of the m joint motors;

[0029] The sample calculation module is used to calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample target motor rotation angle of the m joint motors; and to calculate the reward value corresponding to the sample reference motor rotation angle of the m joint motors.

[0030] A sample training module is used to train the robot control model based on the difference with the goal of maximizing the reward value.

[0031] According to another aspect of this application, a robot is provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the robot control method as described above, or, a robot control model training method.

[0032] According to another aspect of this application, a computer storage medium is provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the program code being loaded and executed by a processor to implement the robot control method as described above, or the robot control model training method.

[0033] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robot control method described above, or the robot control model training method.

[0034] The beneficial effects of the technical solutions provided in this application include at least the following:

[0035] This method utilizes a machine learning model to predict the current state of the robot, using a set of reference states as a reference. The target motor angle is then converted into a torque command, which controls the robot's behavior. Given a set of reference states, this method enables the robot to operate with a smooth and realistic motion. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application;

[0038] Figure 2 This is a schematic diagram of a robot control method provided in an exemplary embodiment of this application;

[0039] Figure 3 This is a flowchart illustrating a robot control method provided in an exemplary embodiment of this application;

[0040] Figure 4 This is a schematic diagram of a machine learning model provided in an exemplary embodiment of this application;

[0041] Figure 5 This is a flowchart illustrating an exemplary embodiment of a policy network implementation method provided in this application;

[0042] Figure 6 This is a schematic diagram of an action intelligent generation model provided in an exemplary embodiment of this application;

[0043] Figure 7This is a flowchart illustrating an implementation method of an action intelligent generation model provided in an exemplary embodiment of this application;

[0044] Figure 8 This is a flowchart illustrating a training method for an action intelligent generation model provided in an exemplary embodiment of this application;

[0045] Figure 9 This is a schematic diagram of motion capture data provided in an exemplary embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the joint motor rotation angle provided in an exemplary embodiment of this application;

[0047] Figure 11 This is a flowchart illustrating a method for training a robot control model provided in an exemplary embodiment of this application;

[0048] Figure 12 This is a schematic diagram of a robot control device provided in an exemplary embodiment of this application;

[0049] Figure 13 This is a schematic diagram of a training device for a robot control model provided in an exemplary embodiment of this application;

[0050] Figure 14 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0052] First, let's introduce the terms used in the embodiments of this application:

[0053] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0054] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0055] Fully connected feedforward networks, also known as feedforward neural networks, have neurons arranged in layers. Each neuron is connected only to neurons in the layer preceding it. It receives the output of the previous layer and outputs it to the next layer; there is no feedback between layers. Layer 0 is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. There can be one or multiple hidden layers.

[0056] Figure 1 A schematic diagram of the structure of a robot provided in an exemplary embodiment of this application is shown. The robot 100 includes: a body 120 and leg mechanical structures 140.

[0057] The robot 100 is controlled by a controller located inside the body 120. The body 120 sends commands to the leg mechanical structure 140 to control the movement of the leg mechanical structure 140.

[0058] The leg mechanical structure 140 is equipped with multiple joints, each of which can have one joint motor, or multiple joint motors can be installed. For example... Figure 1 As shown, taking one of the leg mechanical structures 140 as an example, the leg mechanical structure is provided with joint 160 and joint 180. Joint 160 is provided with one joint motor, and joint 180 is provided with two joint motors.

[0059] Figure 2 A schematic diagram of a robot control method provided in one embodiment of this application is shown.

[0060] This method requires obtaining the robot's reference state set 202. Figure 2 In the context of the reference state set 202, there is a reference video 201 for the robot. The reference state set 202 is used to provide a reference for the robot's behavior, enabling the robot to act in accordance with the reference state set.

[0061] In the actual operation of the robot, the behavior of the robot from the first cycle to the second cycle will be used as an example for explanation. Figure 2The diagram illustrates the robot 203 in the first cycle. The controller acquires the current state 204 of the robot 203 in the first cycle and inputs this state 204 into the machine learning model 205. The machine learning model 205 determines the robot's reference state for the second cycle from the reference state set 202 based on the current state 204. Using the aforementioned reference state as a reference object, the machine learning model 205 determines the target motor rotation angle 206 of the robot based on the current state 204. Subsequently, the PD controller (Proportion Differential controller) 207 converts the target motor rotation angle 206 into a torque command 208. The controller sends the torque command 208 to each motor of the robot, thereby controlling the operation of the motors and obtaining the robot 209 in the second cycle.

[0062] Figure 3 A flowchart illustrating a robot control method according to an embodiment of this application is shown. This method can be... Figure 1 The robot 100 shown executes the method, which includes the following steps:

[0063] Step 302: Obtain the robot's reference state set, which includes the robot's reference body posture and reference motor rotation angle.

[0064] The reference body attitude is used to represent the reference attitude of the robot body. In one optional implementation, the reference body attitude is represented by Euler angles, which is a vector containing three components: nutation angle, precession angle, and rotation angle.

[0065] The reference motor angle is used to represent the reference angle of a robot joint. The number of reference motor angles is related to the number of joint motors in the robot. Optionally, the robot has 12 joint motors, and the robot's movements are controlled by these 12 joint motors. For example, two joint motors are placed between the robot's body and leg mechanical structures, and one joint motor is placed on one of the leg mechanical structures. Then, the reference motor angle is a vector with 12 components.

[0066] Optionally, the reference state set is obtained based on a reference video. Alternatively, the reference state set is obtained based on reference image frames.

[0067] Step 304: Obtain the robot's current state, which includes the robot's current body posture and current joint motor rotation angle in the current cycle.

[0068] The current body pose is used to represent the robot's body pose in the current cycle. In an alternative implementation, the current body pose is represented by Euler angles, and the reference body pose is a vector containing three components.

[0069] The current joint motor angle is used to represent the angle of rotation of the robot's joint motor in the current cycle. In an alternative implementation, the current joint motor angle is a vector with 12 components, each component representing the angle of rotation of a joint motor.

[0070] The reference state set includes reference states for multiple image frames, with a one-to-one correspondence between the image frames and the reference states. For example, the reference state set includes reference states for 10 image frames. By arranging and playing these 10 reference states in the order of the image frames, a reference video of the robot can be obtained.

[0071] Step 306: Call the machine learning model to process the input reference state set and the current state, and output the target motor rotation angle of m joint motors.

[0072] Machine learning models are used to determine the target motor rotation angle of a robot based on its reference state set and current state. For example, a machine learning model can be at least one of a fully connected feedforward network, an attention-based neural network, a residual network, or a generative adversarial network.

[0073] The target motor rotation angle is used to represent the joint motor rotation angle of the robot in the next cycle of the current cycle.

[0074] Step 308: Convert the target motor rotation angle of the m joint motors into torque commands for the m joint motors. The torque commands are used to control the operation of the m joint motors.

[0075] In one optional implementation, a proportional-differential controller (PD controller) is invoked to process the target motor rotation angles of the input m joint motors and output torque commands for the m joint motors.

[0076] In this embodiment, the machine learning model outputs the target motor rotation angle instead of the torque command. When the machine learning model directly outputs the torque command, the following effects will occur: the relationship between the torque command of the m joint motors in the next cycle and the current state of the m joint motors in the current cycle is highly nonlinear and discontinuous. Therefore, if the machine learning model is used to output the torque command directly, the torque command obtained in this way is relatively dangerous in controlling the robot. This is because the small errors generated by the sensors on the robot will generate a large torque at the joint motor, affecting the normal operation of the robot and even causing damage to the robot.

[0077] Step 310: Control the robot by controlling the torque commands of m joint motors.

[0078] Torque commands are used to control the rotation of m joint motors, which in turn drive the robot to move.

[0079] In summary, this embodiment utilizes a machine learning model, taking the robot's reference state set as a reference object, to predict the current state, obtain the target motor rotation angle, and then convert the target motor rotation angle into a torque command. The robot's behavior is then controlled through this torque command. This method enables the robot to operate with a reference state set as a reference, and the robot's movements are smooth and realistic.

[0080] Figure 4 A schematic diagram of a machine learning model provided in one embodiment of this application is shown. The machine learning model includes: a state observer 401, a policy network 402, and a PD controller 403.

[0081] The state observer 401 can acquire the robot's state, which includes, but is not limited to, at least one of the robot's body posture and joint motor rotation angles. The state observer 401 acquires the current state s of robot 404 from robot 404. t The state observer 401 can also obtain the historical state s of the previous two cycles. t-1 and s t-2 State observer 401 will s t s t-1 and s t-2 Send to Policy Network 402.

[0082] Policy network 402 is used to generate the bias target rotation angle α. t The policy network 402 responds to the s from the state observer 401. t s t-1 and s t-2 , and reference state g from reference state set 405 t+1 g t+2 g t+3 and g t+4 Perform data processing and output the offset target rotation angle α. t The offset target rotation angle α t Compared with reference state g t+1 The target motor rotation angle q is obtained by performing a weighted calculation. r t .

[0083] The PD controller 403 is used to convert the joint motor rotation angle into a torque command. The PD controller 403 determines the torque command based on the target motor rotation angle q. r t The command to generate torque τ t The torque command τ tProvided to robot 404, so that robot 404 can respond to torque command τ t To carry out activities.

[0084] In the following embodiments, a specific implementation method of the machine learning model is provided. This method can output a reasonable target motor rotation angle, thereby obtaining a torque command suitable for the current state to control the operation of the robot, so that the robot's operation is smooth and realistic.

[0085] Figure 5 A flowchart illustrating an implementation method of a policy network according to an embodiment of this application is shown.

[0086] This method can be derived from Figure 1 The robot 100 shown executes the method, which includes the following steps:

[0087] Step 501: Obtain the robot's historical state in the previous j cycles.

[0088] Historical states include the robot's historical body posture and historical joint motor rotation angles. For example, to obtain the robot's historical states in the previous two cycles, let's assume the robot's current state is s. t If t represents the current period, then the historical states include s. t-1 and s t-2 Among them, historical state s t-1 Including historical fuselage attitude s t-1 And the historical joint motor rotation angle q s t-1 Historical state s t-2 Including historical fuselage attitude s t-2 And the historical joint motor rotation angle q s t-2 .

[0089] Step 502: Determine the robot's reference states for the next k cycles from the set of reference states, where k is a positive integer.

[0090] The reference state set includes reference states for multiple image frames, with a one-to-one correspondence between the image frames and the reference states.

[0091] For example, the robot's reference state g for the last four cycles is determined from the set of reference states. t+1 g t+2 g t+3 and g t+4 Among them, the reference state g t+1 Including reference fuselage attitude g t+1 and reference motor rotation angle q g t+1 Reference state gt+2 Including reference fuselage attitude g t+2 and reference motor rotation angle q g t+2 Reference state g t+3 Including reference fuselage attitude g t+3 and reference motor rotation angle q g t+3 Reference state g t+4 Including reference fuselage attitude g t+4 and reference motor rotation angle q g t+4 .

[0092] Step 503: Call the machine learning model to process the input current state, the historical state of the previous j cycles, and the reference state of the next k cycles, and output the target motor rotation angle of the m joint motors in the next cycle.

[0093] Optionally, this step includes the following sub-steps:

[0094] 1. Call the policy network to process the input current state, the historical state of the previous j cycles, and the reference state of the next k cycles, and output the bias target rotation angle of m joint motors in the next cycle.

[0095] Optionally, the policy network is a fully connected feedforward network.

[0096] The number of offset target rotation angles is the same as the number of joint motors of the robot.

[0097] Table 1. Input and output definitions of the policy network

[0098]

[0099] 2. Determine the reference motor rotation angles for the m joint motors of the robot in the next cycle from the reference state set.

[0100] For example, if a period of 40 milliseconds is used, and the current period is set to 0, then the target rotation angle obtained based on the current state corresponds to 40 milliseconds. It is necessary to determine the reference motor rotation angles of m joint motors corresponding to 40 milliseconds from the reference state set.

[0101] 3. Calculate the bias target rotation angle of m joint motors in the next cycle and the reference motor rotation angle of the m joint motors in the next cycle using weighted calculation to obtain the target motor rotation angle of the m joint motors in the next cycle.

[0102] By weighted calculation of the bias target rotation angle of m joint motors in the next cycle and the reference motor rotation angle of the corresponding m joint motors in the next cycle, the policy network can obtain a higher reward value during the model training phase, thus accelerating the training of the policy network.

[0103] For example, let a be the target offset rotation angle of m joint motors in the next cycle. t Let g be the reference motor rotation angle of the m joint motors corresponding to the next cycle. t+1 Then the target motor rotation angle q r t =0.25*a t +075*g t+1 .

[0104] In summary, the machine learning model uses the robot's reference state set as a reference object to predict the current state and obtain the target motor rotation angle. This target motor rotation angle allows the robot to operate according to the reference states in the reference state set, resulting in smooth, realistic robot movements that closely match the reference state set.

[0105] Figure 6 The diagram illustrates a motion intelligent generation model according to an embodiment of this application. The motion intelligent generation model includes a gait network 601 and a motion prediction network 602. Both the gait network and the motion language network are fully connected feedforward networks.

[0106] Gait network 601 is used to generate expert coefficients, which are used to determine the network weights of each layer in the action oracle network. The input to gait network 601 is the reference state and the desired aircraft speed in frame t, and the output is the expert coefficients. The network weights of each layer in the action oracle network 602 are determined based on the expert coefficients, and the network weights are then configured in the action language network 602.

[0107] The action oracle network 602 is used to generate a reference state set for the robot. The reference state set includes reference states from multiple image frames, which can guide the robot's activities. The inputs of the action oracle network 602 are the desired state of frame t, the desired state set of frame t, and the historical state set of frame t. The outputs are the desired state of frame t+1 and the desired state set of frame t+1.

[0108] In the following embodiments, an implementation method for an intelligent motion generation model is provided, which can iteratively output a set of reference states, and the set of reference states conforms to the activity logic of the robot.

[0109] Figure 7 This illustration shows a flowchart of an implementation method for an action intelligent generation model provided in one embodiment of this application. The method can be... Figure 1 The robot 100 shown executes the method, which includes the following steps:

[0110] Step 701: Obtain the robot's reference state for frame t, the set of desired states for frame t, and the set of historical states for frame t.

[0111] The desired state in frame t includes the positions and velocities of n joints in the t-th image frame in the t-th fuselage coordinate system. The t-th fuselage coordinate system is the coordinate system corresponding to the fuselage in the t-th image frame.

[0112] The t-th desired state set includes the desired position and desired attitude of the fuselage in the t-th fuselage coordinate system in the i-th frames following the t-th image frame.

[0113] The t-th historical state set includes the historical position and attitude of the fuselage in the t-th fuselage coordinate system in the i-th frames preceding the t-th image frame.

[0114] Step 702: Call the gait network to process the input expected state of frame t and the set of expected states of frame t, and output expert coefficients.

[0115] The expert coefficient is a weighted coefficient of the network weights in an action oracle network, and it consists of 'a' components.

[0116] The gait network's input includes the reference state of frame t and the desired fuselage velocity, where the desired fuselage velocity represents the desired fuselage velocity in the i frames following the t-th image frame. Optionally, the desired fuselage velocity is extracted from the set of desired states for the t-th frame. The gait network is then invoked to process the input reference state of frame t and the desired fuselage velocity, outputting expert coefficients.

[0117] Step 703: Determine the network weights of the action prediction network based on the expert coefficients.

[0118] Optionally, obtain *a* sets of candidate network weights for the action oracle network; use expert coefficients as weights to calculate the weights of *a* sets of candidate networks, thus obtaining the network weights of the action oracle network. For example, let *a* = 4, and denote the four sets of candidate network weights as K1, K2, K3, and K4 respectively. The resulting expert coefficients are [A1, A2, A3, A4], where A1 + A2 + A3 + A4 = 1. Therefore, the network weights of the action oracle network are A1*K1 + A2*K2 + A3*K3 + A4*K4. It should be noted that the expert coefficients and candidate network weights can be combined arbitrarily; for example, the network weights of the action oracle network could also be A1*K4 + A2*K3 + A3*K2 + A4*K1.

[0119] Step 704: Call the action oracle network to process the input expected state of frame t, the expected state set of t, and the historical state set of t, and output the reference state of frame t+1 in the reference state set.

[0120] The action oracle network outputs not only the reference state of frame t+1 in the reference state set, but also the expected state set of frame t+1. The expected state set of frame t+1 includes the expected position and expected pose of the aircraft in the t-th fuselage coordinate system of the i-th frame following frame t+1. For example, the action oracle network is invoked to process the input expected state of frame t, the expected state set of frame t, and the historical state set of frame t, and outputs the expected state set of frame t+1.

[0121] On the other hand, the input to the gait network is a subset of the input to the action prediction network.

[0122] For example, the input and output of the action intelligent generation network are defined as follows:

[0123] Table 2 Input and Output Definitions of the Action Intelligent Generation Network

[0124]

[0125] Step 705: Start incrementing t from 1 and repeat the above three steps. In response to the fulfillment of the output completion condition, output the reference state set.

[0126] According to Table 2, the input and output of the motion intelligent generation model form a loop. For example, (1) in the first image frame, the input of the motion intelligent generation model is the desired state of the first frame, the first desired state set, and the first historical state set, and the output is the desired state of the second frame and the second desired state set; (2) in the second image frame, the input of the motion intelligent generation model is the desired state of the second frame, the second desired state set, and the second historical state set, and the output is the desired state of the third frame and the third desired state set. Among them, the motion of the second image frame... The second frame expected state and the second expected state set in the input of the intelligent generation model can be obtained from the output of the action intelligent generation model in the first image frame; (3) In the third image frame, the input of the action intelligent generation model is the third frame expected state, the third expected state set, and the third historical state set, and the output is the fourth frame expected state and the fourth expected state set. Among them, the third frame expected state and the third expected state set in the input of the action intelligent generation model in the third image frame can be obtained from the output of the action intelligent generation model in the second image frame. And so on, which will not be repeated hereafter.

[0127] The output completion condition can be set by the technician. For example, the output completion condition is that the number of reference states in the reference state set reaches a preset number. Alternatively, the output completion condition is that the reference states in the reference state set enable the robot to complete a preset action.

[0128] In one practical embodiment of this application, the cumulative step size is 1, that is, t starts from 1 and increments by 1 each time. Therefore, the action intelligent generation model only needs to input the desired state of the first frame and the first set of desired states to output a complete set of reference states. Those skilled in the art can set the desired set of states as needed, therefore, the embodiments of this application can obtain a relatively rich set of reference states.

[0129] In summary, this embodiment provides a method for generating a reference state set. This method uses an intelligent motion generation network to generate the reference state set, ensuring that the generated reference state set conforms to the robot's behavior. Furthermore, it defines the robot's reference state for frame t, enabling the robot to successfully complete its start-up.

[0130] Figure 8 The diagram illustrates a flowchart of a training method for an action intelligent generation model according to an embodiment of this application. This method can be... Figure 1 The computer device shown executes the method, which includes the following steps:

[0131] Step 801: Obtain the robot's reference state for frame t, the expected state set for the t-th sample, the historical state set for the t-th sample, and the sample training set.

[0132] The expected state of the t-th sample includes the position and velocity of n joints in the t-th sample image frame in the t-th sample body coordinate system, where the t-th sample body coordinate system is the coordinate system corresponding to the body of the t-th sample image frame.

[0133] The expected state set of the t-th sample includes the expected position and expected attitude of the fuselage in the fuselage coordinate system of the t-th sample in the i-th frames following the t-th sample image frame.

[0134] The historical state set of the t-th sample includes the expected position and expected attitude of the fuselage in the fuselage coordinate system of the t-th sample in the previous i frames of the t-th sample image frame.

[0135] The training set includes the true positions and true velocities of n joints. In an optional implementation, the training set is obtained using electronic motion capture data. Optionally, the robot's electronic motion capture data is acquired; the kinematic equations are invoked, and the training set is obtained based on the electronic motion capture data. In a practical example, the electronic motion capture data includes motion capture data from multiple image frames. Taking one image frame from the electronic motion capture data as an example, such as... Figure 9 As shown, the three-dimensional positions of the four shoulders / hips (circles filled with diagonal lines in the figure) and the tips of the four toes (circles without filling in the figure) of the organism in space are calculated. A plane is linearly regressed from the four shoulder / hip positions, corresponding to the position and posture of the robot's body. Figure 10 As shown, if a robot has four legs, each with three joint motors, then, given the position and orientation of the robot's body in space, and the three-dimensional position of one leg's mechanical structure in space, the following system of equations can be used to obtain the joint motor rotation angle:

[0136]

[0137] Where θ1, θ2, and θ3 are the joint motor rotation angles of the three joint motors, and x, y, and z are the three-dimensional local coordinates of the leg in the fuselage coordinate system. The local coordinates of the first joint motor in the fuselage coordinate system are (L1, L2, 0), the local coordinates of the second joint motor in the coordinate system corresponding to the first joint motor are (0, L3, 0), the local coordinates of the third joint motor in the coordinate system corresponding to the second joint motor are (0, L4, 0), and the local coordinates of the end of the leg mechanical structure in the coordinate system corresponding to the third joint motor are (0, L5, 0).

[0138] Step 802: Call the action intelligent generation model to process the input expected state of the t-th frame sample, the expected state set of the t-th sample, and the historical state set of the t-th sample, and output the sample reference state set frame by frame.

[0139] The motion intelligent generation model consists of a gait network and a motion oracle network. The gait network generates expert coefficients, which determine the network weights of each layer in the motion oracle network. The motion oracle network generates a sample reference state set for the robot, which includes sample reference states from multiple image frames.

[0140] Step 803: Train the action intelligent generation model based on the difference between the sample training set and the sample reference state set.

[0141] Optionally, the action intelligent generation model is trained using gradient descent supervised learning based on the difference between the sample training set and the sample reference state set.

[0142] In summary, this embodiment provides a training method for an intelligent motion generation model. This intelligent motion generation model can be trained using electronic motion capture data. The trained intelligent motion generation model can generate a good set of reference states, allowing the robot to perform activities with reference to the set of reference states. Moreover, the robot's operation is smooth and realistic.

[0143] Figure 11 A flowchart illustrating a method for training a robot control model according to an embodiment of this application is shown. This method can be executed by a computer device and includes the following steps:

[0144] Step 1101: Obtain the robot's sample reference state set.

[0145] The sample reference states in the sample reference state set include the robot's sample reference body posture and sample reference motor rotation angle.

[0146] Step 1102: Obtain the current state of the robot's sample.

[0147] The current state of the sample includes the robot's current body posture and the current joint motor rotation angle.

[0148] Step 1103: Call the robot control model to process the input sample reference state set and sample current state, and output the sample target motor rotation angles of m joint motors.

[0149] The motion intelligent generation model includes a policy network. The policy network is used to generate sample bias target rotation angles. After weighting the sample bias target rotation angles with the sample reference state set, the sample joint motor rotation angles of m joint motors are obtained.

[0150] Step 1104: Calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample target motor rotation angle of the m joint motors.

[0151] For example, the loss function is called to process the rotation angles of the sample reference motors and the rotation angles of the sample target motors corresponding to the m joint motors in the sample reference state set, and the difference is obtained.

[0152] Step 1105: Calculate the reward value corresponding to the sample target rotation angle of m joint motors.

[0153] In one alternative implementation, the reward value r = 0.7·exp((q s t -o g t ) 2 )+0.2·exp((p s t -p g t ) 2 )+0.1·exp((o s t –o g t+1 ) 2 ), where p st With p g t These represent the actual position of the robot's body and the position of the sample target state set, respectively. s t This indicates the joint motor rotation angle corresponding to the current state of the sample. g t This represents the fuselage attitude corresponding to the current state of a sample in the sample reference state set. s t This indicates the fuselage attitude corresponding to the current state of the sample. g t+1 This represents the fuselage attitude corresponding to the next cycle of the sample's current state. Furthermore, p is used when calculating the reward value. s t With p g t However, p is not defined in the input definition of the policy network. s t With p g t This is because obtaining accurate fuselage attitude is costly and difficult, making it unsuitable for using p s t With p g t It is used as input to the policy network.

[0154] Step 1106: Train the robot control model based on the difference, with the goal of maximizing the reward value.

[0155] Optionally, the output layer weights of the policy network should be initialized with a small standard deviation. For example, if the reference signal unit of the PD controller is radians, then the standard deviation of the policy network output layer weights should be set to std = 0.01, in which case the initial output distribution of the policy network will be around 0. Conversely, if std is too large (e.g., std = 1.0), it may cause the robot to produce violent shaking movements during or even after training, affecting the normal training of the robot.

[0156] Optionally, after training the robot control model in the Pybullet simulator, it will be sequentially transferred to the Gazebo simulator and then to a real robot. The Pybullet simulator generally uses a Python interface, while the real robot and the Gazebo simulator provide C++ interfaces. Rewriting the machine learning model written in Python in the Pybullet simulator using C++ in the Gazebo simulator allows for timely detection of errors, preventing damage to the real robot. Furthermore, the control interfaces of the Gazebo simulator and the real robot share the same calling method, making subsequent transfer very convenient. On the other hand, various errors inevitably exist between the simulator and the real robot, ultimately leading to unsatisfactory policy transfer performance. While the Pybullet simulator and the Gazebo simulator are two different simulators, their dynamic simulation ODE (Ordinary Differential Equation) and collision models differ slightly. From a data-driven perspective, a robust policy that can adapt to the differences introduced by transferring between different simulators will also contribute to the robustness of the transfer from the simulator to the real robot.

[0157] In summary, this embodiment provides a method for training a robot control model. This method can train the robot control model with high efficiency. When the trained robot control model controls the robot, the robot can perform activities with reference to a set of reference states, and the robot's operation is smooth and realistic.

[0158] Optionally, the robot control method provided in this application embodiment may further include the following steps:

[0159] The acquired motion capture data is processed into FBX format and sent to the computer device. The computer device then uses Maya software to process the FBX format into BVH format.

[0160] Because motion capture data has a high degree of freedom, it simulates approximately 40 joints in a real organism. However, in this embodiment, the motion capture data is used for robot gait planning, so the joints of the motion capture data need to be mapped onto the 12 motor joints of the quadruped robot. After joint mapping, each frame of the motion capture data is a vector containing 19 components, of which 3 represent the position coordinates of the robot body, 4 represent the posture quaternions of the robot body, and 12 represent the rotation angles of the 12 motors.

[0161] Using a large amount of motion capture data after joint mapping, a motion generation algorithm is employed for supervised learning training to obtain a motion intelligent generation network. New motion fragments with landing point rules generated by this intelligent network are then used as expert data for deep reinforcement learning training based on the deep mimic algorithm in a PyBulet simulator. This process ultimately yields a machine learning model. It accepts the robot's current state and a set of reference states as input, and sends the generated torque commands as control signals to 12 motors.

[0162] After the machine learning model is trained in the PyBulet simulator, it is sequentially transferred to the Gazebo simulator and then to the real robot to control the execution of reference action segments. There is a transfer step between the PyBulet simulator and the real robot, back to the Gazebo simulator, for two reasons:

[0163] The first issue is the program interface. The Pybullet simulator uses a Python language interface, which is beneficial for calling deep reinforcement learning algorithm libraries that also use Python as their interface. However, the real robot and Gazebo both provide C++ interfaces, communicating through the Robot Operating System. Rewriting a machine learning model written in Python in Pybullet within Gazebo using C++ allows errors to be detected immediately in the Gazebo simulator, preventing damage to the real robot. Furthermore, the control interfaces of the Gazebo simulator and the real robot share the same calling method, making subsequent migrations very convenient.

[0164] The second reason is that various errors inevitably exist between the simulator and the real robot, ultimately leading to unsatisfactory policy transfer performance. While the PyBulllet and Gazebo simulators are different, they employ slightly different dynamic simulation ordinary differential equations and collision models. From a data-driven perspective, a robust policy that can adapt to the differences introduced by transferring between different simulators will also contribute to its robustness when transferring from the simulator to the real robot.

[0165] During the migration from the PyBulllet simulator to the Gazebo simulator and finally to a real robot, due to the differences between the simulator and the real world, some parameter randomization techniques can be used to enhance the robustness of the policy network.

[0166] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the corresponding descriptions in the above method embodiments. They will not be repeated here.

[0167] Figure 12 A schematic diagram of a robot control device provided in an exemplary embodiment of this application is shown. This device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 1200 includes:

[0168] The acquisition module 1201 is used to acquire a set of reference states of the robot, wherein the reference states in the set of reference states include the robot's reference body posture and reference motor rotation angle;

[0169] The acquisition module 1201 is also used to acquire the current state of the robot, which includes the robot's current body posture and current motor rotation angle in the current cycle;

[0170] Control module 1202 is used to call a machine learning model to process the input reference state set and the current state, and output the target motor rotation angle of the m joint motors;

[0171] The control module 1202 is further configured to convert the target motor rotation angle of the m joint motors into torque commands for the m joint motors, and the torque commands are used to control the operation of the m joint motors;

[0172] The control module 1202 is also used to control the robot through torque commands of the m joint motors.

[0173] In an optional design of this application, the acquisition module 1201 is further configured to acquire the historical state of the robot in the previous j cycles, the historical state including the robot's historical body posture and historical joint motor rotation angle, where j is a positive integer; determine the reference state of the robot in the next k cycles from the reference state set, where k is a positive integer; the control module 1202 is further configured to call the machine learning model to process the input current state, the historical state of the previous j cycles and the reference state of the next k cycles, and output the target motor rotation angle of the m joint motors in the next cycle.

[0174] In an optional design of this application, the control module 1202 is further configured to call the machine learning model to process the input current state, the historical states of the previous j cycles, and the reference states of the next k cycles, and output the target motor rotation angles of the m joint motors in the next cycle, including: calling the policy network to process the input current state, the historical states of the previous j cycles, and the reference states of the next k cycles, and outputting the bias target rotation angles of the m joint motors in the next cycle; determining the reference motor rotation angles of the m joint motors corresponding to the robot in the next cycle from the reference state set; and weightedly calculating the bias target rotation angles of the m joint motors in the next cycle and the reference motor rotation angles of the m joint motors corresponding to the next cycle to obtain the target motor rotation angles of the m joint motors in the next cycle.

[0175] In one optional design of this application, the reference state set includes reference states of multiple image frames. The acquisition module 1201 is further configured to acquire the t-th frame reference state, the t-th desired state set, and the t-th historical state set of the robot. The t-th frame desired state includes the position and velocity of the n joints in the t-th image frame in the t-th body coordinate system. The t-th desired state set includes the desired position and desired posture of the robot in the t-th body coordinate system in the i-th subsequent frames of the t-th image frame. The t-th historical state set includes the historical position and historical posture of the robot in the t-th body coordinate system in the i-th preceding frames of the t-th image frame. The t-th body coordinate system is the coordinate system corresponding to the robot in the t-th image frame, where i is a positive integer. The control module 1202 is further configured to call the motion intelligent generation model to process the input t-th frame desired state, the t-th desired state set, and the t-th historical state set, and output the reference state set frame by frame.

[0176] In one optional design of this application, the intelligent motion generation model includes a gait network and a motion oracle network. The control module 1202 is further configured to: invoke the gait network to process the input expected state of the t-th frame and the t-th expected state set, and output expert coefficients; determine the network weights of the motion oracle network based on the expert coefficients; invoke the motion oracle network to process the input expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and output the reference state of the (t+1)-th frame in the reference state set; increment t from 1 and repeat the above three steps; and output the reference state set in response to the fulfillment of the output completion condition.

[0177] In an optional design of this application, the acquisition module 1201 is further configured to extract the fuselage expected speed from the t-th expected state set, wherein the fuselage expected speed is used to represent the fuselage expected speed of the i-th frame following the t-th image frame; the control module 1202 is further configured to call the gait network to perform data processing on the input t-th frame reference state and the fuselage expected speed, and output the expert coefficients.

[0178] In an optional design of this application, the control module 1202 is further configured to call the action prediction network to process the input expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and output the (t+1)-th expected state set, wherein the (t+1)-th expected state set includes the expected position and expected attitude of the rack in the t-th fuselage coordinate system of the i-th frame following the (t+1)-th image frame.

[0179] In an optional design of this application, the acquisition module 1201 is further configured to acquire a set of candidate network weights of the action oracle network; the control module 1202 is further configured to use the expert coefficients as weights to calculate the weights of the a set of candidate networks in a weighted manner, thereby obtaining the network weights of the action oracle network.

[0180] In an optional design of this application, the acquisition module 1201 is further configured to acquire the robot's frame t-th sample reference state, the set of desired states for the t-th sample, the set of historical states for the t-th sample, and the sample training set. The desired state for the t-th sample includes the positions and velocities of the n joints in the t-th sample image frame in the t-th sample body coordinate system. The set of desired states for the t-th sample includes the desired positions and desired attitudes of the robot's body in the t-th sample body coordinate system in the i-th frames following the t-th sample image frame. The set of historical states for the t-th sample includes the positions and velocities of the robot's body in the i-th frames preceding the t-th sample image frame. The desired position and desired posture in the fuselage coordinate system of the t-th sample, the sample training set includes the true positions and true velocities of the n joints, and the fuselage coordinate system of the t-th sample is the coordinate system corresponding to the fuselage of the t-th sample image frame; the control module 1202 is also used to call the motion intelligent generation model to process the input desired state of the t-th frame sample, the desired state set of the t-th sample, and the historical state set of the t-th sample, and output the sample reference state set frame by frame; and train the motion intelligent generation model according to the difference between the sample training set and the sample reference state set.

[0181] In one optional design of this application, the acquisition module 1201 is further configured to acquire the electronic motion capture data of the robot; the control module 1202 is further configured to invoke the kinematic equations and obtain the sample training set based on the electronic motion capture data.

[0182] In an optional design of this application, the control module 1202 is further configured to call the proportional-derivative controller to process the input target motor rotation angle of the m joint motors and output the torque command of the m joint motors.

[0183] In summary, this embodiment utilizes a machine learning model to process the robot's reference state set and current state to obtain the target motor rotation angle. This target motor rotation angle is then converted into a torque command, which controls the robot's behavior. This method enables the robot to operate with a reference state set as a reference, and the robot's movements are smooth and realistic.

[0184] Figure 13 A schematic diagram of a training device for a robot control model provided in an exemplary embodiment of this application is shown. This device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 1300 includes:

[0185] The sample acquisition module 1301 is used to acquire the sample reference state set of the robot, wherein the sample reference state in the sample reference state set includes the sample reference body posture and the sample reference motor rotation angle of the robot.

[0186] The sample acquisition module 1301 is also used to acquire the current state of the sample of the robot, the current state of the sample includes the current body posture of the robot and the current motor rotation angle of the sample;

[0187] The sample determination module 1302 is used to call the robot control model, use the sample reference state set as a reference object, predict the current state of the sample, and output the sample target motor rotation angle of the m joint motors;

[0188] The sample calculation module 1303 is used to calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample target motor rotation angle of the m joint motors; and to calculate the reward value corresponding to the sample reference motor rotation angle of the m joint motors.

[0189] The sample training module 1304 is used to train the robot control model based on the difference with the goal of maximizing the reward value.

[0190] In summary, this embodiment provides a training device for a robot control model. This device can train the robot control model with high efficiency. When the trained robot control model controls the robot, the robot can perform activities with reference to a set of reference states, and the robot's operation is smooth and realistic.

[0191] Figure 14 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including Random Access Memory (RAM) 1402 and Read-Only Memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 also includes a basic input / output system (I / O system) 1406 to facilitate information transfer between various devices within the computer device, and a mass storage device 1407 for storing an operating system 1413, application programs 1414, and other program modules 1415.

[0192] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0193] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer device-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0194] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0195] According to various embodiments of this disclosure, the computer device 1400 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via a network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer device systems (not shown).

[0196] The memory also includes one or more programs stored in the memory. The central processing unit 1401 executes the one or more programs to implement all or part of the steps of the robot control method or the robot control model training method.

[0197] In an exemplary embodiment, a robot is also provided, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the robot control method provided in the above method embodiments, or the robot control model training method.

[0198] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the robot control method provided in the above method embodiments, or the robot control model training method.

[0199] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robot control method or the robot control model training method provided in the above embodiments.

[0200] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0201] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0202] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling a robot, characterized in that, The robot includes a body and m joint motors, where m is a positive integer greater than 1. The method includes: Obtain a reference state set for the robot, wherein the reference states in the reference state set include the robot's reference body posture and reference motor rotation angle; Obtain the current state of the robot, which includes the robot's current body posture and current joint motor rotation angle in the current cycle; The machine learning model is invoked to process the input reference state set and the current state, and the target motor rotation angle of the m joint motors is output. The target motor rotation angle of the m joint motors is converted into a torque command for the m joint motors, and the torque command is used to control the operation of the m joint motors; The robot is controlled by torque commands from the m joint motors.

2. The method according to claim 1, characterized in that, The step of calling a machine learning model to process the input reference state set and the current state, and outputting the target motor rotation angles of the m joint motors, includes: Obtain the historical state of the robot in the previous j cycles. The historical state includes the robot's historical body posture and historical joint motor rotation angle, where j is a positive integer. The reference states of the robot for the next k cycles are determined from the set of reference states, where k is a positive integer; The machine learning model is invoked to process the input current state, the historical states of the previous j cycles, and the reference states of the next k cycles, and outputs the target motor rotation angle of the m joint motors in the next cycle.

3. The method according to claim 2, characterized in that, The machine learning model includes a policy network; The process of calling the machine learning model to process the input current state, the historical states of the previous j cycles, and the reference states of the next k cycles, and outputting the target motor rotation angle of the m joint motors in the next cycle, includes: The policy network is invoked to process the input current state, the historical state of the previous j cycles, and the reference state of the next k cycles, and outputs the bias target rotation angle of the m joint motors in the next cycle. Determine the reference motor rotation angles of the m joint motors of the robot in the next cycle from the reference state set; The target motor angle of the m joint motors in the next cycle is obtained by weighted calculation of the bias target rotation angle of the m joint motors in the next cycle and the reference motor rotation angle of the m joint motors in the next cycle.

4. The method according to any one of claims 1 to 3, characterized in that, The reference state set includes reference states based on multiple image frames; The process of obtaining the reference state set of the robot includes: Obtain the robot's desired state in frame t, the set of desired states in frame t, and the set of historical states in frame t. The desired state in frame t includes the position and velocity of n joints in frame t in the t-th body coordinate system. The set of desired states in frame t includes the desired position and desired posture of the robot in the t-th body coordinate system in the i subsequent frames of frame t. The set of historical states in frame t includes the historical position and historical posture of the robot in the t-th body coordinate system in the i preceding frames of frame t. The t-th body coordinate system is the coordinate system corresponding to the robot in frame t, where t and i are positive integers. The action intelligent generation model is invoked to process the input expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and outputs the reference state set frame by frame.

5. The method according to claim 4, characterized in that, The intelligent motion generation model includes a gait network and a motion prediction network; The intelligent generation model for invoking actions processes the input expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and outputs the reference state set frame by frame, including: The gait network is invoked to process the input expected state of the t-th frame and the set of expected states of the t-th frame, and to output expert coefficients. The network weights of the action prediction network are determined based on the expert coefficients. The action oracle network is invoked to process the expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and outputs the reference state of the (t+1)-th frame in the reference state set; Starting from 1, increment t and repeat the above three steps. In response to the fulfillment of the output completion condition, output the reference state set.

6. The method according to claim 5, characterized in that, The step of calling the gait network to process the input expected state of the t-th frame and the set of expected states of the t-th frame, and outputting expert coefficients, includes: Extract the fuselage expected speed from the t-th expected state set, whereby the fuselage expected speed is used to represent the fuselage expected speed in the i-th frame following the t-th image frame; The gait network is invoked to process the reference state of the t-th frame and the desired speed of the fuselage, and the expert coefficients are output.

7. The method according to claim 5, characterized in that, The method further includes: The action prediction network is invoked to process the expected state of the t-th frame, the t-th expected state set, and the t-th historical state set, and outputs the (t+1)-th expected state set. The (t+1)-th expected state set includes the expected position and expected attitude of the frame in the t-th fuselage coordinate system of the i-th frame following the (t+1)-th image frame.

8. The method according to claim 5, characterized in that, Determining the network weights of the action prediction network based on the expert coefficients includes: Obtain a sets of candidate network weights for the action prediction network, where a is a positive integer; Using the expert coefficients as weights, the weights of the a sets of candidate networks are calculated in a weighted manner to obtain the network weights of the action prediction network.

9. The method according to claim 4, characterized in that, The intelligent action generation model is trained through the following steps: The robot's expected state for frame t, expected state set for the tth sample, historical state set for the tth sample, and training set are obtained. The expected state for frame t includes the position and velocity of the n joints in the tth sample body coordinate system. The expected state set for the tth sample includes the expected position and expected posture of the robot in the tth sample body coordinate system in the i subsequent frames of the tth sample image frame. The historical state set for the tth sample includes the expected position and expected posture of the robot in the tth sample body coordinate system in the i preceding frames of the tth sample image frame. The training set includes the actual position and actual velocity of the n joints. The tth sample body coordinate system is the coordinate system corresponding to the robot in the tth sample image frame. The action intelligent generation model is invoked to process the input expected state of the t-th frame sample, the expected state set of the t-th sample, and the historical state set of the t-th sample, and outputs the sample reference state set frame by frame. The action intelligent generation model is trained based on the difference between the sample training set and the sample reference state set.

10. The method according to claim 9, characterized in that, The process of obtaining the robot's sample training set includes: Acquire the electronic motion capture data of the robot; The kinematic equations are invoked, and the sample training set is obtained based on the electronic motion capture data.

11. The method according to any one of claims 1 to 3, characterized in that, The step of converting the target motor rotation angle of the m joint motors into the torque command of the m joint motors includes: The proportional-derivative controller is invoked to process the target motor rotation angles of the m joint motors and output the torque commands of the m joint motors.

12. A method for training a robot control model, characterized in that, The robot includes a body and m joint motors, where m is a positive integer greater than 1. The method includes: Obtain a sample reference state set for the robot, wherein the sample reference states in the sample reference state set include the sample reference body posture and sample reference motor rotation angle of the robot; Obtain the current state of the robot sample, which includes the current body posture and the current joint motor rotation angle of the robot sample; The robot control model is invoked to process the input sample reference state set and the sample current state, and outputs the sample target motor rotation angles of the m joint motors; Calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample target motor rotation angle of the m joint motors; Calculate the reward value corresponding to the rotation angle of the sample reference motor for the m joint motors; The robot control model is trained based on the difference, with the goal of maximizing the reward value.

13. A control device for a robot, characterized in that, The robot includes a body and m joints, which are controlled by m joint motors, where m is a positive integer greater than 1. The device includes: The acquisition module is used to acquire a set of reference states of the robot, wherein the reference states in the set of reference states include the robot's reference body posture and reference motor rotation angle; The acquisition module is further configured to acquire the current state of the robot, which includes the robot's current body posture and current motor rotation angle in the current cycle; The control module is used to call a machine learning model to process the input reference state set and the current state, and output the target motor rotation angle of the m joint motors; The control module is also used to convert the target motor rotation angle of the m joint motors into torque commands for the m joint motors, and the torque commands are used to control the operation of the m joint motors; The control module is also used to control the robot through torque commands of the m joint motors.

14. A training device for a robot control model, characterized in that, The robot includes a body and m joints, which are controlled by m joint motors, where m is a positive integer greater than 1. The device includes: The sample acquisition module is used to acquire a sample reference state set of the robot, wherein the sample reference state in the sample reference state set includes the sample reference body posture and the sample reference motor rotation angle of the robot; The sample acquisition module is also used to acquire the current sample state of the robot, which includes the current body posture of the robot and the current motor rotation angle of the robot. The sample determination module is used to call the robot control model to process the input sample reference state set and the sample current state, and output the sample target motor rotation angle of the m joint motors; The sample calculation module is used to calculate the difference between the sample reference motor rotation angle corresponding to the m joint motors in the sample reference state set and the sample target motor rotation angle of the m joint motors; and to calculate the reward value corresponding to the sample reference motor rotation angle of the m joint motors. A sample training module is used to train the robot control model based on the difference with the goal of maximizing the reward value.

15. A robot, characterized in that, The robot includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the robot control method as described in any one of claims 1 to 11, or the robot control model training method as described in claim 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the robot control method as described in any one of claims 1 to 11, or the robot control model training method as described in claim 12.

17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the robot control method of any one of claims 1 to 11, or the robot control model training method as described in claim 12.

Images