Robot control method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a robot control method and device, electronic equipment, a computer readable storage medium and a computer program product, which are applied to various robot control scenes. The robot control method comprises the steps that domain randomization is conducted on the current motion state of a simulation robot based on the motion difference between the simulation robot and a real robot, the motion state to be processed is obtained, and the simulation robot is a simulation result of the real robot; performing action estimation on the to-be-processed motion state by adopting the to-be-trained control model to obtain a next motion action; controlling the simulation robot to move based on the next motion action to obtain a next motion state; and training the to-be-trained control model based on the action pre-estimation reward of the next motion state to obtain a robot control model which is used for controlling the motion of a real robot. According to the invention, the control accuracy of the robot can be improved.

Description

Technical Field

[0001] This application relates to robot control technology in the field of artificial intelligence, and more particularly to a robot control method, device, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] A robot is an intelligent machine capable of semi-automatic or fully automatic operation. Robots perform tasks through programming and automatic control. In practical applications, robots can perform various complex and repetitive tasks, improving production efficiency, reducing costs, and ensuring operational safety. The ability of a robot to perform tasks is achieved by controlling its movement.

[0003] Generally, to control robot movement, a robot control model is trained using a simulated robot, and then deployed onto a real robot to control its movement. However, in the above process, the robot control model is trained directly on the simulated robot, which affects the accuracy of the robot control model and consequently the accuracy of robot control. Summary of the Invention

[0004] This application provides a robot control method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the control accuracy of robots.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a robot control method, the method comprising:

[0007] Based on the motion differences between the simulated robot and the real robot, the current motion state of the simulated robot is randomized in the domain to obtain the motion state to be processed. The simulated robot is the simulation result of the real robot.

[0008] The next motion action is obtained by using a control model to be trained to predict the motion state to be processed.

[0009] Based on the next motion action, the simulated robot is controlled to move, and the next motion state is obtained;

[0010] The control model to be trained is trained based on the action prediction reward of the next motion state to obtain a robot control model, which is used to control the motion of the real robot.

[0011] This application provides a robot control device, the robot control device comprising:

[0012] The domain randomization module is used to perform domain randomization on the current motion state of the simulated robot based on the motion difference between the simulated robot and the real robot, so as to obtain the motion state to be processed. The simulated robot is the simulation result of the real robot.

[0013] The motion prediction module is used to predict the motion state to be processed using the control model to be trained, so as to obtain the next motion action.

[0014] The motion control module is used to control the simulated robot to move based on the next motion action, and to obtain the next motion state;

[0015] The model training module is used to train the control model to be trained based on the action prediction reward of the next motion state to obtain a robot control model, which is used to control the motion of the real robot.

[0016] In this embodiment, the robot control device further includes a difference determination module, used to optimize the initial servo parameters based on motion samples to obtain target servo parameters; calculate the first simulated motion data of the simulated robot based on the target servo parameters; calculate the second real motion data of the real robot based on the target servo parameters; and fit the difference between the first simulated motion data and the first real motion data to obtain the motion difference between the simulated robot and the real robot.

[0017] In this embodiment of the application, the difference determination module is further configured to control the real robot to move using the motion sample and the initial servo parameters to obtain second real motion data; control the simulated robot to move using the motion sample and the initial servo parameters to obtain second simulated motion data; and optimize the initial servo parameters by combining the difference between the second real motion data and the second simulated motion data to obtain the target servo parameters.

[0018] In this embodiment, the second real motion data includes real joint angles, real joint velocities, real base postures, and real base angular velocities; the second simulated motion data includes simulated joint angles, simulated joint velocities, simulated base postures, and simulated base angular velocities. The difference determination module is further configured to: acquire the joint angle difference between the real joint angles and the simulated joint angles; acquire the joint velocity difference between the real joint velocities and the simulated joint velocities; acquire the base posture difference between the real base posture and the simulated base posture; acquire the base angular velocity difference between the real base angular velocity and the simulated base angular velocity; and, by combining the joint angle difference, the joint velocity difference, the base posture difference, and the base angular velocity difference, determine the difference between the second real motion data and the second simulated motion data.

[0019] In this embodiment, the difference determination module is further configured to: determine the motor drive direction by combining the target joint angle, the current joint angle, and the current joint angular velocity; determine the braking torque based on the total torque and the motor drive efficiency adapted to the motor drive direction; determine the output torque expression of the output torque based on the first mapping relationship between the braking torque, the output torque, and the total torque, and the second mapping relationship between the total torque and the motion drive information; and determine the servo parameters to be optimized based on the output torque expression, wherein the initial servo parameters are the initialized servo parameters to be optimized.

[0020] In this embodiment of the application, the difference determination module is further configured to obtain the joint angle difference between the current joint angle and the target joint angle; when the first numerical sign of the joint angle difference is the same as the second numerical sign of the current joint angular velocity, the forward driving direction is determined as the motor driving direction; when the first numerical sign of the joint angle difference is different from the second numerical sign of the current joint angular velocity, the reverse driving direction is determined as the motor driving direction.

[0021] In this embodiment of the application, the difference determination module is further used to obtain friction loss, which includes at least one of viscous friction, Coulomb friction, hysteresis loss, eddy current loss and wind resistance loss; and to combine the friction loss, the braking torque and the output torque into the total torque to obtain the first mapping relationship.

[0022] In this embodiment of the application, the difference determination module is further configured to map the joint angle difference between the current joint angle and the target joint angle to the motor applied voltage to obtain a third mapping relationship; obtain a fourth mapping relationship between the motor applied voltage, back electromotive force voltage, motor current and motor resistance; obtain a fifth mapping relationship between the total torque and the motor current; and obtain a second mapping relationship between the total torque and the motion drive information based on the third mapping relationship, the fourth mapping relationship and the fifth mapping relationship.

[0023] In this embodiment of the application, the motion prediction reward includes at least one of motion command reward, gait imitation reward, disturbance resistance reward, and regularization reward; wherein, the motion command reward represents the reward for the simulated robot responding to motion commands, the gait imitation reward represents the reward for the simulated robot's gait, and the disturbance resistance reward represents the reward for the simulated robot's disturbance resistance.

[0024] In this embodiment of the application, when the motion prediction reward includes a motion command reward, the model training module is further configured to obtain the directional velocity reward, yaw rate reward, uprightness reward, and velocity matching reward corresponding to the next motion state, wherein the velocity matching reward refers to the linear velocity and angular velocity matching reward of the next motion state; and to obtain the motion command reward by combining the directional velocity reward, the yaw rate reward, the uprightness reward, and the velocity matching reward.

[0025] In this embodiment of the application, when the motion prediction reward includes gait imitation reward, the model training module is further configured to obtain the joint angle reward of the next joint angle and the reference joint angle of the next motion state; obtain the ground contact reward of the next motion state; obtain the height matching reward of the base link height and the reference height of the next motion state; and combine the joint angle reward, the ground contact reward and the height matching reward to obtain the gait imitation reward.

[0026] In this embodiment of the application, when the motion prediction reward includes an anti-disturbance reward, the model training module is further configured to obtain the ground contact force of the next motion state; when the ground contact force is greater than the ground contact force threshold, determine the sliding penalty based on the foot linear velocity of the next motion state; and obtain the anti-disturbance reward that is negatively correlated with the sliding penalty.

[0027] In this embodiment of the application, when the action prediction reward includes a regularization reward, the model training module is further configured to obtain the energy consumption penalty, ground contact force penalty, action difference penalty, joint angular velocity penalty, and angular acceleration penalty for the next motion state; and to obtain the regularization reward by combining the energy consumption penalty, the ground contact force penalty, the action difference penalty, the joint angular velocity penalty, and the angular acceleration penalty.

[0028] In this embodiment, the robot control device further includes a terrain construction module, used to determine the terrain parameters to be trained based on the size of the simulated robot; and to construct the terrain to be trained based on the terrain parameters to be trained.

[0029] In this embodiment of the application, the motion control module is further configured to control the simulated robot to move on the training terrain based on the next motion action, so as to obtain the next motion state.

[0030] In this embodiment of the application, the model training module is further configured to combine the motion state to be processed, the action prediction reward of the next motion state, and random disturbances to calculate the decision index of the control model to be trained. The random disturbances include at least one of external disturbance force, external torque, system delay, ground friction, and power supply load. The control model to be trained is trained based on the decision index to obtain the robot control model.

[0031] In this embodiment, the robot control device further includes a model deployment module for deploying the robot control model to the real robot; determining a target motion action based on the robot control model in response to a motion control command; and controlling the real robot to move on the real terrain based on the target motion action.

[0032] This application provides an electronic device for robot control, the electronic device comprising:

[0033] Memory is used to store executable instructions or computer programs.

[0034] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the robot control method provided in the embodiments of this application.

[0035] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the robot control method provided in this application.

[0036] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the robot control method provided in this application.

[0037] The embodiments of this application have at least the following beneficial effects: In the process of training the robot control model, when predicting actions based on the current motion state of the simulated robot, the current motion state is first randomized in the domain based on the motion difference between the simulated robot and the real robot, and then the next motion action is predicted based on the randomized motion state to be processed, so as to train the robot control model based on the next motion action; Since the domain randomization is based on the motion difference between the simulated robot and the real robot in the above process of training the robot control model, the trained robot control model can reduce the deviation between the simulated robot and the real robot, thus improving the accuracy of the robot control model, and thus improving the control accuracy of the robot. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the architecture of the robot control system provided in an embodiment of this application;

[0039] Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram;

[0040] Figure 3 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 1 ;

[0041] Figure 4 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 2 ;

[0042] Figure 5 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 3 ;

[0043] Figure 6 This is a schematic diagram of the process for obtaining target servo parameters provided in an embodiment of this application;

[0044] Figure 7 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 4 ;

[0045] Figure 8 This is an exemplary robot control framework diagram based on reinforcement learning provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms “first, second, third, etc.” are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, third, etc.” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0051] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] 1) Reinforcement Learning (RL) is a method in Machine Learning (ML) used to enable an agent (e.g., a robot control model) to learn the optimal policy through interaction with the environment, thereby maximizing accumulated rewards. In other words, it is a method of learning the optimal policy based on a reward mechanism. In this embodiment, a robot control model is trained using reinforcement learning.

[0054] 2) Domain randomization refers to the introduction of randomness into the training and testing of robot control models in a simulated environment (also known as a simulation environment) to improve the generalization ability and robustness of the control model in the real world.

[0055] It should be noted that, in order to control robot movement, a robot control model is typically trained using a simulated robot, and then deployed onto a real robot to achieve control over the robot's movement. For example, training the robot control model directly on a simulated robot during the training process affects the accuracy of the transfer of the robot control model from simulation to reality, thus impacting the robot's control accuracy and stability. Furthermore, the randomness of domain randomization affects training accuracy, which in turn affects the robot's control accuracy and stability; and equating the driving efficiency of different motion drive directions of the servo motor model affects the accuracy of the servo motor model, which in turn also affects the robot's control accuracy and stability.

[0056] Based on this, embodiments of this application provide a robot control method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the control accuracy and stability of the robot. The following describes exemplary applications of the electronic device (hereinafter referred to as robot control device) provided in this application for implementing robot control. The robot control device provided in this application can be implemented as various types of terminals such as robots, smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers. It can also be implemented as a server, or a combination of both; this application does not limit this. The following will describe exemplary applications when the robot control device is implemented as a server and a terminal.

[0057] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the robot control system provided in the embodiments of this application; as shown Figure 1As shown, to support a robot control application, in the robot control system 100, terminals 200 (terminals 200-1 and 200-2 are shown as examples) are connected to server 400 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the robot control system 100 also includes a database 500 for providing data support to server 400; and... Figure 1 The example shown illustrates a scenario where the database 500 is independent of the server 400. However, the database 500 can also be integrated into the server 400, and this embodiment does not limit this to any particular case.

[0058] Server 400 is used to perform domain randomization on the current motion state of the simulated robot based on the motion differences between the simulated robot and the real robot, obtaining the motion state to be processed. The simulated robot is the simulation result of the real robot. A control model to be trained is used to predict the action of the motion state to be processed, obtaining the next motion action. Based on the next motion action, the simulated robot is controlled to move, obtaining the next motion state. Based on the action prediction reward of the next motion state, the control model to be trained is trained to obtain the robot control model, which is used to control the motion of the real robot. Server 400 is also used to send the robot control model to terminal 200 via network 300.

[0059] Terminal 200 is used to deploy a robot control model to a real robot 600; in response to motion control commands, it determines a target motion action based on the control model to be trained; and controls the real robot to move on real terrain based on the target motion action (graphical interfaces 200-11 and 200-21 for controlling the movement of the real robot are shown as examples).

[0060] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0061] See Figure 2 , Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram; such as Figure 2As shown, server 400 includes at least one processor 410, memory 450, at least one network interface 420, and user interface 430. The various components in terminal 400 are coupled together via bus system 440. It is understood that bus system 440 is used to implement communication between these components. In addition to a data bus, bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0062] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0064] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0065] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0066] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0067] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0068] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0069] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0070] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0071] In some embodiments, the robot control device provided in this application can be implemented in software. Figure 2 A robot control device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: domain randomization module 4551, motion prediction module 4552, motion control module 4553, model training module 4554, difference determination module 4555, terrain construction module 4556, and model deployment module 4557. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0072] In some embodiments, the robot control device provided in this application can be implemented in hardware. As an example, the robot control device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the robot control method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0073] In some embodiments, the terminal or server can implement the robot control method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as a robot APP; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0074] The robot control method provided in this application will be described below with reference to exemplary applications and implementations of the robot control device provided in the embodiments of this application. Furthermore, the robot control method provided in this application is applicable to various robot control scenarios such as cloud technology, artificial intelligence, and smart transportation. For example, the robot control method provided in this application can be applied to scenarios such as manufacturing, service industries, home assistance, and rescue. For instance, in manufacturing scenarios, by controlling the robot's movement, the robot can grasp and install various parts; in service industries, by controlling the robot's movement, the robot can remove goods from shelves and transport them to designated loading and unloading areas; in rescue scenarios, by controlling the robot's movement, the robot can perform search and rescue operations in the rescue area.

[0075] See Figure 3 , Figure 3 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 1 ,in, Figure 3 The main body executing each step is the robot control equipment; the following will combine... Figure 3 The steps shown are explained.

[0076] Step 101: Based on the motion differences between the simulated robot and the real robot, the current motion state of the simulated robot is randomized in the domain to obtain the motion state to be processed.

[0077] In this embodiment, the robot control device can acquire the motion difference between the simulated robot and the real robot. This motion difference is obtained by fitting the motion data difference between the simulated robot and the real robot. Responding to control commands for the simulated robot, the robot control device acquires the current state of the simulated robot, thus obtaining the current motion state. "Current" refers to the moment when an action needs to be determined based on the state of the simulated robot. After acquiring the current motion state, to improve training accuracy, the robot control device performs domain randomization on the current motion state using the motion difference between the simulated robot and the real robot, adjusting the current motion state based on this difference. The domain-randomized current motion state is the motion state to be processed.

[0078] It should be noted that the simulated robot is the simulation result of the real robot, which is a robot model built based on the real robot and used to simulate the real robot; while the real robot is the robot in actual application, such as food delivery robot, detection robot, etc. The current motion state includes at least one of the following information of the simulated robot: foot support mask, base Euler angles (also known as base orientation angles), base linear velocity, base angular velocity, joint angles, and joint angular velocities; wherein, the foot support mask represents lifting or lowering the foot (e.g., lifting the foot is represented as 1, lowering the foot is represented as 0), the base angular velocity is obtained through a sensor, the base Euler angles are obtained by integrating the base angular velocity, and the base linear velocity is obtained by integrating the linear acceleration obtained from the sensor; therefore, the robot control device performs domain randomization on the current motion state of the simulated robot based on the motion differences between the simulated robot and the real robot, which means performing domain randomization on at least one of the following information of the simulated robot included in the current motion state based on the motion differences between the simulated robot and the real robot: foot support mask, base Euler angles, base linear velocity, base angular velocity, joint angles, and joint angular velocities. In addition, the control commands for the simulated robot can be input periodically based on the task to be executed, or they can be input all at once based on the task to be executed. This application embodiment does not limit this. The task to be executed is, for example, moving towards the destination at a speed of 1 meter per second, or delivering tea.

[0079] It should also be noted that simulated robots are robot models created through computer programs in a virtual environment. They do not exist in the physical world but run on computers or other simulation platforms to simulate the behavior and performance of real robots. The characteristics of simulated robots include virtuality, editability, safety, and economy. Virtuality means that simulated robots do not occupy physical space and exist within computer programs. Editability means that the design and parameters of simulated robots can be modified during various experiments and tests without physical manufacturing. Safety means that simulated robots operate in a virtual environment, reducing the physical harm or damage caused by experiments and tests. Economy means that compared to real robots, the physical manufacturing and maintenance costs of simulated robots are lower. Real robots, on the other hand, are physically existing physical robots, including autonomous or remote-controlled operation functions, capable of performing tasks in real environments. The characteristics of real robots include physics, complexity, risk, and applicability. Physicality refers to the physical form of real robots, enabling them to interact with the real world. Complexity refers to the various physical and engineering factors involved in designing and manufacturing real robots. Risk refers to the safety risks associated with testing and operating real robots compared to simulated robots. Applicability means that real robots can be deployed in real-world application scenarios to solve practical problems. Simulated robots provide the theoretical foundation and experimental platform for real robots. They allow for the prediction of real robot behavior and the optimization of real robot designs. Verifying the relevant processing mechanisms of real robots in a simulation environment reduces the risks of actual operation. Experimenting with simulated robots in a virtual environment enables real robots to face complex and unpredictable challenges in real-world environments. Furthermore, both simulated and real robots can be used to create humanoid robots.

[0080] Step 102: Use the control model to be trained to predict the motion state of the object to be processed, and obtain the next motion action.

[0081] In this embodiment, the robot control device uses a simulated robot instead of a real robot for model training. Once the control model to be trained is obtained, it is trained based on the simulated robot. During model training, the robot control device inputs the motion state to be processed into the control model to be trained, and uses the control model to predict the action at the next moment based on the motion state to be processed. The result obtained is the next motion action.

[0082] It should be noted that the control model to be trained is a neural network model used to control the motion of a real robot; it can be a constructed original neural network model, a pre-trained neural network model, or a large model, etc., and this application does not limit this. In addition, the control model to be trained is trained using reinforcement learning. The next motion action refers to the action of the simulated robot at the next moment, such as a joint angle.

[0083] It's also important to note that joint angles refer to the angle between two adjacent bones or components in a robot. Joint angles reflect the range of motion and position of a joint; in robots, joint angles characterize the joint movement of a robotic arm or similar device. Changes in joint angles enable the robot's limbs to perform various movements, such as flexion, extension, abduction, and adduction. By controlling joint angles, the movement trajectory and posture of the robot's robotic arm can be precisely controlled, allowing the robot to accurately perform various tasks, such as assembly, handling, and welding. The measurement and control of joint angles typically rely on high-precision equipment such as angle encoders and servo control systems.

[0084] In this embodiment, when the robot control device uses the control model to be trained for motion prediction, in addition to the motion state to be processed, it can also combine at least one of the following: control instructions, the previous M motion states to be processed, and the previous M motion actions; where M is a positive integer. The control instructions, for example, are speed instructions.

[0085] Step 103: Control the simulated robot to move based on the next motion action to obtain the next motion state.

[0086] In this embodiment, after the robot control device obtains the next motion action, it drives the simulated robot to move based on the next motion action, thus realizing motion control of the simulated robot; here, the simulated robot executes the next motion action and enters the next motion state.

[0087] It should be noted that the next motion state refers to the motion state of the simulated robot after it performs the next motion action; the content type of the next motion state is the same as that of the current motion state, and will not be described again in this embodiment of the application.

[0088] In this embodiment of the application, the robot control device controls the simulated robot to move based on the next motion action to obtain the next motion state, including: the robot control device controls the simulated robot to move on the training terrain based on the next motion action to obtain the next motion state.

[0089] It should be noted that the training terrain is constructed based on the dimensions of the simulated robot and can include at least one of the following terrain types: random terrain, pyramidal slope terrain, wavy terrain, and discrete obstacle terrain. Random terrain refers to terrain containing random obstacles; that is, the obstacles are randomly distributed on the terrain or generated based on a random process. In the simulation environment, random terrain is typically generated using random algorithms or noise functions (such as Gaussian noise, Perlin noise, etc.). Pyramid slope terrain has a hierarchical structure, with each layer being a slope. The top of the terrain is smaller than the bottom, resembling a pyramid shape. Pyramid slope terrain can be constructed using geometric modeling or terrain generation algorithms, and the slope of each layer can be the same or different. Pyramid slope terrain can be used to test the robot's climbing ability. Wavy terrain resembles the shape of ocean waves, exhibiting periodic undulations. It can be a two-dimensional or three-dimensional waveform, which is not limited in this embodiment. Wavy terrain can be simulated using mathematical equations (such as sine waves, cosine waves, etc.), and different wave sizes and speeds can be simulated by adjusting the amplitude and frequency of the wave terrain. Wavy terrain is used to test the motion stability of the simulated robot. Discrete obstacle terrain consists of obstacles scattered on a plane. These obstacles can be regular or irregular, such as cylinders, blocks, rocks, etc. This application embodiment does not limit this. Discrete obstacle terrain can place obstacles randomly or by a specific algorithm, or it can be modeled based on actual terrain data. This application embodiment does not limit this. Discrete obstacle terrain is suitable for robot obstacle avoidance testing.

[0090] Step 104: Train the control model to be trained based on the action prediction reward of the next motion state to obtain the robot control model.

[0091] In this embodiment, the robot control device calculates a reward based on the next motion state, thus obtaining a motion prediction reward. Since the motion prediction reward represents the accuracy of the next motion prediction and is positively correlated with it (i.e., a higher reward indicates higher accuracy, and a lower reward indicates lower accuracy), the robot control device adjusts the model parameters of the control model to be trained based on the motion prediction reward. When training is complete, the trained control model is the robot control model. This robot control model is used to control the motion of a real robot.

[0092] It should be noted that the training of the control model to be trained can be iterative. When the iteration termination condition is met, the training ends, and the control model trained in the last iteration is determined as the robot control model. The iteration termination condition can be reaching an accuracy threshold, reaching an iteration count threshold, reaching an iteration duration threshold, or a combination of the above, etc., and this application does not limit this. Furthermore, the motion prediction reward includes at least one of motion command reward, gait imitation reward, disturbance resistance reward, and regularization reward; wherein, the motion command reward represents the reward for the simulated robot's response to motion commands, the gait imitation reward represents the reward for the simulated robot's gait, and the disturbance resistance reward represents the reward for the simulated robot's disturbance resistance.

[0093] It should also be noted that in reinforcement learning, the reward serves as immediate feedback to the control model being trained. It conveys to the model the accuracy of the agent's actions in a given state in achieving the target. Rewards include positive and negative rewards. A positive reward indicates that the model's output action is close to the final target, while a negative reward indicates that the model's output action deviates from the final target. Since reinforcement learning aims to maximize the cumulative reward for the control model during movement, the reward can be used to adjust the model, resulting in increasingly higher cumulative rewards during the adjustment process. In summary, the reward serves as training guidance information for the control model, guiding its learning by providing positive or negative feedback. Through rewards, the model can determine effective actions. Furthermore, the magnitude and frequency of the reward value can affect the learning rate of the control model. Positive rewards greater than a specified value accelerate the learning process, while negative rewards less than a specified value negative reward affect the learning efficiency.

[0094] See Figure 4 , Figure 4 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 2 ,in, Figure 4 The main body responsible for executing each step is robot control; such as Figure 4 As shown, step 104 can be implemented through steps 1041 and 1042; that is, the robot control device trains the control model to be trained based on the action prediction reward of the next motion state to obtain the robot control model, including steps 1041 and 1042. Each step is explained below.

[0095] Step 1041: Combine the motion state to be processed, the predicted reward of the next motion state, and the random perturbation to calculate the decision index of the control model to be trained.

[0096] In this embodiment, the robot control device can directly train the control model based on the motion state to be processed and the action prediction reward for the next motion state. Alternatively, it can combine the motion state to be processed, the action prediction reward, and random disturbances for training; this embodiment does not limit the specific training method. When the robot control device trains the control model by combining the motion state to be processed, the action prediction reward, and random disturbances, the prediction deviation for the next motion action is first determined by combining these factors, thus obtaining the decision index. Therefore, the decision index is negatively correlated with the accuracy of action prediction; that is, a higher decision index results in lower accuracy of action prediction, and vice versa. However, when the robot control device directly trains the control model based on the motion state to be processed and the action prediction reward, the decision index is calculated based on these two factors.

[0097] It should be noted that random disturbances include at least one of the following: external disturbance force, external torque, system delay, ground friction, and power supply load. Among them, external disturbance force refers to the external thrust applied to the simulated robot, such as the external thrust applied to the belly of the simulated robot; external torque is the external rotational force applied to the simulated robot, such as the external thrust applied to the shoulder of the simulated robot; system delay refers to the delay in reading data from the simulated sensor; and power supply load refers to the load corresponding to the external power supply equipment of the simulated robot.

[0098] For example, the decision metric could be the Temporal Difference Error (TD Error); in reinforcement learning, the T Temporal Difference Error measures the difference between the predicted value and the actual observed value, and is the difference between the target value (the predicted sum of future rewards) and the current value (the estimated reward of the current state).

[0099] Step 1042: Train the control model to be trained based on decision indicators to obtain the robot control model.

[0100] In this embodiment of the application, the robot control device trains the control model to be trained based on the action prediction reward of the next motion state, which is essentially training the control model to be trained based on decision indicators.

[0101] It should be noted that the robot control device uses reinforcement learning to train the control model. Reinforcement learning is a subfield of machine learning that learns to take actions in a specific environment to maximize cumulative rewards. Reinforcement learning provides a framework for modeling agents making decisions in uncertain environments, improving the accuracy of robot motion control in various scenarios such as games, autonomous vehicles, and resource management. Reinforcement learning learns through the interaction between the agent and the environment, without requiring labeled data, making it suitable for scenarios with uncertain labels and time-consuming labeling processes (greater than a specified time), thus improving the applicability of the scenario. Reinforcement learning allows the agent to learn through trial and error, enabling it to autonomously learn to act in a given environment, reducing human intervention and improving learning intelligence and efficiency. Reinforcement learning models can adaptively adjust according to different tasks and environments, making them suitable for handling dynamic and changing environments. Reinforcement learning focuses on long-term (greater than a specified time) rewards, which can improve training accuracy. Reinforcement learning can learn in real-time environments, making it suitable for real-time decision-making applications. Since robot control is a real-time decision-making application, using reinforcement learning to train the robot control model can improve the accuracy of the robot control model.

[0102] It is understandable that when a robot control device incorporates random disturbances to train a control model, it indicates that the simulated robot is executing its next motion under random disturbances. This increases the complexity of the training process, enhances the similarity between the simulation environment and the real environment, and consequently improves the accuracy of the robot control model.

[0103] See Figure 5 , Figure 5 This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 3 ,in, Figure 5 The main body responsible for executing each step is robot control; such as Figure 5 As shown, steps 105 to 108 are included before step 101; that is, before the robot control device performs domain randomization on the current motion state of the simulated robot based on the motion difference between the simulated robot and the real robot to obtain the motion state to be processed, the robot control method also includes steps 105 to 108. Each step is explained below.

[0104] Step 105: Optimize the initial servo parameters based on motion samples to obtain the target servo parameters.

[0105] It should be noted that the robot control device can acquire initial servo parameters, which represent the initial parameters of the servo motors. The servo motors are used to drive the robot's movement based on motion samples. Motion samples refer to samples of motion actions, i.e., the motion samples used to drive the robot's movement. Here, the robot control device drives both the simulated robot and the real robot to move based on the motion samples, adjusting the initial servo parameters based on the differences between the respective motion results to optimize the initial servo parameters; the optimized initial servo parameters are the target servo parameters.

[0106] For example, servo motors can control the various joints of a robotic arm to achieve position and angle control; servo motors can also be used to control the robot's gait and movement; in robots that autofocus or follow targets, servo motors can be used to adjust the position and angle of the camera; servo motors can also be used to control the stability of the robot's movement; servo motors can also control the robot's head movement and the opening and closing of the gripper, etc.

[0107] See Figure 6 , Figure 6 This is a schematic diagram of the process for obtaining target servo parameters provided in an embodiment of this application, wherein, Figure 6 The main body responsible for executing each step is robot control; such as Figure 6 As shown, step 105 can be achieved through steps 1051 to 1053; that is, the robot control device optimizes the initial servo parameters based on the motion sample to obtain the target servo parameters, including steps 1051 to 1053. Each step is explained below.

[0108] Step 1051: Use motion samples and initial servo parameters to control the real robot to move, and obtain the second real motion data.

[0109] It should be noted that the robot control device uses initial servo parameters to control the real robot to execute motion samples to move, and the result obtained is the second real motion data; therefore, the second real motion data represents the motion data obtained by using the initial servo parameters to control the real robot to execute motion samples.

[0110] Step 1052: Use motion samples and initial servo parameters to control the simulated robot to move, and obtain the second simulation motion data.

[0111] In this embodiment, the robot control device uses initial servo parameters to control the simulated robot to execute motion samples to move, and the result obtained is the second simulated motion data; thus, the second simulated motion data represents the motion data obtained by controlling the simulated robot to execute motion samples using the initial servo parameters.

[0112] It should be noted that steps 1051 and 1052 are not sequential in execution; they can be executed simultaneously, alternately, or sequentially, or a combination thereof. This application does not limit this.

[0113] Step 1053: Optimize the initial servo parameters by combining the differences between the second real motion data and the second simulated motion data to obtain the target servo parameters.

[0114] In this embodiment, the robot control device adjusts the initial servo parameters by combining the difference between the second real motion data and the second simulated motion data, so as to optimize the initial servo parameters.

[0115] It should be noted that the difference between the second real motion data and the second simulated motion data includes at least one of the following: joint angle difference, joint velocity difference, base posture difference, and base angular velocity difference; wherein, the joint angle difference represents the difference between the joint angles of the real robot and the joint angles of the simulated robot, the joint velocity difference represents the difference between the joint velocities of the real robot and the joint velocities of the simulated robot, the base posture difference represents the difference between the base postures of the real robot and the base postures of the simulated robot, and the base angular velocity difference represents the difference between the base angular velocities of the real robot and the base angular velocities of the simulated robot.

[0116] Understandably, by optimizing the initial servo parameters from the perspectives of joint angle differences, joint speed differences, base attitude differences, and base angular velocity differences, the initial servo parameters are optimized from multiple dimensions, including joint angle, joint speed, base attitude, and base angular velocity. This improves the optimization effect and, consequently, the accuracy of the target servo parameters.

[0117] In this embodiment, the target servo parameters can be further optimized by first using motion samples to optimize the initial servo parameters, and then optimized again based on the motion samples, and the optimization result at this time is used as the target servo parameters. This embodiment does not limit this.

[0118] In the embodiments of this application, the target servo parameters can be fixed parameter values ​​or an optimal range of parameter values; the embodiments of this application do not limit this.

[0119] It is understandable that the robot control device drives the real robot and the simulated robot respectively based on motion samples, and then optimizes the target servo parameters based on the difference between the second real motion data and the second simulated motion data. In this way, the target servo parameters can reduce the motion difference between the real robot and the simulated robot.

[0120] Step 106: Calculate the first simulation motion data of the simulated robot based on the target servo parameters.

[0121] It should be noted that the robot control equipment calculates the motion data of the simulated robot based on the target servo parameters, thus obtaining the first simulated motion data; therefore, the first simulated motion data refers to the motion data obtained by the simulated robot moving based on the target servo parameters.

[0122] Step 107: Calculate the first real motion data of the real robot based on the target servo parameters.

[0123] It should be noted that the robot control equipment calculates the motion data of the real robot based on the target servo parameters, thus obtaining the first real motion data; therefore, the first real motion data refers to the motion data obtained by the real robot moving based on the target servo parameters.

[0124] In this embodiment of the application, steps 106 and 107 are not ordered in any particular order. They can be executed simultaneously, alternately, or sequentially, or a combination thereof. This embodiment of the application does not limit this. Figure 5 The example shown is an exemplary execution order.

[0125] Step 108: Fit the difference between the first simulated motion data and the first real motion data to obtain the motion difference between the simulated robot and the real robot.

[0126] It should be noted that the robot control device uses a specified distribution to fit the difference between the first simulated motion data and the first real motion data, and the obtained fitting result is the motion difference between the simulated robot and the real robot; the motion difference between the simulated robot and the real robot can also be an extension of the specified quantity corresponding to the fitting result, which is not limited in this embodiment of the application.

[0127] In the embodiments of this application, the robot control device may use at least one of the following distributions to fit the difference between the first simulated motion data and the first real motion data: normal distribution, binomial distribution, Poisson distribution, exponential distribution, uniform distribution, and gamma distribution.

[0128] It's important to note that the normal distribution, also known as the Gaussian distribution, is a continuous probability distribution. Its probability density function is symmetric about the mean, resembling a bell curve. The normal distribution is determined by two parameters: the mean and the standard deviation. The mean determines the center of the distribution, while the standard deviation determines its width. The normal distribution is unimodal, and the area under the normal distribution is 1. It is applied in various fields, including natural sciences, social sciences, and engineering. The binomial distribution is a discrete probability distribution describing the probability of success in a fixed number of independent experiments. It is determined by two parameters: the number of experiments and the probability of success in each experiment. The binomial distribution is discrete, with a range of {0, 1, 2, ...}. The Poisson distribution is also a discrete probability distribution, describing the probability of a specified event occurring within a fixed time or spatial interval. It is determined by one parameter: the average occurrence rate of the event. The Poisson distribution is discrete. The exponential distribution is a continuous probability distribution that describes the probability distribution of time intervals between independent random events. It is determined by one parameter, the average occurrence rate of the event. The uniform distribution is a continuous probability distribution that describes a random variable where all values ​​have the same probability of occurring within a specified interval. It is determined by two parameters, representing the left and right endpoints of the interval. The gamma distribution is a continuous probability distribution used to describe waiting times or the time intervals between cumulative events. It is determined by two parameters: a shape parameter and a scale parameter.

[0129] For example, the robot control device uses a normal distribution to fit the difference between the first simulated motion data and the first real motion data. The fitted target normal distribution is the motion difference between the simulated robot and the real robot. The motion difference between the simulated robot and the real robot can also be an extension of the target normal distribution by a specified amount, such as extending the distribution parameters of the target normal distribution by a specified amount.

[0130] In this embodiment, when the difference between the second real motion data and the second simulated motion data includes differences in joint angles, joint velocities, base postures, and base angular velocities, the second real motion data includes real joint angles, real joint velocities, real base postures, and real base angular velocities, and the second simulated motion data includes simulated joint angles, simulated joint velocities, simulated base postures, and simulated base angular velocities. In this case, before the robot control device optimizes the initial servo parameters and obtains the target servo parameters by combining the differences between the second real motion data and the second simulated motion data in step 1053, the robot control method further includes: the robot control device acquiring the joint angle difference between the real joint angle and the simulated joint angle; acquiring the joint velocity difference between the real joint velocity and the simulated joint velocity; acquiring the base posture difference between the real base posture and the simulated base posture; acquiring the base angular velocity difference between the real base angular velocity and the simulated base angular velocity; and finally, combining the joint angle difference, joint velocity difference, base posture difference, and base angular velocity difference, determining the difference between the second real motion data and the second simulated motion data.

[0131] It should be noted that the difference between the second real motion data and the second simulated motion data is the combined result of the differences in joint angles, joint velocities, base postures, and base angular velocities. The combination of the differences in joint angles, joint velocities, base postures, and base angular velocities can be by accumulation, by weighted summation, by multiplication, etc., and this application embodiment does not limit this.

[0132] In this embodiment of the application, before the robot control device optimizes the initial servo parameters based on motion samples to obtain the target servo parameters in step 105, the robot control method further includes: the robot control device first determines the motor drive direction by combining the target joint angle, the current joint angle, and the current joint angular velocity; then, based on the total torque and the motor drive efficiency adapted to the motor drive direction, it determines the braking torque; next, based on the first mapping relationship between the braking torque, the output torque, and the total torque, and the second mapping relationship between the total torque and the motion drive information, it determines the output torque expression; finally, based on the output torque expression, it determines the servo parameters to be optimized, and the initial servo parameters are the initialized servo parameters to be optimized.

[0133] It should be noted that the target joint angle is the joint angle to be moved to. The robot control device combines the target joint angle, the current joint angle, and the current joint angular velocity to determine the motor drive direction. Different motor drive directions correspond to different motor drive efficiencies. The robot control device can combine information negatively correlated with motor drive efficiency with the total torque to obtain the braking torque. The braking torque represents the torque used to brake the motor drive. In the forward drive direction, the braking torque can be ignored, while in the reverse drive direction, the braking torque exists. For example, the robot control device obtains the efficiency difference between the maximum drive efficiency and the motor drive efficiency, and uses the product of the efficiency difference and the total torque as the braking torque. Motion drive information represents the information about the drive motor rotation, such as current, resistance, voltage, drive parameters (e.g., torque constant, back EMF constant, approximate linear slope of motor output, etc.), and angular velocity. The output torque expression represents the mapping relationship between the braking torque, motion drive information, and total torque. The servo parameters to be optimized are the parameters to be optimized in the output torque expression. The robot control device initializes the servo parameters to be optimized, and the obtained initialization result is the initial servo parameters.

[0134] In this embodiment of the application, the robot control device determines the motor driving direction by combining the target joint angle, the current joint angle, and the current joint angular velocity. This includes: the robot control device first obtains the joint angle difference between the current joint angle and the target joint angle; when the first numerical sign of the joint angle difference is the same as the second numerical sign of the current joint angular velocity, the forward driving direction is determined as the motor driving direction; and when the first numerical sign of the joint angle difference is different from the second numerical sign of the current joint angular velocity, the reverse driving direction is determined as the motor driving direction.

[0135] For example, in the case of forward rotation, the angular velocity has a positive sign, meaning the second numerical sign is positive. If the current joint angle is -3 degrees and the target joint angle is -5 degrees, then the joint angle difference is -2 degrees, and the data sign of the joint angle difference is negative, meaning the first numerical sign is negative. Therefore, the first numerical sign and the second numerical sign are different. However, in the case of forward rotation, the angular velocity has a positive sign, meaning the second numerical sign is positive. If the current joint angle is -5 degrees and the target joint angle is -3 degrees, then the joint angle difference is 2 degrees, and the data sign of the joint angle difference is positive, meaning the first numerical sign is positive. Therefore, the first numerical sign and the second numerical sign are the same.

[0136] In this embodiment of the application, before the robot control device determines the output torque expression based on the first mapping relationship between braking torque, output torque and total torque, and the second mapping relationship between total torque and motion drive information, the process of obtaining the first mapping relationship is also included; therefore, at this time, the robot control method further includes: the robot control device first obtains friction loss; and then combines friction loss, braking torque and output torque into total torque to obtain the first mapping relationship.

[0137] It should be noted that frictional losses include at least one of viscous friction, Coulomb friction, hysteresis loss, eddy current loss, and wind resistance loss. Among these, viscous friction is positively correlated with angular velocity; Coulomb friction is a constant and occurs in the opposite direction to the motor drive.

[0138] In this embodiment, before determining the output torque expression based on the first mapping relationship between braking torque, output torque, and total torque, and the second mapping relationship between total torque and motion drive information, the robot control device also includes a process of establishing the second mapping relationship. Therefore, the robot control method further includes: the robot control device first maps the joint angle difference between the current joint angle and the target joint angle to the applied voltage of the motor, obtaining a third mapping relationship; then it obtains a fourth mapping relationship between the applied voltage of the motor, the back electromotive force voltage, the motor current, and the motor resistance; and it obtains a fifth mapping relationship between total torque and motor current; finally, based on the third, fourth, and fifth mapping relationships, it obtains the second mapping relationship between total torque and motion drive information.

[0139] It should be noted that the third mapping relationship represents the mapping relationship between the voltage applied to the motor and the joint angle difference; the fourth mapping relationship represents the mapping relationship between the combined result of the voltage applied to the motor, the motor current and the motor resistance and the back electromotive force.

[0140] It should also be noted that in the embodiments of this application, the output torque, specified torque, and total torque are all torque. Torque refers to the moment when an object rotates about a specified point or axis. Torque is equal to the magnitude of the force multiplied by the perpendicular distance (lever arm) from the point of application of the force to the axis of rotation, reflecting the measure of the force's effect on the object's rotation. The unit of torque can be Newton-meter, or other units such as kilogram-meter, pound-force ruler, etc., which are not limited in this embodiment. Torque exists in mechanical devices such as engines and motors, that is, mechanical equipment is driven to rotate by generating torque. Torque is used to determine the robot's motion execution speed; in addition, torque has a direction and can be transmitted along the line of action of the force without changing the magnitude and direction of the force. Under constant power, torque is inversely proportional to rotational speed, that is, the higher the rotational speed, the smaller the torque, and the lower the rotational speed, the larger the torque.

[0141] In this embodiment of the application, when the motion prediction reward includes a motion command reward, before the robot control device trains the control model to be trained based on the motion prediction reward of the next motion state in step 104, the robot control method further includes: the robot control device first obtains the directional velocity reward, yaw rate reward, uprightness reward and speed matching reward corresponding to the next motion state; and then combines the directional velocity reward, yaw rate reward, uprightness reward and speed matching reward to obtain the motion command reward.

[0142] It should be noted that the velocity matching reward refers to the linear velocity and angular velocity matching reward of the next motion state; the motion command reward is the combined result of the directional velocity reward, yaw angular velocity reward, uprightness reward and velocity matching reward. The combination of the directional velocity reward, yaw angular velocity reward, uprightness reward and velocity matching reward can be by accumulation, by weighted summation, by multiplication, etc., and this application embodiment does not limit this.

[0143] In this embodiment, the robot control device determines the directional velocity reward from the sagittal and coronal directions.

[0144] In this embodiment of the application, when the motion prediction reward includes gait imitation reward, before the robot control device trains the control model to be trained based on the motion prediction reward of the next motion state in step 104, the robot control method further includes: the robot control device first obtains the joint angle reward of the next joint angle and the reference joint angle in the next motion state; and obtains the ground contact reward of the next motion state; and obtains the height matching reward of the base link height and the reference height in the next motion state; finally, the gait imitation reward is obtained by combining the joint angle reward, the ground contact reward and the height matching reward.

[0145] It should be noted that the combination of joint angle reward, ground contact reward and height matching reward can be in the form of accumulation, weighted summation, multiplication, etc., and the embodiments of this application do not limit this.

[0146] In this embodiment of the application, when the motion prediction reward includes an anti-disturbance reward, before the robot control device trains the control model to be trained based on the motion prediction reward of the next motion state in step 104, the robot control method further includes: the robot control device first obtains the ground contact force of the next motion state; when the ground contact force is greater than the ground contact force threshold, it determines the sliding penalty based on the foot linear velocity of the next motion state; finally, it obtains the anti-disturbance reward that is negatively correlated with the sliding penalty.

[0147] In this embodiment of the application, when the motion prediction reward includes a regularization reward, before the robot control device trains the control model to be trained based on the motion prediction reward of the next motion state in step 104, the robot control method further includes: the robot control device first obtains the energy consumption penalty, ground contact force penalty, motion difference penalty, joint angular velocity penalty, and angular acceleration penalty of the next motion state; and then combines the energy consumption penalty, ground contact force penalty, motion difference penalty, joint angular velocity penalty, and angular acceleration penalty to obtain the regularization reward.

[0148] In this embodiment of the application, before step 103, the robot control device controls the simulated robot to move based on the next motion action to obtain the next motion state, the robot control method further includes: the robot control device first determines the training terrain parameters based on the size of the simulated robot; and then constructs the training terrain based on the training terrain parameters.

[0149] In this embodiment of the application, the robot control device can also select a target terrain type from a variety of terrain types, determine the training terrain parameters that are compatible with the target terrain type based on the size of the simulated robot, and construct the training terrain by combining the terrain direction and the training terrain parameters.

[0150] It should be noted that the terrain direction includes a first terrain direction and a second terrain direction, wherein the first terrain direction and the second terrain direction are opposite. For example, the first terrain direction and the second terrain direction can be two opposite directions relative to the terrain plane.

[0151] In this embodiment, the robot control device can further determine the range of terrain parameters based on the size of the simulated robot, divide the range of terrain parameters to obtain a sequence of terrain parameters, and determine each terrain parameter in the sequence as a training terrain parameter for model training based on the sequence order represented by the terrain parameter sequence. When the survival time of the simulated robot on the training terrain corresponding to the current training terrain parameter is greater than a specified time, it is determined that model training will be performed on the training terrain corresponding to the next terrain parameter.

[0152] Understandably, constructing training terrain based on terrain type and orientation increases its complexity, thereby improving model training effectiveness and the generalization ability of the robot control model. Furthermore, determining the range of terrain parameters by simulating the robot's dimensions, and then progressively introducing increasingly complex training terrain based on this range, enhances the stability of model training and ultimately improves the accuracy of the robot control model.

[0153] See Figure 7 , Figure 7This is a flowchart illustrating the robot control method provided in the embodiments of this application. Figure 4 ,in, Figure 7 The main body executing each step is the robot control equipment; such as Figure 7 As shown, steps 109 to 111 are included before step 101; that is, after the robot control device trains the control model to be trained based on the action prediction reward of the next motion state to obtain the robot control model, the method further includes steps 109 to 111. Each step is described below.

[0154] Step 109: Deploy the robot control model onto the real robot.

[0155] It should be noted that the robot control model is a pre-trained neural network model used to control the movement of a real robot; thus, the robot control device deploys the robot control model onto the real robot to achieve motion control of the real robot through the robot control model.

[0156] Step 110: In response to motion control commands, determine the target motion action based on the robot control model.

[0157] It should be noted that when the real robot performs a task through movement, the robot control model also receives the motion control command. At this time, the robot control model responds to the motion control command and uses the robot control model to determine the motion action that matches the motion control command, thus obtaining the target motion action. Therefore, the target motion action represents the motion action output to the real robot.

[0158] Step 111: Control the real robot to move on the real terrain based on the target motion action.

[0159] It should be noted that the robot control device drives the real robot to move on the real terrain based on the target motion action, in order to control the real robot to move on the real terrain.

[0160] It is understandable that when a robot control model is deployed on a real robot for motion control, the accuracy and generalization of the robot control model can be improved because the training process of the robot control model is based on the motion differences between the simulated robot and the real robot, thereby improving the control accuracy of the real robot.

[0161] In the embodiments of this application, the processing related to the training of the robot control model can be implemented based on a server or a terminal; the processing related to the deployment of the robot control model can be implemented based on various terminals; and the processing related to the training of the robot control model and the processing related to the deployment of the robot control model can be executed using the same device or different devices, which is not limited in this embodiment. Furthermore, when the processing related to the training of the robot control model and the processing related to the deployment of the robot control model can be executed using the same device, the same device is the robot control device; and when the processing related to the training of the robot control model and the processing related to the deployment of the robot control model can be executed using multiple devices, the multiple devices are the robot control devices.

[0162] The following describes an exemplary application of the embodiments of this application in a practical application scenario. This exemplary application describes the process of accurately establishing a servo motor model by combining the energy flow direction of the servo motor (referred to as the motor drive direction), and identifying system parameters based on the established servo motor model to achieve fitting of the difference between simulated motion data and actual motion data, and then training a robot control model based on the fitting results, terrain construction, and external force disturbances.

[0163] See Figure 8 , Figure 8 This is an exemplary reinforcement learning-based robot control framework diagram provided in an embodiment of this application; as shown... Figure 8As shown, the first step involves establishing a servo model 8-1 by combining the positive energy flow 8-11 (referred to as the positive driving direction) and the reverse energy flow 8-12 (referred to as the reverse driving direction). The combination of the positive energy flow 8-11 and the reverse energy flow 8-12 is achieved by defining a torque 8-13. Typically, under the condition corresponding to the positive energy flow 8-11, the braking torque 8-13 is close to zero, while under the condition corresponding to the reverse energy flow 8-12, the braking torque 8-13 is greater than zero. The second step involves system identification 8-2 based on the model parameters of the servo model 8-1, obtaining the model parameter boundaries (referred to as the target servo parameters). System identification 8-2 includes simulation robot identification 8-21 and actual robot (referred to as the real robot) identification 8-22. The third step is to perform reinforcement training based on the model parameter boundaries. Reinforcement training includes simulation environment 8-3 and model training 8-4. Simulation environment 8-3 includes domain randomization 8-31, which in turn includes sensor noise 8-311 (referred to as the motion difference between the simulated robot and the real robot), system delay 8-312, external disturbance 8-313, and random load 8-314 (system delay 8-312, external disturbance 8-313, and random load 8-314 are collectively referred to as random disturbance). The simulation environment 8-3 also includes terrain learning 8-32, in which various terrains are constructed based on parameters 8-321, including random terrain 8-322, pyramid slope terrain 8-323, wave terrain 8-324, and discrete obstacle terrain 8-325 (all referred to as training terrain); reinforcement training also includes a reward module 8-33 to obtain instruction rewards 8-331, periodic gait imitation rewards 8-332, disturbance resistance rewards 8-333, and regularization rewards 8-334. In model training 8-4, the model network includes a policy model network 8-41 (referred to as the control model to be trained) and a policy evaluation network 8-42. The policy model network 8-41 takes velocity commands 8-411 and observation information 8-412 as inputs and outputs actions 8-413, which are executed by the simulated robot 8-43 to obtain a reward 8-421 based on the reward module 8-33. The policy evaluation network 8-42 determines evaluation information (TD Error) 8-415 based on the velocity commands 8-411, observation information 8-412, reward 8-421, and privilege information 8-414, and adjusts the model parameters of the policy model network 8-41 based on the evaluation information 8-415. Upon completion of training, the trained policy model network 8-41 is deployed to the real robot 8-5 to achieve motion control of the real robot 8-5. Furthermore, system identification 8-2 based on the model parameters of the servo model 8-1 is achieved by combining the basic motion data of the real robot 8-5.

[0164] Below, we will first explain the process of establishing the servo motor model based on the energy flow direction. Here, the energy flow direction refers to the transmission direction of the servo motor (also known as the motion drive direction), including the forward transmission direction and the reverse transmission direction; the servo motor refers to the motor of the servo motor model; the servo motor model is used to drive the humanoid robot to move.

[0165] The servo motor, based on its own control strategy, will adjust the desired joint angle q. desire The difference between the target joint angle (q) and the current actual joint angle (q) (called the current joint angle) (called the joint angle difference) is mapped to the voltage U applied to the servo motor. pwm (referred to as the voltage applied to the motor), U pwm Less than or equal to the supply voltage U battery As shown in formulas (1) and (2).

[0166]

[0167] Here, clip represents voltage cutoff processing, used based on (-U battery U battery )right Truncate the U to obtain pwm Less than or equal to the supply voltage U battery And greater than or equal to -U battery ; Represents the desired joint angle q desire The mapping result of k with the difference between the current actual joint angle q and the current joint angle q. pwm This represents the approximate linear slope of the servo motor output.

[0168] And because of voltage U pwm It can also be expressed by formula (3), which is shown below.

[0169]

[0170] Among them, U emf The back electromotive force voltage is shown in formula (4); I represents the current of the servo motor, R represents the resistance of the servo motor, and L represents the coil coefficient. This represents the change in current. Under steady-state conditions, the change in current is minimal, which is...

[0171]

[0172] Where, k e Represents the back electromotive force constant. This indicates the angular velocity of the servo motor's output shaft.

[0173] The total torque T of the servo motor is shown in formulas (1), (3) and (5). motor We can obtain the expression shown in formula (6). Formulas (5) and (6) are shown below.

[0174] T motor =k t ·I (5);

[0175]

[0176] Where, k t This represents the torque constant of the servo motor.

[0177] It should be noted that the total torque T motor This includes the output torque τ of the motor shaft driving the servo motor and the viscous friction τ inside the servo motor. vf Coulomb friction τ inside the servo motor cf Hysteresis loss τ h Eddy current loss τ v and wind resistance loss τ w In reality, τ h , τ v , τ w ≈0, viscous friction τ vf With angular velocity Proportional (as shown in formula (7)), Coulomb friction τ cf The value is constant and its direction is opposite to the direction of motion drive (as shown in formula (8)). Since the transmission efficiency of the servo motor is asymmetrical in different transmission directions, a braking torque τ is introduced. brake (As shown in formula (9)), based on the angular velocity and the desired joint angle q desire The difference between the current actual joint angle q and the current actual joint angle determines the transmission direction. Different transmission efficiencies (called motor drive efficiencies) are determined for different transmission directions, and the different transmission efficiencies are integrated into the servo motor physics engine (as shown in formula (10)). Formulas (7) to (10) are shown below.

[0178]

[0179] τ motor =τ+τ vf +τ cf +τ brake (10);

[0180] Where, k vf The coefficient of viscous friction is represented by `sign`, which is used to determine the sign of the coefficient. coulomb η represents the value of the Coulomb friction torque. forward η represents the forward transmission efficiency.backward This indicates the efficiency of reverse transmission.

[0181] when Less than the supply voltage U battery Based on formulas (6) to (10), formula (11) can be obtained; formula (11) is shown below.

[0182]

[0183] Based on formula (12), formula (11) is simplified to obtain formula (13); formula (13) is shown below.

[0184]

[0185] Where η is η forward or η backward k pwm (qq desire Within the power supply voltage range.

[0186] In this embodiment of the application, the servo motor model can be established using a neural network model.

[0187] Secondly, the system identification process of the humanoid robot will be explained.

[0188] It should be noted that, based on formula (13), the parameter μ to be optimized can be determined. motor (referred to as the servo parameters to be optimized) is (k) p k d k s η forward η backward q(m, f), where m is mass and f is friction force. In this embodiment, real-world motion segments are collected by performing basic motions of different amplitudes and frequencies to obtain time-series data q of joint angles. target Time-series data is input into the servo motor to control the movement of the actual robot, and actual sensor information H of the actual robot is collected during the movement. real (referred to as second true motion data); actual sensor information H real Including the joint angle q of each joint real Joint angular velocity and current I real It also includes the inertial measurement unit (IMU) information of the actual robot, which is used to estimate the base link position of the actual robot. The base link position includes the base quaternion p. quat，real and base angular velocity p avel，real , base quaternion p quat，real This represents the actual robot's posture, with the base angular velocity p. avel，realThis represents the actual angular velocity of the robot.

[0189] Time-series data is input into the servo motor to control the movement of the simulated robot, and simulation sensor information H of the robot is collected during the movement. sim (referred to as the second simulated motion data), thus, the simulated sensor information H sim Compared with actual sensor information H real The difference between L robot It can be as shown in formula (14).

[0190]

[0191]

[0192] Where T represents the total number of datasets collected (j is used to iterate through T), and N represents the length of each dataset (i is used to iterate through N). (Referred to as simulated joint angle) represents the joint angle of the simulated robot corresponding to the i-th data point in the j-th dataset. (Referred to as the actual joint angle) represents the actual robot joint angle corresponding to the i-th data point in the j-th dataset. (Referred to as simulated joint velocity) represents the joint angular velocity of the simulated robot corresponding to the i-th data point in the j-th dataset. (Referred to as the actual joint velocity) represents the actual joint angular velocity of the robot corresponding to the i-th data point in the j-th dataset. (Referred to as the simulation base posture) represents the posture of the simulated robot corresponding to the i-th data point in the j-th dataset. (Referred to as the actual base pose) represents the actual robot pose corresponding to the i-th data point in the j-th dataset. (Referred to as the simulated base angular velocity) represents the base angular velocity of the simulated robot corresponding to the i-th data point in the j-th dataset. (Referred to as the actual base angular velocity) represents the actual base angular velocity of the robot corresponding to the i-th data point in the j-th dataset.

[0193] It should be noted that, based on formula (14), μ motor Optimization can yield optimized parameters. Next, a dataset is randomly selected from the full dataset, and for each dataset, the optimized parameters are used. To initialize the parameters, continue optimizing the parameters. The optimization process is similar to that shown in formula (14), and will not be described again in this embodiment. In this case, optimization coefficients can also be combined. The parameters are optimized; finally, based on the optimization results of each randomly selected dataset, the maximum optimization parameter μ is obtained. max and minimum optimization parameter μ min And based on ω h (For example, 0.1) Expand the maximum and minimum optimization parameters (by 10%) to obtain the maximum parameter boundary μ. motor，min and minimum parameter boundary μ motor，max As shown in formulas (15) and (16).

[0194] μ motor，min =μ min -ω h (μ max -μ min (15);

[0195] μ motor，max =μ max -ω h (μ max -μ min (16);

[0196] Finally, the reinforcement learning process is explained.

[0197] It should be noted that the neural network model in reinforcement learning includes a policy network model and a policy evaluation model. The input to the policy network model includes the velocity command v. com (3D) Periodic Support Mask e m (2D), Euler angles (direction angles) of the base [B] R B P B Y (3D), base angular velocity (3D), joint angles (21D), joint angular velocity (21D), and the action output from the previous step (21D). Among these, the velocity command v com =[v x v y ,ω],v x vx and vx are the desired forward and lateral velocities specified in the simulated robot base, while ω is the desired global yaw velocity; the Euler angles of the base [B] can be used. R B P B Y ] and the Cartesian coordinates of the simulated robot in the world [B x B y B z [] indicates the robot's base posture.

[0198] In this embodiment of the application, in order to improve the robustness of the neural network model, multi-frame historical information h can be used. times(For example, 2); that is, the first h times -1 input information and h-th times The input information from each step is used together as the input to the policy network model; therefore, the input dimension of the policy network model is 74·h. times In addition, the output action of the policy network model can be the target joint position, so as to control the movement of the humanoid robot based on the target joint position.

[0199] It should be noted that the input to the strategy evaluation model includes not only the input to the strategy network model, but also the reference joint position (21-dimensional), random perturbation forces (3-dimensional), random perturbation torques (3-dimensional), environmental friction (1-dimensional), the mass of the virtual link (6-dimensional), terrain height (187-dimensional), and foot collision state (2-dimensional). Therefore, the input dimension of the strategy evaluation model is 297·h. times .

[0200] It is understandable that by inputting the terrain height into the policy evaluation model, and having the policy evaluation model evaluate its output based on the terrain height, the motion stability of humanoid robots on uneven terrain can be improved.

[0201] In this embodiment of the application, the input to the policy network model further includes the reward r of the output action, and the reward r includes the instruction reward r. commmand Periodic gait imitation reward r imitation Anti-disturbance reward r disturbance and regularization reward r norm As shown in formula (17).

[0202] r = r command +r imitation +r disturbance +r norm (17);

[0203] In this embodiment of the application, the following error index K(e, λ) is used to determine various rewards, where e represents the following error and λ represents the relevant weight; as shown in formula (18).

[0204] κ(e,λ)=exp(-λ·||e||2) (18);

[0205] Instruction reward r commmand As shown in formula (19).

[0206]

[0207] in, and Calculated based on formula (18); κ([B) represents the velocity following reward in the sagittal and coronal directions of the simulated robot. R B P ],5) indicates the yaw rate follow-up reward of the simulated robot. The simulation represents the reward for the robot's upright posture, while the simulation represents the reward for the mismatch between the linear velocity and angular velocity of the humanoid robot.

[0208] Periodic gait imitation reward r imitation As shown in formula (20).

[0209]

[0210] Among them, κ(qq) target ,2),κ(I-exp(-0.045·F 2 ),5) andκ(B z -G p [2], 100) is calculated based on formula (18); F represents the force of the simulated robot's foot contacting the ground; 1.6·K(qq) target ,2) indicates the reference angle following the reward, 1.2·K(I-exp(-0.045·F 2 ),5) represents the reward for foot contact with the ground consistent with the reference gait phase, κ(B z -G p [2], 100) represents the matching reward between the base link height of the simulated robot and the reference height in the gait library.

[0211] Disturbance resistance reward r disturbance As shown in formula (21).

[0212] r disturbance = -0.05·K(v) feet ,1) 0.5 (twenty one);

[0213] Wherein, κ(v) feet ,1) 0.5 Based on formula (18), v is calculated. feet This represents the linear velocity of the robot's foot. This penalty is activated when there is contact force between the foot and the ground.

[0214] Regularized reward r norm As shown in formula (22).

[0215]

[0216] Where -0.002·|τ| represents the energy penalty, and -0.1·max(F L，R-45, 0, 40) represents the penalty for the ground contact force exceeding the specified range (0, 40), -0.002·||a t -2a t-1 +a t-2 ||2 represents continuous action a t a t-1 and a t-2 Difference penalties between them Indicates joint angular velocity penalty. This indicates the angular acceleration penalty.

[0217] In the simulation environment of this application embodiment, terrain parameters are generated based on information such as the size of the simulated robot, and various types of terrain are then constructed based on these parameters; for example, random terrain, pyramid slope terrain, wavy terrain, and discrete obstacle terrain; the proportions of each type of terrain are the same. Furthermore, for pyramid slope terrain and wavy terrain, two opposing terrain parameters are used to construct the terrain. During training on the constructed terrain, a fixed-order training progression is employed; for example, for each type of terrain, the total range of terrain parameters is divided into 10 stages, and iterative training is performed starting from stage 1. When the survival time in the current stage exceeds a specified time threshold, training proceeds to the next stage of terrain.

[0218] Furthermore, in the simulation environment of this application embodiment, based on the maximum parameter boundary μ motor，min and minimum parameter boundary μ motor，max The difference distribution between the motion data of the simulated robot (referred to as the first simulated motion data) and the motion data of the actual robot (referred to as the first real motion data) is calculated. This difference distribution is then fitted using a normal distribution, and the fitted result is expanded by a specified amount to serve as sensor noise input for training the policy network model. Additionally, domain randomization includes external disturbances, system delays, and power supply loads. Table 1 shows the value range, unit, addition method, and type of the domain randomization terms other than sensor noise.

[0219] Table 1

[0220]

[0221]

[0222] It is understood that the embodiments of this application, by introducing the reverse drive characteristics of a high-ratio (greater than a specified gear ratio) servo motor and combining it with the energy flow direction to establish a servo motor model including reverse drive efficiency, can improve the accuracy and reliability of the servo motor model. Furthermore, by identifying system parameters for the motor model and robot dynamics model based on the servo motor model, the deviation between the simulation environment and the actual environment is reduced, improving the modeling and calibration capabilities of the simulation engine. In addition, during the system parameter identification process, introducing the difference distribution between simulation and real data for domain randomization can improve the applicability of the trained strategy in the actual environment; and, by performing reinforcement learning under complex terrain and random disturbances, the trained controller can cope with complex terrain and external disturbances, improving the robustness of the strategy in different environments and ensuring that the robot maintains stable motion performance when facing various uncertainties. This enables the robot to move forward, backward, and turn quickly and flexibly during motion tasks, ensuring efficient deployment and operation in practical applications.

[0223] The following description continues to illustrate the exemplary structure of the robot control device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the robot control device 455 in the memory 450 may include:

[0224] The domain randomization module 4551 is used to perform domain randomization on the current motion state of the simulated robot based on the motion difference between the simulated robot and the real robot, so as to obtain the motion state to be processed, wherein the simulated robot is the simulation result of the real robot.

[0225] The motion prediction module 4552 is used to predict the motion state to be processed using the control model to be trained, so as to obtain the next motion action.

[0226] The motion control module 4553 is used to control the simulated robot to move based on the next motion action to obtain the next motion state;

[0227] The model training module 4554 is used to train the control model to be trained based on the action prediction reward of the next motion state to obtain a robot control model, which is used to control the motion of the real robot.

[0228] In this embodiment, the robot control device 455 further includes a difference determination module 4555, which is used to optimize the initial servo parameters based on motion samples to obtain target servo parameters; calculate the first simulated motion data of the simulated robot based on the target servo parameters; calculate the second real motion data of the real robot based on the target servo parameters; and fit the difference between the first simulated motion data and the first real motion data to obtain the motion difference between the simulated robot and the real robot.

[0229] In this embodiment of the application, the difference determination module 4555 is further configured to control the real robot to move using the motion sample and the initial servo parameters to obtain second real motion data; control the simulated robot to move using the motion sample and the initial servo parameters to obtain second simulated motion data; and optimize the initial servo parameters by combining the difference between the second real motion data and the second simulated motion data to obtain the target servo parameters.

[0230] In this embodiment, the second real motion data includes real joint angles, real joint velocities, real base postures, and real base angular velocities; the second simulated motion data includes simulated joint angles, simulated joint velocities, simulated base postures, and simulated base angular velocities. The difference determination module 4555 is further configured to: acquire the joint angle difference between the real joint angles and the simulated joint angles; acquire the joint velocity difference between the real joint velocities and the simulated joint velocities; acquire the base posture difference between the real base posture and the simulated base posture; acquire the base angular velocity difference between the real base angular velocity and the simulated base angular velocity; and, by combining the joint angle difference, the joint velocity difference, the base posture difference, and the base angular velocity difference, determine the difference between the second real motion data and the second simulated motion data.

[0231] In this embodiment, the difference determination module 4555 is further configured to: determine the motor drive direction by combining the target joint angle, the current joint angle, and the current joint angular velocity; determine the braking torque based on the total torque and the motor drive efficiency adapted to the motor drive direction; determine the output torque expression of the output torque based on the first mapping relationship between the braking torque, the output torque, and the total torque, and the second mapping relationship between the total torque and the motion drive information; and determine the servo parameters to be optimized based on the output torque expression, wherein the initial servo parameters are the initialized servo parameters to be optimized.

[0232] In this embodiment of the application, the difference determination module 4555 is further configured to obtain the joint angle difference between the current joint angle and the target joint angle; when the first numerical sign of the joint angle difference is the same as the second numerical sign of the current joint angular velocity, the forward driving direction is determined as the motor driving direction; when the first numerical sign of the joint angle difference is different from the second numerical sign of the current joint angular velocity, the reverse driving direction is determined as the motor driving direction.

[0233] In this embodiment of the application, the difference determination module 4555 is further used to obtain friction loss, which includes at least one of viscous friction, Coulomb friction, hysteresis loss, eddy current loss and wind resistance loss; and to combine the friction loss, the braking torque and the output torque into the total torque to obtain the first mapping relationship.

[0234] In this embodiment of the application, the difference determination module 4555 is further configured to map the joint angle difference between the current joint angle and the target joint angle to the motor applied voltage to obtain a third mapping relationship; obtain a fourth mapping relationship between the motor applied voltage, back electromotive force voltage, motor current and motor resistance; obtain a fifth mapping relationship between the total torque and the motor current; and obtain a second mapping relationship between the total torque and the motion drive information based on the third mapping relationship, the fourth mapping relationship and the fifth mapping relationship.

[0235] In this embodiment of the application, the motion prediction reward includes at least one of motion command reward, gait imitation reward, disturbance resistance reward, and regularization reward; wherein, the motion command reward represents the reward for the simulated robot responding to motion commands, the gait imitation reward represents the reward for the simulated robot's gait, and the disturbance resistance reward represents the reward for the simulated robot's disturbance resistance.

[0236] In this embodiment of the application, when the motion prediction reward includes a motion command reward, the model training module 4554 is further configured to obtain the directional velocity reward, yaw rate reward, uprightness reward, and velocity matching reward corresponding to the next motion state, wherein the velocity matching reward refers to the linear velocity and angular velocity matching reward of the next motion state; and to obtain the motion command reward by combining the directional velocity reward, the yaw rate reward, the uprightness reward, and the velocity matching reward.

[0237] In this embodiment of the application, when the motion prediction reward includes gait imitation reward, the model training module 4554 is further configured to obtain the joint angle reward of the next joint angle and the reference joint angle of the next motion state; obtain the ground contact reward of the next motion state; obtain the height matching reward of the base link height and the reference height of the next motion state; and combine the joint angle reward, the ground contact reward and the height matching reward to obtain the gait imitation reward.

[0238] In this embodiment of the application, when the motion prediction reward includes an anti-disturbance reward, the model training module 4554 is further configured to obtain the ground contact force of the next motion state; when the ground contact force is greater than the ground contact force threshold, a sliding penalty is determined based on the foot linear velocity of the next motion state; and the anti-disturbance reward negatively correlated with the sliding penalty is obtained.

[0239] In this embodiment of the application, when the action prediction reward includes a regularization reward, the model training module 4554 is further configured to obtain the energy consumption penalty, ground contact force penalty, action difference penalty, joint angular velocity penalty, and angular acceleration penalty of the next motion state; and to obtain the regularization reward by combining the energy consumption penalty, the ground contact force penalty, the action difference penalty, the joint angular velocity penalty, and the angular acceleration penalty.

[0240] In this embodiment of the application, the robot control device 455 further includes a terrain construction module 4556, which is used to determine the terrain parameters to be trained based on the size of the simulated robot; and to construct the terrain to be trained based on the terrain parameters to be trained.

[0241] In this embodiment of the application, the motion control module 4553 is further configured to control the simulated robot to move on the training terrain based on the next motion action, so as to obtain the next motion state.

[0242] In this embodiment of the application, the model training module 4554 is further configured to combine the motion state to be processed, the action prediction reward of the next motion state, and random disturbances to calculate the decision index of the control model to be trained. The random disturbances include at least one of external disturbance force, external torque, system delay, ground friction, and power supply load. The control model to be trained is trained based on the decision index to obtain the robot control model.

[0243] In this embodiment, the robot control device 455 further includes a model deployment module 4557, which is used to deploy the robot control model to the real robot; determine a target motion action based on the robot control model in response to a motion control command; and control the real robot to move on the real terrain based on the target motion action.

[0244] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of a robot control device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the robot control device to perform the robot control method described in this application.

[0245] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the robot control method provided in this application. For example, ... Figure 3 The robot control method is shown.

[0246] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0247] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0248] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0249] As an example, computer-executable instructions can be deployed to execute on a single electronic device (in which case, this single electronic device is the robot control device), or to execute on multiple electronic devices located in one location (in which case, the multiple electronic devices located in one location are the robot control devices), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in which case, the multiple electronic devices distributed across multiple locations and interconnected via a communication network are the robot control devices).

[0250] It is understood that in the embodiments of this application, data related to robots and the like is involved. When the embodiments of this application are applied to specific products or technologies, the permission or consent of the information subject is required. The collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and conform to the principles of legality, legitimacy and necessity. It does not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0251] In summary, during the training of the robot control model, when predicting actions based on the current motion state of the simulated robot, the embodiments of this application first perform domain randomization on the current motion state based on the motion difference between the simulated robot and the real robot, and then predict the next motion action based on the domain randomized motion state to be processed, so as to train the robot control model based on the next motion action. Since the domain randomization is based on the motion difference between the simulated robot and the real robot during the above training process, the trained robot control model can reduce the deviation between the simulated robot and the real robot, improve the accuracy of the robot control model, and thus improve the control accuracy of the robot. In addition, during the model training process, by adding random perturbations and various types of training terrain, the complexity of the training environment can be increased, the gap between the training environment and the real environment can be reduced, and the accuracy of model training can be improved. Furthermore, when the trained robot control model is deployed on the real robot, the motion stability and anti-interference ability of the real robot in real terrain can be improved, thereby improving the control effect of robot motion.

[0252] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A robot control method, characterized in that, The method includes: Based on the motion differences between the simulated robot and the real robot, the current motion state of the simulated robot is randomized in the domain to obtain the motion state to be processed. The simulated robot is the simulation result of the real robot. The next motion action is obtained by using a control model to be trained to predict the motion state to be processed. Based on the next motion action, the simulated robot is controlled to move, and the next motion state is obtained; The control model to be trained is trained based on the action prediction reward of the next motion state to obtain a robot control model, which is used to control the motion of the real robot.

2. The method according to claim 1, characterized in that, Before performing domain randomization on the current motion state of the simulated robot based on the motion differences between the simulated robot and the real robot to obtain the motion state to be processed, the method further includes: The initial servo parameters are optimized based on motion samples to obtain the target servo parameters; The first simulated motion data of the simulated robot is calculated based on the target servo parameters; Calculate the first real motion data of the real robot based on the target servo parameters; The difference between the first simulated motion data and the first real motion data is fitted to obtain the motion difference between the simulated robot and the real robot.

3. The method according to claim 2, characterized in that, The optimization of the initial servo parameters based on motion samples to obtain the target servo parameters includes: The motion samples and the initial servo parameters are used to control the real robot to move, thereby obtaining the second real motion data; The simulated robot is controlled to move using the motion samples and the initial servo parameters to obtain the second simulated motion data. The initial servo parameters are optimized by combining the differences between the second real motion data and the second simulated motion data to obtain the target servo parameters.

4. The method according to claim 3, characterized in that, The second real motion data includes real joint angles, real joint velocities, real base posture, and real base angular velocity; the second simulated motion data includes simulated joint angles, simulated joint velocities, simulated base posture, and simulated base angular velocity. Before optimizing the initial servo parameters by combining the differences between the second real motion data and the second simulated motion data to obtain the target servo parameters, the method further includes: Obtain the joint angle difference between the actual joint angle and the simulated joint angle; Obtain the joint velocity difference between the actual joint velocity and the simulated joint velocity; Obtain the difference in base posture between the actual base posture and the simulated base posture; Obtain the difference in base angular velocity between the actual base angular velocity and the simulated base angular velocity; By combining the differences in joint angles, joint velocities, base postures, and base angular velocities, the differences between the second real motion data and the second simulated motion data are determined.

5. The method according to any one of claims 2 to 4, characterized in that, Before optimizing the initial servo parameters based on motion samples to obtain the target servo parameters, the method further includes: The motor drive direction is determined by combining the target joint angle, the current joint angle, and the current joint angular velocity. The braking torque is determined based on the total torque and the motor drive efficiency adapted to the motor drive direction. Based on the first mapping relationship between the braking torque, output torque and total torque, and the second mapping relationship between the total torque and motion drive information, the output torque expression is determined. The servo parameters to be optimized are determined based on the output torque expression, and the initial servo parameters are the initialized servo parameters to be optimized.

6. The method according to claim 5, characterized in that, The step of determining the motor drive direction by combining the target joint angle, the current joint angle, and the current joint angular velocity includes: Obtain the joint angle difference between the current joint angle and the target joint angle; When the first numerical sign of the joint angle difference is the same as the second numerical sign of the current joint angular velocity, the positive driving direction is determined as the motor driving direction; When the first numerical sign of the joint angle difference is different from the second numerical sign of the current joint angular velocity, the reverse driving direction is determined as the motor driving direction.

7. The method according to claim 5, characterized in that, Before determining the output torque expression based on the first mapping relationship between the braking torque, output torque, and total torque, and the second mapping relationship between the total torque and motion drive information, the method further includes: The friction loss is obtained, and the friction loss includes at least one of viscous friction, Coulomb friction, hysteresis loss, eddy current loss and wind resistance loss; The friction loss, the braking torque, and the output torque are combined to form the total torque, thus obtaining the first mapping relationship.

8. The method according to claim 5, characterized in that, Before determining the output torque expression based on the first mapping relationship between the braking torque, output torque, and total torque, and the second mapping relationship between the total torque and motion drive information, the method further includes: The joint angle difference between the current joint angle and the target joint angle is mapped to the voltage applied by the motor to obtain a third mapping relationship; Obtain a fourth mapping relationship between the applied voltage, back electromotive force voltage, motor current, and motor resistance of the motor; Obtain the fifth mapping relationship between the total torque and the motor current; Based on the third, fourth, and fifth mapping relationships, a second mapping relationship between the total torque and the motion drive information is obtained.

9. The method according to claim 1, characterized in that, The motion prediction reward includes at least one of the following: motion instruction reward, gait imitation reward, anti-perturbation reward, and regularization reward; Wherein, the motion command reward represents the reward for the simulated robot's response to a motion command, the gait imitation reward represents the reward for the simulated robot's gait, and the disturbance resistance reward represents the reward for the simulated robot's disturbance resistance.

10. The method according to any one of claims 1 to 4, 9, characterized in that, When the action prediction reward includes a motion command reward, before training the control model to be trained based on the action prediction reward of the next motion state, the method further includes: Obtain the directional velocity reward, yaw rate reward, uprightness reward, and velocity matching reward corresponding to the next motion state, wherein the velocity matching reward refers to the linear velocity and angular velocity matching reward of the next motion state; The motion command reward is obtained by combining the directional speed reward, the yaw rate reward, the uprightness reward, and the speed matching reward.

11. The method according to any one of claims 1 to 4, 9, characterized in that, When the action prediction reward includes a gait imitation reward, before training the control model to be trained based on the action prediction reward of the next motion state, the method further includes: Obtain the joint angle reward by comparing the next joint angle with the reference joint angle in the next motion state; Obtain the ground contact reward for the next motion state; Obtain the height matching reward between the base link height and the reference height in the next motion state; The gait imitation reward is obtained by combining the joint angle reward, the ground contact reward, and the height matching reward.

12. The method according to any one of claims 1 to 4, 9, characterized in that, When the action prediction reward includes a disturbance resistance reward, before training the control model to be trained based on the action prediction reward of the next motion state, the method further includes: Obtain the ground contact force in the next motion state; When the ground contact force is greater than the ground contact force threshold, a sliding penalty is determined based on the foot linear velocity in the next motion state; Obtain the anti-disturbance reward that is negatively correlated with the sliding penalty.

13. The method according to any one of claims 1 to 4, 9, characterized in that, When the action prediction reward includes a regularization reward, before training the control model to be trained based on the action prediction reward of the next motion state, the method further includes: The energy consumption penalty, ground contact force penalty, motion difference penalty, joint angular velocity penalty, and angular acceleration penalty for the next motion state are obtained. The regularized reward is obtained by combining the energy consumption penalty, the ground contact force penalty, the motion difference penalty, the joint angular velocity penalty, and the angular acceleration penalty.

14. The method according to any one of claims 1 to 4, 9, characterized in that, Before controlling the simulated robot to move based on the next motion action to obtain the next motion state, the method further includes: Based on the dimensions of the simulated robot, the terrain parameters to be trained are determined; Construct the training terrain based on the aforementioned terrain parameters; The process of controlling the simulated robot to move based on the next motion action to obtain the next motion state includes: Based on the next motion action, the simulated robot is controlled to move on the training terrain to obtain the next motion state.

15. The method according to any one of claims 1 to 4, 9, characterized in that, The process of training the control model to be trained based on the action prediction reward of the next motion state to obtain the robot control model includes: By combining the motion state to be processed, the predicted reward of the next motion state, and random disturbances, the decision index of the control model to be trained is calculated. The random disturbances include at least one of external disturbance force, external torque, system delay, ground friction, and power supply load. The robot control model is obtained by training the control model to be trained based on the decision index.

16. The method according to any one of claims 1 to 4, 9, characterized in that, After training the control model to be trained based on the action prediction reward of the next motion state to obtain the robot control model, the method further includes: Deploy the robot control model onto the real robot; In response to motion control commands, the target motion action is determined based on the robot control model; The robot is controlled to move on the real terrain based on the target motion.

17. A robot control device, characterized in that, The robot control device includes: The domain randomization module is used to perform domain randomization on the current motion state of the simulated robot based on the motion difference between the simulated robot and the real robot, so as to obtain the motion state to be processed. The simulated robot is the simulation result of the real robot. The motion prediction module is used to predict the motion state to be processed using the control model to be trained, so as to obtain the next motion action. The motion control module is used to control the simulated robot to move based on the next motion action, and to obtain the next motion state; The model training module is used to train the control model to be trained based on the action prediction reward of the next motion state to obtain a robot control model, which is used to control the motion of the real robot.

18. An electronic device for robot control, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the robot control method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the robot control method according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the robot control method according to any one of claims 1 to 16 is implemented.

Images