Robot control method and apparatus, device, storage medium and program product

Abstract

A robot control method and apparatus, a device, a storage medium and a program product. The robot control method comprises: acquiring first state data and reference action data of a robot corresponding to each of at least two moments (210); under a control policy, and on the basis of first observation data at the t-th moment of the at least two moments and reference action data at the t-th moment, predicting predicted action data of the robot at the t-th moment (220); on the basis of the predicted action data at the t-th moment and first state data at the t-th moment, predicting the state of the robot at the k-th moment to obtain second state data at the k-th moment (230); and, on the basis of the second state data at the k-th moment and reference action data at the k-th moment, training the control policy to obtain a trained control policy (240).

Description

Robot control method, device, equipment, storage medium and program product

[0001] This application claims priority to Chinese patent application No. 202410142136.X, filed on January 31, 2024, entitled “Robot control method, device, equipment, storage medium and program product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of machine learning, and in particular to a robot control method, apparatus, device, storage medium, and program product. Background Art

[0003] With the continuous development of robotics technology, the functions of robots are becoming increasingly powerful. Different types of robots can cope with various working environments and perform different operating tasks according to operating instructions.

[0004] In related technologies, a world model that simulates and predicts the physical world is usually used to provide a simulation environment for the robot, allowing the robot to learn and experiment in an internally generated, simulated environment. This can improve the robot's learning efficiency and safety without direct contact with the physical world, and can also have higher motor performance in the physical world.

[0005] In the above process, although the robot can learn efficiently with the help of the world model, when the robot needs to perform high-precision tasks (such as accurately imitating animals, moving precisely on a specified route, etc.), the simulation characteristics of the world model make it impossible to fully utilize the advantages of the physical world when training the robot solely through the world model, thereby reducing the accuracy of the robot in performing high-precision tasks. Summary of the Invention

[0006] The present invention provides a robot control method, apparatus, device, storage medium, and program product that fully utilize the robot's physical world to implement an accurate supervised learning process. This method uses a trained control strategy to control the robot's movements, enabling the robot to operate with greater accuracy and stability. The technical solution is as follows.

[0007] In one aspect, a robot control method is provided, the method comprising:

[0008] Obtaining first state data and reference motion data corresponding to the robot at at least two moments, respectively, where the first state data is data converted from first observation data, the first observation data is data collected by sensors in an operating environment of the robot, and the reference motion data is used to represent an expected posture of the robot in the operating environment;

[0009] Under the control strategy, based on the first observation data at the tth moment of the at least two moments and the reference motion data at the tth moment, predicting the predicted motion data of the robot at the tth moment, the control strategy being used to guide the motion of the robot, where t is a positive number;

[0010] Based on the predicted motion data at the t-th time and the first state data at the t-th time, predicting the state of the robot at the k-th time to obtain second state data at the k-th time, where the k-th time is a time that is later than the t-th time among the at least two time points;

[0011] Based on the second state data at the k-th moment and the reference action data at the k-th moment, the control strategy is trained to obtain a trained control strategy; wherein the trained control strategy is used to control the action of the robot.

[0012] In another aspect, a robot control device is provided, comprising:

[0013] a data acquisition module, configured to acquire first state data and reference motion data corresponding to the robot at at least two moments, wherein the first state data is data converted from first observation data, the first observation data is data collected by sensors within the robot's operating environment, and the reference motion data is used to represent an expected posture of the robot within the operating environment;

[0014] an action prediction module, configured to predict predicted action data of the robot at time t based on the first observation data at time t of the at least two time moments and the reference action data at time t under a control strategy, wherein the control strategy is used to guide the action of the robot, and t is a positive number;

[0015] a state prediction module, configured to predict the state of the robot at a kth moment based on the predicted motion data at the tth moment and the first state data at the tth moment, to obtain second state data at the kth moment, wherein the kth moment is a moment that is later than the tth moment among the at least two moments;

[0016] A strategy training module is used to train the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy; wherein the trained control strategy is used to control the action of the robot.

[0017] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the robot control method as described in any of the above-mentioned embodiments of the present application.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a robot control method as described in any of the above-mentioned embodiments of the present application.

[0019] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robot control method described in any one of the above embodiments.

[0020] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0021] After acquiring first state data and reference motion data corresponding to at least two time instants, the robot's predicted motion data at time t is predicted under the control strategy. Furthermore, based on the predicted motion data and the first state data, the robot's state at time k is predicted and second state data is obtained. The trained control strategy is obtained by training the robot using the second state data at time k and the reference motion data at time k. The collected first observation data allows for accurate supervised learning based on the robot's physical world operation. Within the constraints of the reference motion data, the control strategy obtains predicted motion data, enabling the prediction of second state data at a subsequent time instant using the predicted motion data and the first state data. The control strategy is then adjusted using the second state data and the reference motion data, enabling targeted adjustments to be made to the trained control strategy, thereby obtaining more accurate predicted motion data and improving the stability and application accuracy of the trained control strategy. This allows the robot to achieve more accurate motion accuracy and more stable operation during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0023] FIG2 is a flow chart of a robot control method provided by an exemplary embodiment of the present application;

[0024] FIG3 is a flow chart of a robot control method provided by another exemplary embodiment of the present application;

[0025] FIG4 is a schematic diagram of a world model obtained through training provided by an exemplary embodiment of the present application;

[0026] FIG5 is a flowchart of a robot control method provided by another exemplary embodiment of the present application;

[0027] FIG6 is a schematic diagram of a training control strategy provided by an exemplary embodiment of the present application;

[0028] FIG7 is a flowchart of a robot control method provided by another exemplary embodiment of the present application;

[0029] FIG8 is a schematic diagram of a training control strategy provided by yet another exemplary embodiment of the present application;

[0030] FIG9 is a schematic diagram showing a comparison of losses between a robot control method provided by an exemplary embodiment of the present application and related technologies;

[0031] FIG10 is a schematic diagram showing a comparison of losses after fine-tuning provided by an exemplary embodiment of the present application;

[0032] FIG11 is a schematic diagram of four trajectories provided by an exemplary embodiment of the present application;

[0033] FIG12 is a schematic diagram showing a comparison of following losses under different loads provided by an exemplary embodiment of the present application;

[0034] FIG13 is a schematic diagram showing a comparison of following losses at different speeds provided by an exemplary embodiment of the present application;

[0035] FIG14 is a schematic diagram showing a speed comparison of different iteration times provided by an exemplary embodiment of the present application;

[0036] FIG15 is a schematic diagram showing a speed comparison of different control strategies provided by an exemplary embodiment of the present application;

[0037] FIG16 is a comparative schematic diagram of trajectory following corresponding to four different trajectories provided by an exemplary embodiment of the present application;

[0038] FIG17 is a structural block diagram of a robot control device provided by an exemplary embodiment of the present application;

[0039] FIG18 is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0041] Robots: A robot is a mechanical or virtual device that performs tasks automatically, usually designed to complete specific human tasks or perform specific functions. Robots can have the ability to perceive, make decisions, and execute, enabling them to interact with the environment and complete complex tasks. Robots generally have multiple capabilities, such as learning ability (usually achieved through machine learning and deep learning technologies), a certain degree of autonomy, environmental perception (achieved through sensors such as cameras, lidar, and sonar), decision-making ability based on perceived information (such as path planning, target recognition, and task prioritization), task execution ability, and interactive ability. As a versatile engineering system, robots are widely used in many fields, including industry, medicine, services, and exploration. With the advancement of technology, the application of robots will continue to evolve.

[0042] In an embodiment of the present application, a robot control method is introduced that can fully utilize the robot's operation in the physical world to achieve an accurate supervised learning process. The control strategy is targetedly trained through predicted second state data and reference motion data obtained based on first observation data, thereby controlling the robot's motion with the help of the trained control strategy, enabling the robot to have more accurate motion accuracy and more stable operation capabilities during operation. The robot control method provided in the embodiment of the present application can be applied to robot types such as quadruped robots, push robots, manipulator robots, wheeled robots, and can also be applied to various scenarios such as industrial fields, medical fields, service fields, and exploration fields, and the embodiment of the present application is not limited to this.

[0043] In some embodiments, the robot control method is applied to a robotic arm robot in the industrial field as an example for illustration.

[0044] Schematically, robotic arm robots offer flexibility, high precision, and programmability, making them widely applicable in the industrial field, such as performing high-precision assembly and assembly tasks, high-risk welding tasks, high-intensity handling and loading and unloading tasks, and high-quality packaging tasks. To maximize the motion accuracy of robotic arm robots, various types of sensors deployed on the robotic arm robot can be used to collect first observation data of the robotic arm robot within its operating environment. State processing is then performed on the first observation data to obtain first state data. Under a control strategy, predicted motion data of the robotic arm robot can be predicted based on the first observation data and reference motion data. The state of the robotic arm robot at a later time is then predicted based on the predicted motion data and the first state data to obtain second state data. A control strategy is then trained based on the second state data and the reference motion data to obtain a trained control strategy. The trained control strategy can more accurately control the motion of the robotic arm robot, enabling the robotic arm robot to perform various motion types, such as linear motion, rotational motion, arc motion, joint motion, and grasping and releasing motion, according to the needs of the user, providing more efficient operation capabilities in the industrial field and improving industrial operation efficiency.

[0045] In some embodiments, the robot control method is applied to a quadruped robot in the service field as an example for illustration.

[0046] Schematically, the quadruped robot is implemented as a robot dog. In service scenarios, including emotional companionship services, various types of sensors deployed on the robot dog can collect first observation data within its operating environment. State processing is then performed on the first observation data to obtain first state data. This first state data can effectively avoid the influence of acquisition errors or noise, and more comprehensively present the robot dog's operating status. Under a control strategy, the robot dog's predicted motion data can be predicted based on the first observation data and reference motion data. The robot dog's state at a later time is then predicted based on the predicted motion data and the first state data to obtain second state data. A control strategy is then trained based on the second state data and the reference motion data, resulting in a trained control strategy. The trained control strategy can be used to control the robot dog's movements, enabling it to provide emotional companionship to the user through actions such as moving, jumping, running, and sitting, according to the user's needs, thereby improving the user's quality of life.

[0047] It is worth noting that the above application scenarios are only illustrative examples, and the robot types and robot application fields can be combined arbitrarily and are not limited here.

[0048] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first observation data, first state data, reference action data and other contents involved in this application are all obtained with full authorization.

[0049] Next, the implementation environment involved in the embodiments of this application is described. The robot control method provided in the embodiments of this application can be implemented by a robot alone or by the robot and a server through data interaction, and this embodiment of the application is not limited to this. Optionally, the robot and server interacting to implement the robot control method is used as an example for description.

[0050] Schematically, please refer to FIG. 1 , the implementation environment involves a robot 110 and a server 120 , and the robot 110 and the server 120 are connected via a communication network 130 .

[0051] In some embodiments, the robot 110 has a data acquisition function. For example, the robot 110 is equipped with a variety of sensors, such as an inertial measurement unit (IMU), a visual sensor, an infrared sensor, a contact sensor, a pressure sensor, a temperature sensor, a force / torque sensor, an angle sensor, an encoder for measuring joint angle and position, and an optical sensor.

[0052] Illustratively, first observation data can be collected using various sensors deployed on the robot 110. Specifically, the first observation data is data collected by the sensors within the operating environment of the robot 110. For example, the first observation data includes linear velocity and angular velocity collected by an IMU deployed on the robot 110, and joint positions and joint velocities collected by encoders.

[0053] Optionally, taking the example of collecting the corresponding first observation data at at least two moments, reference motion data corresponding to at least two moments can also be obtained to represent the expected posture of the robot 110 in the operating environment at the corresponding moments.

[0054] In some embodiments, in order to more systematically express the system state of the robot 110 in the operating environment, state processing is performed on the first observation data to obtain the first state data at the corresponding moment, such as: state processing is performed on the first observation data at the tth moment through the state processing method to obtain the first state data at the tth moment, where t is a positive number.

[0055] Optionally, the robot 110 sends the first state data corresponding to at least two moments and the reference action data corresponding to at least two moments to the server 120 through the communication network 130, so that the server 120 obtains the first state data and reference action data corresponding to the robot 110 at at least two moments.

[0056] Illustratively, the server 120 may obtain the first observation data through reverse reasoning based on the first state data; the server 120 may also receive the first observation data sent by the robot 110 through the communication network 130, etc.

[0057] In some embodiments, under the control strategy, the server 120 predicts the predicted motion data of the robot 110 at time t based on the first observation data at time t and the reference motion data at time t in at least two moments.

[0058] Schematically, the control strategy is a pre-set control strategy, which is used to obtain first observation data and determined reference action data based on the sensor collection on the robot 110, and predict the action that the robot needs to perform at the current moment, that is, to obtain predicted action data.

[0059] In some embodiments, the server 120 predicts the state of the robot 110 at the kth moment based on the predicted motion data and the first state data to obtain the second state data.

[0060] The kth moment is a moment that is later than the tth moment among at least two moments.

[0061] Illustratively, after determining the predicted state data at time t and the first state data at time t, the server can predict the state of the robot 110 at time k after time t to obtain the second state data at time k. In other words, the second state data is the predicted state result at time k.

[0062] In some embodiments, the server 120 trains the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy.

[0063] The trained control strategy is used to control the actions of the robot 110 .

[0064] Optionally, the server 120 sends the trained control strategy to the robot 110 through the communication network 130 so that the robot 110 controls its own actions based on the trained control strategy.

[0065] It is worth noting that the above-mentioned robots include but are not limited to quadruped robots, push robots, robotic arm robots, wheeled robots, etc.; the above-mentioned servers can be independent physical servers, or server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0066] Cloud technology refers to a hosting technology that unifies hardware, applications, networks, and other resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand.

[0067] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0068] In combination with the above-mentioned term introduction and application scenarios, the robot control method provided in this application is explained. The method is executed by a computing device. Optionally, the computing device can be implemented as a robot, a server, a terminal device, etc. In the embodiment of this application, the method is applied to a server as an example for explanation. As shown in Figure 2, the method includes the following steps 210 to 240.

[0069] Step 210: Obtain first state data and reference motion data corresponding to the robot at at least two moments.

[0070] The first state data is data converted from the first observation data, and the first observation data is data collected by the robot through sensors in the operating environment.

[0071] Schematically, a robot is equipped with a variety of sensors. These sensors can be deployed on joints (such as elbows, knees, and shoulders), end effectors (such as the end of a robotic arm), and various other parts of the robot, including the head, torso, and touch points (such as fingers and toes). The sensor deployment location typically depends on the robot's design, mission, and application requirements.

[0072] Optionally, at least one of the following sensors can be deployed on the robot: inertial measurement unit, visual sensor, infrared sensor, contact sensor, pressure sensor, temperature sensor, optical sensor, angle sensor, force / torque sensor, encoder for measuring joint angle and position, etc.

[0073] Among them, the inertial measurement unit, also known as the inertial sensor, is usually composed of a gyroscope and an accelerometer. It is usually deployed on the joints of the robot. The gyroscope is used to measure angular velocity, and the accelerometer is used to measure linear acceleration. The linear velocity can be obtained by integrating the linear acceleration.

[0074] Visual sensors such as cameras are used to capture images or video data for object recognition, scene interpretation, navigation, and finding moving objects; infrared sensors are used to detect infrared light (thermal radiation) to measure temperature (contactless thermometers), detect organisms or heat sources, etc.; contact sensors are used to detect physical contact to determine whether robot parts touch objects or surfaces; pressure sensors are used to measure the pressure of gas or liquid to ensure that the grasped objects are not damaged; temperature sensors are used to measure the temperature of the environment or objects to maintain thermal conditions and avoid overheating; optical sensors use light to detect objects, measure distances, or perceive the environment to achieve accurate distance measurement and environmental scanning; angle sensors are used to measure rotation angles or positions to precisely control the angles of robot joints and rotating parts; force / torque sensors are used to measure force and torque (rotational force), which are particularly important in robot arms and actuators, and can help robots perform tasks with appropriate force, such as precision assembly or object handling.

[0075] Encoders are used to measure joint angles and positions, thereby providing high-precision angle information and position information, which is crucial for precise motion control of robots. Encoders can be incremental (providing relative position changes) or absolute (providing absolute position information).

[0076] The above sensor types are only illustrative. Each sensor has its specific uses and advantages. Typically, in complex robotic systems, multiple different types of sensors are used in combination to achieve higher functionality and adaptability, thereby achieving accurate motion control and perception.

[0077] In some embodiments, first observation data is collected by sensors deployed on the robot, and the first observation data is used to characterize the operating conditions of the robot in the operating environment.

[0078] Schematically, the first observation data includes at least one of the data collected by sensors such as the robot, linear velocity (such as the value obtained by integrating the joint linear velocities corresponding to multiple joints on the robot), robot angular velocity (such as the value obtained by integrating the joint angular velocities corresponding to multiple joints on the robot), joint position (the positions corresponding to multiple joints on the robot), and joint speed (the movement speed corresponding to multiple joints on the robot).

[0079] Optionally, at least two moments are moments within a historical time period, and at least two moments each correspond to first observation data. The first observation data collected at different moments are used to represent data collected by the robot via sensors at the current moment. For example, first observation data G1 is collected at moment 1, and first observation data G1 is data collected by the robot via multiple sensors deployed on the robot at moment 1; first observation data G2 is collected at moment 2, and first observation data G2 is data collected by the robot via multiple sensors deployed on the robot at moment 2, etc.

[0080] In some embodiments, the first observation data is converted into the first state data through state processing, that is, the first state data is obtained after the state processing is performed on the first observation data.

[0081] Illustratively, a state processing method is employed to perform state processing on the first observation data to obtain the first state data. Optionally, the state processing method is a pre-defined data conversion method for performing a comprehensive analysis on the first observation data collected by the sensor at the current moment to extract the first state data representing the robot's overall state relative to the operating environment.

[0082] For example: the first observation data is data directly collected by the robot, which may contain a lot of noise and redundant information, making it difficult to accurately estimate the robot as a whole. At least one of the multiple technologies such as filtering technology, weight adjustment technology, data model mapping technology, etc. is used as a state processing method to perform state processing on the first observation data, so that the first state data can better reflect the important information in the first observation data and filter out unnecessary noise information. While simplifying the analysis, it reduces the difficulty of understanding the robot and improves the analysis efficiency of the robot, which facilitates better understanding and control of the robot's behavior.

[0083] Optionally, a state processing method for implementing state processing is illustrated as follows.

[0084] (1) Filtering algorithm: Using a filtering algorithm (such as a Kalman filter, etc.) as a state processing method, a relatively pure robot operation state can be extracted from the first observation data measured by the sensor, while reducing the influence of noise, that is, obtaining the first state data.

[0085] (2) Mathematical model: If the dynamic behavior of the robot can be represented by a mathematical model, the mathematical model can be used as a state processing method to map the first observation data obtained by the sensor measurement into the first state data through the mathematical model, where the data model may involve mathematical tools such as differential equations and integration.

[0086] (3) Feature extraction: Use the feature extraction network as the state processing method to extract the features of the sensor data with the feature extraction network to identify the key features of the robot state and obtain the first state data. This process can be achieved through technologies such as signal processing and pattern recognition.

[0087] (4) Machine learning: Machine learning algorithms can be used as state processing methods. Pre-trained machine learning models can be used to learn the mapping relationship between the robot's corresponding observation data and state data from sensor data, so that the first state data can be obtained based on the collected first observation data. This process is more useful for nonlinear and complex systems.

[0088] State processing is a key step in mapping the first observation data collected by the sensor to the robot state (first state data), which helps understand and control the behavior of the robot or other automated system. It is worth noting that the above state processing method is only an illustrative example and is not limited to this embodiment of the present application.

[0089] Optionally, the first state data includes at least one of robot position, robot direction, robot linear velocity, robot angular velocity, joint position and joint velocity; wherein, the first state data is obtained after performing state processing on the first observation data, such as: the robot includes multiple joints, and the first state data includes the joint position P corresponding to the A1 joint among the multiple joints. The joint position P is obtained after adjusting the joint position p corresponding to the A joint under the first observation data, and comprehensively considering other data such as the joint positions corresponding to other joints under the first observation data, the robot linear velocity, the robot angular velocity, etc.

[0090] In some embodiments, when at least two moments correspond to first observation data respectively, first state data corresponding to at least two moments can be obtained by performing state processing on the first observation data corresponding to at least two moments respectively.

[0091] For example: the first observation data G1 is collected at time 1, and after state processing is performed on the first observation data G1, the first state data Z1 corresponding to time 1 is obtained; the first observation data G2 is collected at time 2, and after state processing is performed on the first observation data G2, the first state data Z2 corresponding to time 2 is obtained, and so on.

[0092] The reference motion data is used to represent the expected posture of the robot in the operating environment. The expected posture refers to the position and posture that the robot is expected to achieve in the operating environment.

[0093] Illustratively, at least two moments each correspond to a reference motion data set. Each reference motion data set includes at least one of various motion postures, such as a robot motion (representing the robot's overall motion, such as standing, crawling, or raising its leg), joint motions (such as joint A bending 45°, joint B bending 90°, etc.). By combining the reference motion data sets corresponding to at least two moments, a reference motion sequence can be generated. For example, a reference motion sequence is (Q1, Q2, Q3, ...), where moment 1 corresponds to reference motion data Q1, moment 2 corresponds to reference motion data Q2, and so on. The reference motion sequence can be used to describe the movements a robot should take in a specific task or environment.

[0094] Optionally, a reference action sequence is obtained, and reference action data corresponding to at least two moments are obtained from the reference action sequence based on at least two moments, each reference action data corresponds to one moment, and the expected postures represented by multiple reference action data are combined. Multiple expected postures can be executed continuously to achieve the reference action.

[0095] Schematically, the reference action sequence is implemented as at least one data set representing continuous actions, such as a teaching motion sequence, a simulation-generated trajectory, a learning algorithm-generated action, and human motion capture.

[0096] A teachable motion sequence is typically a series of motions performed by a human operator, an animal, or other similar user. These motion trajectories can be used to represent the desired posture of the robot in similar situations. For example, if a robotic arm needs to grasp an object in space, a human operator can demonstrate manual grasping by manually operating the robotic arm. The recorded grasping trajectory serves as a reference motion sequence, where at least two moments correspond to a reference motion data set.

[0097] Simulation-generated trajectories are robot motion trajectories generated using physical simulations or motion planning algorithms. These trajectories can serve as reference action sequences for performing similar tasks in real-world environments. For example, by planning a path for a robot to move to a target location in simulation, the generated path can be used as a reference action sequence for actual robot navigation.

[0098] Learning algorithms to generate actions, such as machine learning methods like reinforcement learning, can be used to allow robots to learn through trial and error to generate reference action sequences. In this case, the robot may form an optimized reference action sequence by repeatedly trying in the environment and adjusting its action strategy based on feedback.

[0099] Human motion capture: For example, when a robot's task involves collaborating with humans or imitating human movements, human motion capture technology can be used to record the motion trajectory generated by humans when performing tasks, and convert the motion trajectory into a reference motion sequence for the robot, in order to achieve the desired posture corresponding to different moments through multiple reference motion data.

[0100] That is, with the help of reference motion data, robots can learn how to adjust their posture and perform actions in different scenarios, which can help robots better perform tasks in the physical world and improve their adaptability and flexibility.

[0101] Step 220 , under the control strategy, based on the first observation data at the tth moment and the reference motion data at the tth moment in at least two moments, predict the predicted motion data of the robot at the tth moment.

[0102] Where t is a positive number.

[0103] Schematically, the control strategy is a pre-set control strategy. The control strategy refers to the rules, algorithms or methods used to guide and regulate the behavior of the robot. The control strategy can predict the robot's movement at the current moment based on the first observation data and reference action data obtained at any moment, so as to perform appropriate motion behavior according to the first observation data as much as possible within the limitation of the reference action data.

[0104] Optionally, the control strategy is implemented as at least one of a variety of algorithms such as the A* algorithm, the D* algorithm, the Proportional, Integral, Differential (PID) algorithm, etc.; it can also be implemented as a model predictive control method (such as predicting future states through a data model); it can also be implemented as a reinforcement learning strategy (a strategy used to optimize robot behavior through trial and error learning, which can adjust its actions through reward signals in the interaction between the robot and the environment, thereby gradually learning the optimal strategy), such as: Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) are widely used as network implementation methods of reinforcement learning strategies.

[0105] Indicatively, the tth moment is any moment except the last moment among at least two moments; taking the prediction of the predicted action data at the tth moment by the first observation data at the tth moment and the reference action data at the tth moment as an example, the first observation data at the tth moment and the reference action data at the tth moment are used as independent variables of the control strategy, that is: the first observation data at the tth moment and the reference action data at the tth moment are substituted or input into the control strategy to predict the predicted action data of the robot at the tth moment.

[0106] That is, the predicted action data is the result of the control strategy output, which is used to predict the robot's action at the current moment.

[0107] Step 230 , based on the predicted action data at the t-th moment and the first state data at the t-th moment, predict the state of the robot at the k-th moment, and obtain the second state data at the k-th moment.

[0108] The kth moment is a moment that is later than the tth moment among at least two moments.

[0109] Schematically, after obtaining the predicted action data at the tth moment and the first state data at the tth moment, the predicted action data at the tth moment is used to predict how the robot will change its state in a short time to obtain the predicted state change. This process involves integrating the motion equation or using a mathematical model for deduction; then, the first state data at the tth moment is combined with the predicted state change to achieve the purpose of transitioning from the first state data, and obtain the second state data at the kth moment.

[0110] Step 240 : training the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy.

[0111] Illustratively, the kth moment is used as one of the at least two moments, and when obtaining the reference action data corresponding to the at least two moments, the reference action data at the kth moment is included.

[0112] After predicting the second state data at the kth moment, based on the collected first observation data at the kth moment, it can be determined that the reference action data at the kth moment has a strong purpose. Therefore, the predicted second state data at the kth moment and the reference action data at the kth moment can be compared to determine the difference between the second state data at the kth moment and the reference action data at the kth moment.

[0113] Optionally, taking supervised learning as an example, a loss value can be determined by the reference action data and the predicted second state data. The loss value is used to indicate the difference between the reference action data and the predicted second state data. The loss value can be calculated by the mean squared error (MSE), or by the cross-entropy loss (Cross-Entropy Loss), or by a custom loss function. The choice of the relevant loss function depends on the nature of the task and the definition of the problem, and is not specifically limited here.

[0114] In some embodiments, the control policy is trained using the loss value to obtain a trained control policy.

[0115] Schematically, the control strategy is implemented as an algorithm, which includes multiple algorithm parameters. When the control strategy is trained through the loss value, the algorithm parameters are changed to achieve the purpose of optimizing the control strategy.

[0116] Optionally, the rate of change of the loss function relative to the algorithm parameters when determining the loss value is calculated to obtain the gradient value, and then an optimization algorithm (such as a gradient descent algorithm) is used to update the algorithm parameters of the control strategy according to the gradient value to reduce the loss value until the number of training times is reached or the loss value drops to a preset threshold, thereby obtaining a trained control strategy.

[0117] Among them, the trained control strategy is used to control the robot's movements.

[0118] Optionally, after obtaining the trained control strategy, it is possible to predict the robot's predicted motion data at the current moment based on the first observation data at any moment and the reference motion data at that moment under the trained control strategy, so that the robot can execute the motion process at the current moment through the predicted motion data.

[0119] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0120] To sum up, with the help of the collected first observation data, we can make full use of the robot's operation in the physical world to realize an accurate supervised learning process. Under the limitation of the reference action data, we can obtain predicted action data through the control strategy, so that we can use the predicted action data and the second state data at the moment after the first state data is predicted. Through the second state data and the reference action data, we can make targeted adjustments to the control strategy, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable operation capabilities during operation.

[0121] In an optional embodiment, a world model is used to perform state prediction based on the predicted action data and the first state data to obtain the second state data. The world model is trained based on the first state data and the first observation data. As shown in FIG3 , the embodiment shown in FIG2 can also be implemented as steps 310 to 350 as follows; step 230 can also be implemented as steps 330 to 340 as follows.

[0122] Step 310: Obtain first state data and reference motion data corresponding to the robot at at least two moments.

[0123] Among them, the first state data is the data obtained after state processing is performed on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment; the reference action data is used to represent the expected posture of the robot in the operating environment.

[0124] Illustratively, a robot is equipped with multiple sensors. When the robot operates in its operating environment, the multiple sensors are in operation and can collect data in real time or periodically to obtain first observation data corresponding to at least two moments in time. For example, if data is collected at a periodic interval of one second, then the at least two moments in time represent multiple seconds, with one first observation data corresponding to each second. The first observation data includes at least one of the data collected by sensors such as the robot, linear velocity, angular velocity, joint position, and joint velocity.

[0125] Optionally, after state processing is performed on the first observation data corresponding to at least two moments respectively, first state data corresponding to the at least two moments respectively are obtained.

[0126] For example: the first observation data G1 is collected at time 1, and after state processing is performed on the first observation data G1, the first state data Z1 corresponding to time 1 is obtained; the first observation data G2 is collected at time 2, and after state processing is performed on the first observation data G2, the first state data Z2 corresponding to time 2 is obtained, and so on.

[0127] Illustratively, the first state data includes at least one of robot position, robot orientation, robot linear velocity, robot angular velocity, joint position, and joint velocity.

[0128] The reference motion data is used to represent the expected posture of the robot in the operating environment.

[0129] In some embodiments, first observation data collected by the robot at at least two moments are obtained; state processing is performed on the first observation data corresponding to the at least two moments to obtain first state data corresponding to the at least two moments.

[0130] In some embodiments, a reference action sequence is obtained.

[0131] The reference motion sequence represents the expected posture sequence of the robot within its operating environment. The reference motion sequence includes reference motion data corresponding to at least two moments in time. Specifically, each of the at least two moments in time corresponds to a piece of reference motion data. Multiple reference motion data constitute a reference motion sequence, representing the expected posture changes of the robot at at least two moments in time.

[0132] Step 320 , under the control strategy, based on the first observation data at the tth moment and the reference motion data at the tth moment in at least two moments, predict the predicted motion data of the robot at the tth moment.

[0133] The t-th moment is any moment other than the last moment among at least two moments, and t is a positive number.

[0134] Schematically, a control strategy is a pre-set strategy that is used to guide the actions of the robot.

[0135] Optionally, the control strategy can be implemented as an algorithm to guide and regulate the behavior of the robot; under the control strategy, the first observation data at the tth moment and the reference action data at the tth moment can be substituted into the control strategy to calculate the predicted action data of the robot at the tth moment.

[0136] Optionally, the control strategy can be implemented as a machine learning model that guides and regulates the behavior of the robot; under the control strategy, the first observation data at the tth moment and the reference action data at the tth moment can be input into the control strategy to learn the deep information therein through the machine learning model, and predict the predicted action data of the robot at the tth moment, etc.

[0137] Step 330: Obtain a world model.

[0138] The world model is used to predict the state of the robot, and the world model is a model trained based on the first state data and the first observation data.

[0139] Alternatively, think of the world model as a component or application deployed on the robot; in robotics, a world model is typically designed as an internal representation used to simulate and understand the robot's physical environment.

[0140] Schematically, the world model is the robot's internal representation of its operating environment, usually an abstract expression of the physical world. It is constructed by selectively capturing information related to the robot's task execution. As a generative model that attempts to learn the interaction between the robot and its operating environment, the world model is the basis for the robot to understand and act in the physical world. It is a dynamic, constantly updated model.

[0141] Optionally, the world model is used to estimate the state of the robot in the operating environment so that the robot can predict future states, for example, predicting the robot's position, speed changes, or other behaviors. In addition, based on the state prediction process of the world model, the robot can also implement task planning such as path planning and motion planning, that is, determining the state required to reach a specific position.

[0142] Among them, the world model can be continuously updated so that the robot can learn new knowledge from experience, constantly adapt to changes in the environment and improve performance; this is conducive to the robot's ability to understand and interact with complex environments, and through the improvement of complexity and accuracy, it can improve the robot's ability to perform tasks in the physical world.

[0143] In an optional embodiment, an environmental simulation model is obtained.

[0144] Among them, the environment simulation model is the model to be trained to obtain the world model.

[0145] In which, the environmental simulation model is used to predict the predicted state data at the jth moment based on the first state data at the i-th moment and the first observation data at the i-th moment, the i-th moment is a moment among at least two moments, the j-th moment is a moment after the i-th moment among at least two moments, i is a positive number, and j is a positive number.

[0146] Schematically, the environment simulation model can be regarded as an initialized world model, which has certain state prediction functions based on the model structure.

[0147] In some embodiments, the environment simulation model performs a state prediction process based on the first state data and the first observation data.

[0148] Schematically, the i-th moment is any one of at least two moments, and the j-th moment is a moment after the i-th moment among at least two moments. The first state data at the i-th moment is determined from the first state data corresponding to the at least two moments, and the first observation data at the i-th moment can also be determined from the first observation data corresponding to the at least two moments. Then, the first state data at the i-th moment and the first observation data at the i-th moment are input into the environmental simulation model, and the predicted state data at the j-th moment after the i-th moment is output.

[0149] In some embodiments, first motion data corresponding to at least two moments are acquired, and the first motion data are used to describe posture data generated when the robot moves in the motion environment.

[0150] Schematically, the first motion data includes at least one information describing the motion state of the robot, such as the position, direction, speed, acceleration, joint target angle, etc.; the first motion data is data collected by the motion sensor deployed by the robot.

[0151] At least two moments correspond to a first action data respectively, for example, moment 1 corresponds to the first action data D1, moment 2 corresponds to the first action data D2, and so on.

[0152] In some embodiments, the environment simulation model implements a state prediction process based on the first action data, the first state data, and the first observation data.

[0153] Illustratively, the environment simulation model predicts the predicted state data at the jth moment based on the first action data at the i-th moment, the first state data at the i-th moment, and the first observation data at the i-th moment.

[0154] Optionally, the environmental simulation model predicts the state change difference at the jth moment based on the first action data at the i-th moment and the first observation data at the i-th moment; the state change difference at the j-th moment is added to the first state data at the i-th moment to obtain the predicted state data at the j-th moment.

[0155] Schematically, as shown in Formula 1 below, it is a formula for obtaining predicted state data based on the environmental simulation model.

[0156] Formula 1:

[0157] Among them, s t is the first state data at time t; The predicted state data at time t+1 is obtained by predicting the state at time t+1 after time t; f w is the environmental simulation model; f w (δs t |o t ,a t ,π) is the neural network expression of the environmental simulation model, parameterized as θ w , express input w including o t and a t , o t represents the first observation data at time t, a t represents the first action data at time t, π represents the control strategy (in a fixed form); δst is the state change difference at the jth moment; therefore, f w (δs t |o t ,a t ,π) is expressed in the control strategy π by simulating the environment model f w Input the first observation data o at time t t And the first action data a at time t t , we can predict the state change difference δs at time t+1 t ; Then according to the state change difference δs at time t+1 t And the first state data at time t, obtain the predicted state data at time t+1.

[0158] In an optional embodiment, a predicted loss value is obtained based on the predicted state data at the jth moment and the first state data at the jth moment, and the predicted loss value is used to indicate the difference between the predicted state data at the jth moment and the first state data at the jth moment.

[0159] Schematically, the first state data at the jth moment is determined from the first state data corresponding to at least two moments, and the predicted state data at the jth moment is compared with the first state data at the jth moment obtained based on the first observation data to obtain the predicted loss value.

[0160] In some embodiments, based on the predicted state data at the jth moment and the first state data at the jth moment, a state loss value corresponding to the jth moment is obtained, and the state loss value is used to indicate the difference between the predicted state data at the jth moment and the first state data at the jth moment.

[0161] Optionally, in at least two moments, except for the first moment, corresponding predicted state data can be obtained through the prediction process, and at least two moments correspond to a first state data respectively. Therefore, based on the first state data at the jth moment in at least two moments and the predicted state data at the jth moment, the state loss values ​​corresponding to the at least two moments can be obtained.

[0162] In some embodiments, the difference between the predicted state data at the jth moment and the first state data at the jth moment is determined by a preset loss function, that is, the predicted state data at the jth moment and the first state data at the jth moment are substituted into the preset loss function to obtain the state loss value corresponding to the jth moment.

[0163] Optionally, the above-mentioned preset loss function can be implemented as at least one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, a minimum absolute deviation loss (Least Absolute Deviations Loss, L1 Loss) function, etc., which is not limited here.

[0164] Schematically, the first state data at the jth moment is determined from the first state data corresponding to at least two moments, and the difference between the predicted state data at the jth moment and the first state data at the jth moment obtained based on the first observation data is determined to obtain a predicted loss value representing the difference between the first state data at the jth moment and the first state data at the jth moment.

[0165] In some embodiments, the state loss values ​​corresponding to at least two moments are summed to obtain a predicted loss value.

[0166] Optionally, the state loss values ​​corresponding to at least two moments are determined according to the above process, and then the multiple state loss values ​​are summed to obtain the predicted loss value.

[0167] Schematically, as shown in the following formula 2, a formula for obtaining a predicted loss value by integrating the state loss values ​​corresponding to at least two moments.

[0168] Formula 2:

[0169] in, is the predicted loss value; n is the number of loss training steps, which can be regarded as the number of moments of at least two moments; is the predicted state data at time t; s t is the first state data at time t.

[0170] In an optional embodiment, the environment simulation model is trained by predicting the loss value to obtain a world model.

[0171] In an illustrative manner, the environment simulation model is trained for a preset number of times by predicting the loss value, thereby obtaining a world model; alternatively, the environment simulation model is trained for a preset number of times by predicting the loss value until the loss value no longer decreases, thereby obtaining a world model, etc.

[0172] As shown in Figure 4, it is a schematic diagram of the world model obtained through training.

[0173] The world model is a model obtained through training, and has the same network structure as the environment simulation model. Therefore, the environment simulation model before training can also be called the world model. The input of the world model 410 includes the state-action sequence τ={s0,a0,s1,a1,…,s n,a n}, which contains the first state data s t (such as s0 at time 0, s1 at time 1, etc.) and the first action data a t (such as a0 at time 0, a1 at time 1, etc.); also includes the first observation data o t (Such as o0 at time 0, o1 at time 1, etc.)

[0174] The state action prediction sequence is obtained by predicting the world model 410 It includes at least two moments corresponding to the predicted state data and the predicted action data; through the state action prediction sequence State-Action Sequence The predicted loss value may be calculated, and the world model 410 may be trained using the predicted loss value.

[0175] With the help of the state loss values ​​corresponding to at least two moments, the predicted loss value after comprehensive analysis of at least two moments is obtained, so that the world model can be centrally trained through the predicted loss value, thereby improving the training efficiency of the world model.

[0176] Step 340 , performing state prediction on the predicted action data at the t-th moment and the first state data at the t-th moment through the world model to obtain the second state data at the k-th moment.

[0177] The kth moment is a moment that is later than the tth moment among at least two moments.

[0178] Illustratively, after obtaining the predicted action data at time t and the first state data at time t, the state prediction process is performed using the world model. The world model is obtained after training the environment simulation model. Therefore, the world model and the environment simulation model have the same neural network structure, but the network parameters may differ due to training.

[0179] As shown in Formula 1 above, f w Can represent the world model obtained after training; f w (δs t |o t ,a t ,π) can represent the neural network expression of the world model. When the state prediction is performed at the kth moment through the world model, the predicted action data at the tth moment and the first action data at the tth moment can be obtained. By inputting the predicted action data at the tth moment and the first action data at the tth moment into the world model f wAfter that, the output is the state change difference from the tth moment to the kth moment, that is, the state change difference corresponding to the kth moment is obtained, and then the first state data at the tth moment and the state change difference corresponding to the kth moment are summed to obtain the second state data corresponding to the kth moment.

[0180] Step 350 : training the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy.

[0181] In some embodiments, taking the process of training the control strategy as supervised learning as an example, the purpose of supervised learning is to enable the robot to learn to map states to corresponding actions. Then the loss value is usually implemented as the loss between the reference action data and the predicted second state data, and the action represented by the predicted second state data is made as close as possible to the reference action represented by the reference action data.

[0182] Schematically, after predicting the second state data at the kth moment, based on the collected first observation data at the kth moment, it can be determined that the reference action data at the kth moment has a stronger purpose, so the predicted second state data at the kth moment and the reference action data at the kth moment can be compared to determine the difference between the second state data at the kth moment and the reference action data at the kth moment.

[0183] In some embodiments, taking reinforcement learning as an example of the process of training a control strategy, the purpose of reinforcement learning is to enable the robot to learn the value of taking a certain action in a certain state, then the loss value is usually realized as the difference between the acquired first state data and the predicted second state data.

[0184] Schematically, after predicting the second state data at the kth moment, based on the collected first observation data at the kth moment, it can be determined that the first state data at the kth moment has strong authenticity, so the predicted second state data at the kth moment and the first state data at the kth moment can also be compared to determine the difference between the second state data at the kth moment and the reference action data at the kth moment.

[0185] In an optional embodiment, the control strategy is trained using the loss value to obtain a trained control strategy.

[0186] In some embodiments, based on the difference between the second state data at the kth moment and the reference action data at the kth moment, the loss value corresponding to the kth moment is obtained; the strategy parameters in the control strategy are adjusted with the loss value to obtain the trained control strategy.

[0187] The difference between the second state data at the kth moment and the reference action data at the kth moment can be determined by a preset loss function corresponding to the stage, that is, the second state data at the kth moment and the reference action data at the kth moment are substituted into the preset loss function, and the loss value corresponding to the kth moment is output through the preset loss function.

[0188] Optionally, the above-mentioned specified loss function can be implemented as at least one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, a least absolute deviation loss (Least Absolute Deviations Loss, L1 Loss) function, etc., which is not limited here.

[0189] Optionally, loss values ​​corresponding to at least two moments are obtained; and strategy parameters in the control strategy are iteratively adjusted using the loss values ​​corresponding to at least two moments to obtain a trained control strategy.

[0190] Illustratively, the control strategy includes an encoder and a decoder, and the strategy parameters are implemented as network parameters corresponding to the encoder; and / or, the strategy parameters are implemented as network parameters corresponding to the decoder, etc.

[0191] Among them, the trained control strategy is used to control the robot's movements.

[0192] Schematically, after a given planned path, the robot can efficiently control its movements according to the trained control strategy. For example, the trained control strategy can more accurately obtain predicted motion data, thereby performing an action process based on the predicted motion data.

[0193] In an optional embodiment, the environment simulation model and control strategy are co-deployed on the robot to jointly control the robot and assist in its movement. Beyond deployment on the robot, the environment simulation model and control strategy can be collaboratively optimized. Through the trained world model and the trained control strategy, the robot can adapt to different environments and tasks, improving its autonomous decision-making and execution capabilities.

[0194] Optionally, the training processes corresponding to the above-mentioned environment simulation model and control strategy can be implemented as a sequential execution process or as an alternating execution process. The following content describes the training process.

[0195] (1) Sequential execution process

[0196] In schematic form, after obtaining the control strategy, while keeping the predictive control strategy unchanged, the world model is obtained by training the environment simulation model through the first state data and first observation data corresponding to at least two moments. Then, while keeping the world model unchanged, the control strategy is trained through the first state data, first observation data and reference action data corresponding to at least two moments, and finally the trained control strategy is obtained.

[0197] That is: first control the control strategy and train to obtain the world model; then control the world model unchanged and train to obtain the trained control strategy.

[0198] (2) Alternating execution process

[0199] Illustratively, at least two moments are divided into two moment groups, and the two moment groups include a first moment group and a second moment group.

[0200] After obtaining the control strategy, the environment simulation model is trained using the first state data and the first observation data corresponding to each moment in the first moment group of at least two moments while the predictive control strategy remains unchanged, thereby obtaining a simulation model after the first training. Furthermore, while the simulation model after the first training remains unchanged, the control strategy is trained for the first time using the first state data, the first observation data, and the reference action data corresponding to each moment in the first moment group, thereby obtaining a control strategy after the first training.

[0201] Afterwards, the simulation model after the first training is trained by using the first state data and the first observation data corresponding to each moment in the second moment group in at least two moments to obtain a world model. Then, while keeping the world model unchanged, the control strategy after the first training is trained for the second time using the first state data, the first observation data and the reference action data corresponding to each moment in the second moment group until the trained control strategy is obtained.

[0202] That is: first control the control strategy, and train the environment simulation model through the data at each moment in the first moment group; then control the simulation model after the first training to remain unchanged, and obtain the control strategy after the first training through the data training at each moment in the first moment group; then control the control strategy after the first training, and train the simulation model after the first training through the data at each moment in the second moment group; then control the simulation model after the first training to remain unchanged, and obtain the trained control strategy through the data training at each moment in the second moment group, etc.

[0203] The above division of at least two moment groups into two moment groups is only an illustrative example. The at least two moments can also be divided into more moment groups, so that the above alternating training process is performed through the data corresponding to the moments in multiple moment groups. The embodiments of the present application are not limited to this.

[0204] In some embodiments, the world model and the trained control strategy jointly assist the robot in the motion control process.

[0205] Schematically, the robot's software architecture integrates a world model and a trained control strategy. The world model provides information about the environment's state, such as obstacle locations and target positions. The trained control strategy leverages this information to more accurately generate the robot's actions, responding to the current environment and task requirements. This integration makes the robot more flexible and adaptive, enabling it to perform various tasks in complex and dynamic environments. Future data collection, reference motion data, and other data allow the world model and trained control strategy to be continuously learned and optimized, fully enhancing the robot's performance in the physical world.

[0206] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0207] To sum up, with the help of the collected first observation data, we can make full use of the robot's operation in the physical world to carry out an accurate supervised learning process. Under the limitation of the reference action data, we can obtain predicted action data through the control strategy, so that we can use the predicted action data and the second state data at the moment after the first state data is predicted. Through the second state data and the reference action data, we can make targeted adjustments to the control strategy, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable operation capabilities during operation.

[0208] In the embodiments of this application, the second state data is predicted based on a world model. The world model is a model trained based on the first state data and the first observation data. The trained world model is used as the state prediction model. This fully integrates the world model with the control strategy to cope with the complex environment during robot movement, improve the robot's adaptability and generalization performance, facilitate more efficient online planning and decision-making, and improve the robot's efficiency and performance.

[0209] In an optional embodiment, during the process of predicting the predicted state data, the first observation data at time t and the reference motion data at time t are mapped into a latent space to obtain a coded feature representation. Then, under the control strategy, the predicted motion data of the robot at time t is obtained based on the coded feature representation. Schematically, as shown in FIG5 , the embodiment shown in FIG2 can also be implemented as follows: steps 510 to 550; wherein, step 220 can also be implemented as follows: steps 520 to 530.

[0210] Step 510: Obtain first state data and reference motion data corresponding to the robot at at least two moments.

[0211] Among them, the first state data is the data obtained after state processing is performed on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment; the reference action data is used to represent the expected posture of the robot in the operating environment.

[0212] The content of step 510 has been described in the above steps 210 and 310 and will not be repeated here.

[0213] Step 520 : Under the control strategy, encode the first observation data at the t-th moment and the reference action data at the t-th moment to obtain an encoded feature representation.

[0214] Schematically, the purpose of obtaining reference motion data is to enable the robot to better imitate the motion sequence from real organisms. Therefore, the process of the robot imitating the reference motion data can be converted into an encoder-decoder architecture, that is, the execution process of the control strategy is realized through the encoder-decoder architecture.

[0215] Among them, the encoder is used to map the input data (such as the first observation data, reference action data, etc.) into the latent space for representation to capture the key features of the input data, and its output encoded feature representation contains an abstract expression of the input data.

[0216] Among them, the decoder is used to map the encoded feature representation output by the encoder back to the original data space to generate robot actions similar to the reference actions, that is, to output predicted action data for realizing the robot action.

[0217] In an optional embodiment, a first encoder performs a first encoding on the first observation data at the tth moment and the reference action data at the tth moment to obtain a first feature representation.

[0218] The first encoder is used to implement imitation learning of the robot's movements based on reference movement data.

[0219] Schematically, the first encoder is implemented as an Imitation Learning (IL) encoder. Imitation learning is a learning strategy that trains machine learning models by imitating expert behavior. Specifically, reference action data is obtained by collecting expert behavior. The reference action data is then used for imitation learning. The IL encoder attempts to imitate or replicate the behavior observed in the reference action data. The IL encoder is typically responsible for mapping the first observation data and the reference action data into a latent representation, such that the output generated by the latent representation is similar to the behavior demonstrated by the reference action data.

[0220] Optionally, when analyzing the first observation data at the tth moment and the reference action data at the tth moment, the first observation data at the tth moment and the reference action data at the tth moment are input into the first encoder so as to generate a first feature representation under the condition that the behavior represented by the reference action data at the tth moment is imitated by the first observation data at the tth moment, based on the reference action data at the tth moment.

[0221] In an optional embodiment, a second encoder is used to perform a second encoding on the first observation data at the t-th moment in the historical time period to obtain a second feature representation.

[0222] The second encoder is used to predict and analyze the robot's movements with the help of prior knowledge, and the prior knowledge is the knowledge learned during the training process of the second encoder.

[0223] Optionally, the prior knowledge includes at least one of task structure knowledge, domain characteristic knowledge, or model prior expectation knowledge; in many cases, introducing prior knowledge into the learning process through a prior encoder can make the learning task more efficient.

[0224] Schematically, the second encoder is implemented as a prior encoder. Prior usually refers to the model's prior knowledge or expectations of certain information. The Prior encoder is an encoder used to capture this prior knowledge. By inputting the first observation data into the Prior encoder, a latent representation can be mapped, which contains prior information about the input data, thereby helping the control strategy to better utilize prior knowledge to perform tasks.

[0225] Optionally, when analyzing the first observation data at time t and the reference action data at time t, the first observation data at time t is input into the second encoder so as to analyze the first observation data at time t with the help of prior knowledge and generate a second feature representation.

[0226] In an optional embodiment, the first feature representation and the second feature representation are fused to obtain an encoded feature representation.

[0227] Schematically, the first feature representation and the second feature representation are feature concatenated to obtain the encoded feature representation.

[0228] That is, by using the imitation information expressed by the first feature representation and the prior information expressed by the second feature representation, it is possible to splice together a coding feature representation that expresses deeper and more accurate information.

[0229] Step 530: Based on the decoding of the encoded feature representation, the predicted action data of the robot at the tth moment is predicted.

[0230] Schematically, the encoded feature representation is decoded by the decoder to obtain the predicted action data of the robot at the tth moment.

[0231] In an optional embodiment, the predicted action data of the robot at the tth moment is predicted based on the encoded feature representation and the first observation data at the tth moment.

[0232] Schematically, the encoded feature representation and the first observation data at the t-th moment are input into the decoder for decoding, and the predicted action data of the robot at the t-th moment is predicted.

[0233] Step 540 : Based on the predicted action data at the t-th moment and the first state data at the t-th moment, predict the state of the robot at the k-th moment to obtain the second state data.

[0234] Here, the kth moment is the moment after the tth moment.

[0235] In an optional embodiment, the world model obtained through training executes a state prediction process, and the predicted action data at the tth moment and the first state data at the tth moment are input into the world model to obtain the second state data of the robot at the kth moment.

[0236] Schematically, in the sequential execution process described above, after the world model is obtained by training with data corresponding to at least two moments respectively, the predicted action data at the t-th moment and the first state data at the t-th moment are input into the world model to obtain the second state data at the k-th moment; or, in the alternating execution process described above, at least two moments are divided into at least two moment groups, and the process of predicting the second state data at the k-th moment is implemented as follows: determining the t-th moment corresponding to the k-th moment, determining the moment group to which the t-th moment belongs, and when the t-th moment belongs to the q-th moment group in at least two moment groups, obtaining the q-th world model by training with data corresponding to each moment in the q-th moment group, inputting the predicted action data at the t-th moment and the first state data at the t-th moment into the q-th world model to obtain the second state data at the k-th moment, and so on, where q is a positive number.

[0237] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0238] Step 550 : training the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy.

[0239] Among them, the trained control strategy is used to control the robot's movements.

[0240] As shown in FIG6 , it shows a schematic diagram of a training process of a control strategy provided by an exemplary embodiment of the present application.

[0241] When the control strategy is implemented as a model or algorithm, the network structure or algorithm expression corresponding to the control strategy before and after training is the same, so the strategies before and after training can be collectively referred to as the control strategy. The control strategy is implemented as a combination of the IL encoder 610, the Prior encoder 620, and the motor decoder (Motor Decoder) 630. Therefore, the process of training the control strategy can be regarded as the process of optimizing and adjusting the network parameters corresponding to the IL encoder 610, the Prior encoder 620, and the motor decoder 630.

[0242] Illustratively, the input of the IL encoder 610 includes the first state data s t The corresponding first observation data o t , also includes reference action data q t , the output of the IL encoder 610 is the first feature representation; the input of the Prior encoder 620 is the first observation data o t , the output of the Prior encoder 620 is the second feature representation; the first feature representation and the second feature representation are fused to obtain the encoded feature representation z t ; By encoding feature representation z t Input motor decoder 630, decode to get predicted motion data a t In addition, the world model 640 is used to predict the action data a t and the first state data s t Execute state prediction to obtain the predicted second state data Finally, by predicting the second state data The control strategy is trained based on the loss value determined by the reference action data, that is, the network parameters corresponding to the IL encoder 610, the Prior encoder 620 and the motor decoder 630 are optimized and adjusted until the trained control strategy is obtained.

[0243] Among them, the prior distribution p(z t |o t) and the posterior distribution q(z t |o t ,q t ) is modeled as a Gaussian distribution, as shown in Formula 3 and Formula 4 below.

[0244] Formula 3:

[0245] p(z t |o t )~N(π prior (z t |o t ),σ 2 I)

[0246] Formula 4:

[0247] q(z t |o t ,q t )~N(π IL (z t |o t ,q t )+π prior (z t |o t ),σ 2 I)

[0248] Where N() represents Gaussian distribution; π prior (z t |o t ) represents the neural network θ represented by the Prior encoder 620 prior Parameterization of π IL (z t |o t ,q t ) represents the neural network θ represented by the IL encoder 610 IL parameterization of ; σ represents a fixed standard deviation; I is the identity matrix.

[0249] Among them, the predicted second state data The loss function of the loss value corresponding to the reference action data is implemented as the following formula 5.

[0250] Formula 5:

[0251] in, is the loss value at time t; is the joint position loss; is the loss of joint velocity; is the robot position loss; is the robot speed loss.

[0252] Joint position loss As shown in the following formula 6.

[0253] Formula 6:

[0254] in, is the joint position predicted in the second state data at time t; is the joint position in the reference motion data at time t.

[0255] Joint velocity loss As shown in Formula 7 below.

[0256] Formula 7:

[0257] in, is the joint velocity predicted from the second state data at time t; is the joint velocity in the reference motion data at time t.

[0258] Robot position loss As shown in the following formula 8.

[0259] Formula 8:

[0260] in, is the robot body position (e.g., the robot's mass point position) predicted from the second state data at time t; is the robot body position in the reference motion data at time t; is the robot body direction (e.g., the robot's mass point direction) predicted from the second state data at time t; is the robot body orientation in the reference motion data at time t.

[0261] Robot speed loss As shown in the following formula 9.

[0262] Formula 9:

[0263] in, is the robot body velocity (e.g., the robot's mass velocity) predicted from the second state data at time t; is the robot body velocity in the reference motion data at time t; is the robot body angular velocity (e.g., the robot's mass point angular velocity) predicted from the second state data at time t; is the angular velocity of the robot in the reference motion data at time t.

[0264] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0265] To sum up, with the help of the collected first observation data, we can make full use of the robot's operation in the physical world to carry out an accurate supervised learning process. Under the limitation of the reference action data, we can obtain predicted action data through the control strategy, so that we can use the predicted action data and the second state data at the moment after the first state data is predicted. Through the second state data and the reference action data, we can make targeted adjustments to the control strategy, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable operation capabilities during operation.

[0266] In the embodiments of this application, a method for determining the content of predicted motion data by mapping to a coded feature representation is described. The control strategy is implemented as an encoder and decoder architecture. The encoder performs in-depth analysis of the acquired data to improve the accuracy of the coded feature representation, thereby obtaining more comprehensive and in-depth robot data information. The decoder then decodes the data to obtain predicted motion data, improving the standardization of motion predictions, increasing the flexibility of the robot's learning process, and enhancing the performance and adaptability of the control strategy.

[0267] In an optional embodiment, when predictive motion data is acquired through the control strategy, motion instruction data can also be acquired, so that the robot can learn to follow the instructions and perform movements. This provides more diversity in the acquisition of predictive motion data and facilitates the application of the robot in multiple motion scenarios. As shown in FIG7 , step 220 shown in FIG2 can also be implemented as steps 710 to 750 as follows.

[0268] Step 710: Obtain motion instruction data corresponding to at least two moments.

[0269] The motion instruction data is used to represent the data that guides the robot to execute the motion process.

[0270] Optionally, the motion instruction data is implemented as at least one of a plurality of data values ​​such as linear velocity, angular velocity, position, and direction. The motion instruction data can provide the robot with a general direction and speed of motion. If the motion instruction data is implemented as linear velocity and angular velocity values, the motion instruction data can be used to understand the robot's movement, rotation, and other processes in the environment.

[0271] Schematically, motion command data is guidance information transmitted from an external source to a robot, enabling it to perform specific actions. For example, motion command data originates from a human operator, a higher-level decision-making system, a remote control, or other automated system, and to some extent represents expectations for the robot to perform a specific action.

[0272] Wherein, at least two moments correspond to a motion instruction data respectively, thereby representing the situation of guiding the robot to move at the current moment.

[0273] In some embodiments, the desired linear velocity and the desired angular velocity are randomly acquired to form the motion instruction data.

[0274] Schematically, given a random motion instruction data in represents the expected linear velocity at any time t, represents the expected angular velocity at any time t. That is, in the process of training the control strategy, the strategy training process can be performed with random motion command data.

[0275] Step 720 : Under the control strategy, perform a third encoding on the motion instruction data through a third encoder to obtain a third feature representation.

[0276] Schematically, the control strategy is implemented as an encoder-decoder architecture, where the encoder is used to map the input data into a latent space for representation to capture the key features of the input data, and its output encoded feature representation contains an abstract expression of the input data; the decoder is used to map the encoded feature representation output by the encoder back to the original data space to generate a robot action similar to the reference action, that is, to output predicted action data for realizing the robot action.

[0277] Optionally, the control strategy includes a third encoder. The third encoder is used to analyze the robot's motion as it executes command following based on motion command data. Command following (CF) refers to the ability of the machine learning model to accurately understand and execute user-provided commands. Command following is manifested as the machine learning model's understanding and execution of a series of commands, optionally including single-round commands or commands indicated by multiple rounds of dialogue. In commands implemented in multiple rounds of dialogue, the machine learning model needs to remember the previous contextual semantics and commands to ensure coherent and accurate responses.

[0278] Schematically, the third encoder is implemented as a command following encoder, and the CF encoder is responsible for encoding the given motion command data (for example, linear velocity, angular velocity) into a third feature representation. The purpose of this encoding is to map the motion command data to a latent space, which contains the motion command data and information that may be the first observation data; then the third feature representation generated by the CF encoder is used to analyze the robot's motion to analyze whether the robot has successfully complied with the motion command represented by the motion command data, and whether there are deviations or errors during the execution process.

[0279] That is, the third feature representation can be used to evaluate the robot's understanding of motion instruction data and the accuracy of its execution.

[0280] Optionally, when predicting the predicted action data at the t-th moment, the motion instruction data at the t-th moment is input into the third encoder to generate a third feature representation at the t-th moment.

[0281] The introduction of a third encoder to analyze and learn the robot's movement based on the motion instruction data to follow the instructions, so as to further improve the accuracy of the acquisition of the encoding feature representation, has certain authorization prospects.

[0282] In an optional embodiment, when the control strategy includes a second encoder and a third encoder, the following steps 731 to 732 are performed.

[0283] Step 731: Perform a second encoding on the first observation data at time t through a second encoder to obtain a second feature representation.

[0284] Schematically, the control strategy includes a second encoder, which is used to predict and analyze actions with the help of prior knowledge. The prior knowledge is knowledge learned during the training process of the second encoder.

[0285] Optionally, the second encoder is implemented as a Prior encoder, and the first observation data at the tth time is input into the second encoder so as to analyze the first observation data at the tth time with the help of prior knowledge and generate a second feature representation.

[0286] Step 732: fuse the second feature representation and the third feature representation to obtain a coded feature representation.

[0287] Illustratively, the third feature representation is concatenated with the second feature representation to obtain the encoded feature representation.

[0288] In an optional embodiment, when the control strategy includes a first encoder, a second encoder, and a third encoder, the following steps 741 to 742 are performed.

[0289] Step 741: Under the control strategy, a first encoder is used to perform a first encoding on the first observation data at the tth moment to obtain a first feature representation; a second encoder is used to perform a second encoding on the first observation data at the tth moment to obtain a second feature representation.

[0290] Illustratively, the content of step 741 has been described in the above step 520 and will not be repeated here.

[0291] Step 742: fuse the first feature representation, the second feature representation, and the third feature representation to obtain a coded feature representation.

[0292] Schematically, the first feature representation, the second feature representation, and the third feature representation are concatenated to obtain an encoded feature representation. The encoded feature representation can fully combine instruction following information, imitation learning information, and prior knowledge information, and has more accurate characteristics.

[0293] Step 750: Based on the decoding of the encoded feature representation, the predicted action data of the robot at the tth moment is predicted.

[0294] Schematically, the decoder in the control strategy decodes the encoded feature representation to predict the robot's predicted action data at time t. During the decoding process implemented by the decoder, the decoder converts the encoded feature representation as a latent feature in the latent space into predicted action data as an output sequence. That is, the encoded feature representation before the decoder is a low-dimensional vector representation, and the predicted action data after decoding is high-dimensional data in the original data space.

[0295] In an optional embodiment, the predicted action data at the tth moment is state analyzed through the world model to obtain the second state data at the kth moment, and then the control strategy is trained based on the second state data at the kth moment and the reference state data at the kth moment.

[0296] FIG8 is a schematic diagram of training a control strategy and obtaining a trained control strategy.

[0297] The control strategy based on training is the strategy obtained through training. If it is implemented as a model or algorithm, it has the same network structure or algorithm expression as the control strategy. Therefore, the strategies before and after training can be collectively referred to as control strategies.

[0298] The control strategy is implemented as a combination of the CF encoder 810, the Prior encoder 820 and the motor decoder 830. Therefore, the process of training the control strategy can be regarded as a process of optimizing and adjusting the network parameters corresponding to the CF encoder 810, the Prior encoder 820 and the motor decoder 830 respectively.

[0299] Schematically, the input of the CF encoder 810 is the motion instruction data c t , the output is the third feature representation; the input of Prior encoder 820 is the first state data s t The corresponding first observation data o t , the output of the Prior encoder 820 is the second feature representation; the third feature representation and the second feature representation are fused to obtain the encoded feature representation z t ; By encoding feature representation z t Input motor decoder 830, decode to get predicted motion data a t ; In addition, the world model 840 is used to predict the action data a t and the first state data s t Execute state prediction to obtain the predicted second state data Finally, by predicting the second state data The control strategy is trained based on the following loss value between the CF encoder 810 and the motion command data, that is, the network parameters corresponding to the CF encoder 810, the Prior encoder 820 and the motor decoder 830 are optimized and adjusted respectively until the trained control strategy is obtained.

[0300] Among them, the posterior distribution q(z t |o t ,q t ) is modeled as a Gaussian distribution, as shown in the following formula 10.

[0301] Formula 10:

[0302] q(z t |o t ,c t )~N(π prior (z t |o t )+π CF (z t |o t ,c t ),σ 2 I)

[0303] Where N() represents Gaussian distribution; π prior (z t |o t ) represents the neural network θ represented by the Prior encoder 820 prior Parameterization of π CF (z t |o t ,c t ) represents the neural network θ represented by the CF encoder 810 CF parameterization of ; σ represents a fixed standard deviation; I is the identity matrix.

[0304] Taking the action command data as angular velocity and linear velocity as an example, since the training goal is to make the robot follow the action command represented by the action command data, the following loss value includes the linear velocity loss and angular velocity loss

[0305] Among them, the following loss value and the linear speed loss and angular velocity loss The relationship between is shown in the following formula 11.

[0306] Formula 11:

[0307] Among them, the line speed loss As shown in the following formula 12.

[0308] Formula 12:

[0309] in, represents the linear velocity in the motion following data at time t; is the linear velocity predicted in the second state data at time t.

[0310] Among them, angular velocity loss As shown in the following formula 13.

[0311] Formula 13:

[0312] in, represents the angular velocity in the motion following data at the tth moment; is the angular velocity predicted in the second state data at time t.

[0313] In some embodiments, the schematic diagrams of training the control strategy and obtaining the trained control strategy shown in FIG6 and FIG8 respectively can be used in combination, that is, the control strategy includes an IL encoder, a Prior encoder, a CF encoder, and a motor decoder.

[0314] That is: after obtaining the encoded feature representation z t When , the IL encoder outputs the first feature representation, the Prior encoder outputs the second feature representation, and the CF encoder outputs the third feature representation, thereby fusing the encoded feature representation z t Then, based on the loss value obtained in FIG6 (also called the imitation loss value), the IL encoder, Prior encoder and motor decoder are trained, and based on the following loss value obtained in FIG8, the CF encoder, Prior encoder and motor decoder are trained, and the trained control strategy is obtained.

[0315] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0316] To sum up, with the help of the collected first observation data, we can make full use of the robot's operation in the physical world to carry out an accurate supervised learning process. Under the limitation of the reference action data, we can obtain predicted action data through the control strategy, so that we can use the predicted action data and the second state data at the moment after the first state data is predicted. Through the second state data and the reference action data, we can make targeted adjustments to the control strategy, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable operation capabilities during operation.

[0317] In an embodiment of the present application, it is introduced that in the process of obtaining motion instruction data to participate in generating a coding feature representation, a third encoder is used to analyze the movement of the robot executing instructions to follow based on the motion instruction data, so that the robot can learn the following information of the following instructions when moving through the control strategy, which facilitates the robot to adapt to various motion scenarios and improves the robot's motion adaptability.

[0318] In an optional embodiment, the above-mentioned robot control method can be applied to the usage scenario of a quadruped robot. The above-mentioned robot control method can also be called a model-based quadruped robot control method for effectively learning agile movement skills.

[0319] First, we briefly introduce the overall framework of the method. The overall framework consists of two parts: world model and control strategy.

[0320] The world model learns to approximate the unknown dynamics of simulation and reality. Given the current robot state (e.g., first state data) and action (e.g., first action data in training, or reference action data during application), it can predict the next state (e.g., second state data). The control strategy learns flexible behaviors by mimicking the movements of real animals. It can directly collect samples predicted by the trained world model to implement the analysis process.

[0321] Both the world model and the control policy are updated and iteratively trained in a supervised manner. First, state-action pairs are collected under a fixed control policy to adapt to the system dynamics and fit the world model (i.e., train the world model). The control policy is then updated and trained by interacting with the fixed world model. This process is repeated until the control policy converges.

[0322] Schematically, the robot control method is described using the following parts.

[0323] (1) World Model

[0324] Schematically, from training the world model f w Initially, it predicts the next state based on the current state and action, using the residual form as shown in Formula 1 above.

[0325] When training the world model, the robot collects state-action sequences under the control strategy; the world model is trained using a supervised learning method with n-step prediction loss, which is beneficial for long-term prediction, as shown in Formula 2 above.

[0326] (2) Imitation learning

[0327] In the context of the imitation task, the goal of the embodiments of the present application is to imitate motion sequences collected from real animals. The control strategy can be converted into an encoder-decoder architecture, for example, using a variational auto-encoder (VAE) architecture to implement the analysis process. This is shown in Equations 3 to 9 above.

[0328] Optionally, in order to ensure that the latent space is well formed so that suitable encoding feature representations can be further found in downstream command following tasks, a relative entropy (Kullback-Leible, KL) divergence regularization loss as shown in the following formula 14 can be added.

[0329] Formula 14:

[0330] in, represents the divergence loss value; D KL represents the prior distribution p(z t |o t ) and the posterior distribution q(z t |o t ,q t ) between the two; π IL (o t ,q t ) represents the neural network θ represented by the IL encoder IL is a parameterization of ; σ represents a fixed standard deviation.

[0331] (3) Command-following tasks

[0332] Schematically, a policy can be trained to follow the user-specified linear and angular velocities. By introducing the command to follow the encoder π CF (z t |o t ,c t), encoding action instruction data into the latent space.

[0333] In some embodiments, in order to maintain the naturalness of the robot's motion behavior, only the network parameters corresponding to the instruction following encoder can be updated during the training process, while the network parameters of the prior network and the motor decoder remain unchanged.

[0334] (4) Fine-tune on a real device

[0335] Due to the gap between simulation and reality, the policy learned from simulation may fail when deployed to the actual robot. Therefore, the command following encoder and motor decoder can be fine-tuned on the actual robot to follow the desired path. In order to keep the original motor encoder The natural behavior of can be regulated by introducing a regularization term, as shown in the following formula 15.

[0336] Formula 15:

[0337] in, represents the regularization term; Parameterization of the neural network representing the motor encoder representation; π M (a t |o t ,z t ) represents the adjusted network parameters.

[0338] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0339] In an optional embodiment, to evaluate the effectiveness of the above robot control method, a comparative experiment was conducted on the reinforcement learning environment Isaac Gym and a real quadruped robot. The purpose of the experiment was to answer the following key questions.

[0340] (1) Improvement in sample efficiency of robot control methods compared to reinforcement learning methods.

[0341] (2) The effectiveness of the fine-tuning process performed on the real robot in narrowing the simulation-to-reality gap.

[0342] (3) The generalization ability of the fine-tuned strategy on tasks that it has not been trained on before.

[0343] Among them, the following three experimental processes were carried out to address the above three issues.

[0344] (1) Experiments were conducted in both the simulation world and the physical world to compare the robot control method provided by the embodiments of the present application with a baseline method based on reinforcement learning in terms of sample efficiency;

[0345] (2) In physical-world experiments, the fine-tuning process was conducted on a real quadruped robot to demonstrate the real gap effect;

[0346] (3) In order to further demonstrate its generalization ability, the robot control method provided in the embodiments of the present application is used to perform path following tasks on four unseen paths.

[0347] Schematically, the following content describes the simulation experiment and the physical world experiment.

[0348] (1) Simulation experiment

[0349] (1) Sample efficiency in imitation learning tasks.

[0350] To address the first issue regarding sample efficiency, the compared models were trained from scratch on an imitation task using the Isaac Gym reinforcement learning environment. Isaac Gym is a high-performance GPU-based physics simulator for robotic learning that can simulate a fleet of robots simultaneously. Optionally, 128 agents can be trained simultaneously for this task.

[0351] Optionally, the robot control method provided in the embodiment of the present application is compared with the Proximal Policy Optimization (PPO) algorithm in terms of the number of samples collected from Isaac Gym. The reward function of the PPO algorithm is defined as r t is the reward function at time step t; is the calculated loss value.

[0352] Illustratively, the same policy network structure is maintained for the two methods (the method of the embodiment of the present application and the PPO algorithm) to facilitate meaningful comparison.

[0353] As shown in Figure 9, the average reward during training is shown in curve 910, which shows the method of the embodiment of the present application, where an average reward of 0.8 is achieved with approximately 5 million samples - shown by the dotted line in Figure 9; in contrast, curve 920 shows the PPO algorithm, which requires more than 70 million samples to achieve similar results. This shows that the sample efficiency of the method of the embodiment of the present application is more than ten times higher than that of the PPO algorithm. Among them, the gray area distributed in the weekly test represents the distribution of discrete data.

[0354] (2) Comparison of sample efficiency in adapting to new environments.

[0355] Schematically, training the PPO algorithm directly on a real robot is dangerous and may easily damage the robot. Therefore, a method of changing physical parameters and fine-tuning in simulation can be introduced to achieve training.

[0356] Optionally, some physical parameters can be varied for the simulation task. Table 1 below shows the physical parameters for the original environment (the environment used when training the robot) and the test environment (which may or may not be the same as the original environment). The test environment shows Environment 1, Environment 2, Environment 3, and Environment 4.

[0357] Table 1

[0358] For example, in environment 1, increasing the mass of the robot from 5.74 kg to 14 kg may make it impossible for the robot to apply the original strategy because of the significant change in the mass of the robot itself, that is, making the original strategy extremely difficult in the new environment.

[0359] In some embodiments, to simulate a scenario similar to robotic data collection in the physical world, two robots may be used in a simulation environment.

[0360] For example, each training iteration accumulates 3000 samples, which is equivalent to 1 minute of data collection at a control frequency of 50 Hz. For the PPO algorithm, the policy update is performed every 32 steps.

[0361] The training curve is shown in Figure 10. Curve 1010 in Figure 10 highlights that the method of the embodiment of the present application can obtain an average reward of 0.8 in this challenging environment using approximately 50,000 samples (equivalent to approximately 17 minutes of data).

[0362] In contrast, the PPO algorithm represented by curve 1020 in Figure 10 still performs poorly even with a sample size ten times larger than this. The gray area distributed in the weekly test represents the discrete data distribution.

[0363] In some embodiments, to further investigate the performance of command following, the task can also be extended to path following, where the robot aims to follow a predefined path.

[0364] FIG11 is a schematic diagram of four desired trajectories, including a trajectory 1110 (Oblong), a trajectory 1120 (Lemniscate), a trajectory 1130 (U-shape), and a trajectory 1140 (Star). An arrow 1150 on each trajectory indicates an initial position.

[0365] Optionally, a Purely Reactive Algorithm (PR algorithm) is used to convert the path information into instructions (motion instruction data).

[0366] Illustratively, taking the rectangular trajectory shown by trajectory 1110 with a target speed of 0.9 m / s as an example, trajectory 1110 is the trajectory involved in training, and the three environments shown in Table 1 above can be used for motion analysis.

[0367] Optionally, to simulate the fine-tuning process in the physical world, each training iteration involves collecting 1500 samples (30 seconds of data).

[0368] As shown in Figure 12, a training curve of loss 1210 is depicted. As can be observed from Figure 12, under workloads of 3kg, 5kg, and 7kg, the method of the embodiment of the present application requires approximately 4 iterations (2 minutes), 6 iterations (3 minutes), and 8 iterations (4 minutes) of data to achieve a loss below 0.6. These results indicate relatively good performance at these speeds, although some discrete data distributions may exist, which are not shown in the figure. In contrast, the loss of the PPO algorithm remains almost unchanged under such a limited sample size, so the results are not plotted.

[0369] That is, through the embodiments of the present application, it is possible to demonstrate the high sample efficiency and adaptability of this method to different environments in imitation learning and path following tasks.

[0370] (2) Real machine experiment

[0371] (1) Adaptation from simulation to reality.

[0372] To answer the second question, physical experiments can be conducted using real robots. Due to the gap between simulation and reality, policies trained in simulation may not be able to follow the path at the desired speed and may exhibit significant speed lag at high target speeds. This process emphasizes the need for fine-tuning in the physical world.

[0373] Optionally, three adaptation experiments are performed on the trajectory 1110 as shown in FIG11 , with target speeds of 0.6 m / s, 0.9 m / s, and 1.2 m / s, respectively; in order to fine-tune the policy in the real world, each iteration requires collecting 30 seconds of data (1500 samples) to train the world model, and then updating the policy network using the data predicted by the adapted world model.

[0374] As shown in FIG13 , a graph showing the change of the command following loss 1310 for four iterations (2 minutes of data) on a real robot for target velocities of 0.6 m / s, 0.9 m / s, and 1.2 m / s.

[0375] Table 2 below shows the average linear velocity error calculated over a 30-second trajectory after each iteration of physical world adaptation. and angular velocity loss Strategy 0 is the original strategy, and strategies 1 to 4 are other comparative strategies.

[0376] Table 2

[0377] As shown in Table 2, the loss is significantly reduced after the first iteration. In particular, for the case of a speed of 1.2 m / s, the speed error is reduced by more than 0.26 m / s. After four iterations, the loss converges, and the final performance is very effective in command following.

[0378] As shown in FIG14 , the following speed 1410 of real-world adaptation on a real robot is shown to be 1.2 m / s. Obviously, in the original strategy (iteration 0), the actual speed lags significantly behind the target speed. After the first iteration, the actual speed can follow the target to some extent, but there will be significant fluctuations. In the fourth iteration, the strategy effectively follows the target speed with minimal vibration. To avoid data congestion, the statistics of 0 iterations, 1 iteration, and 4 iterations are shown in FIG14 . Compared with the problem of large data fluctuations in one iteration, the data fluctuations after 4 iterations are smaller.

[0379] That is, after real-world adaptation on the real robot, it moves along the trajectory 1110 at a speed of 1.2 m / s.

[0380] (2) Generalization on unknown terrains.

[0381] To answer the last question, we evaluated the speed and path performance of the strategies of the present invention on unseen motion command data. In the previous experiment, we collected real robot data for a total of 7.5 minutes, with target speeds of 0.6 m / s, 0.9 m / s, and 1.2 m / s.

[0382] Optionally, this data is used for offline fine-tuning to obtain an adaptive policy.

[0383] As shown in FIG11 , the performance on all paths was tested, including the unseen paths of trajectory 1120 , trajectory 1130 , and trajectory 1140 , as well as the generalization ability of the unseen target speeds of 0.7 m / s, 0.8 m / s, and 1.0 m / s.

[0384] As shown in Table 3 below, the average linear velocity error (e) calculated on each of the four paths lasting 30 seconds is: v), angular velocity error (e ω ) and distance error (e p ); where the distance error is defined as are the robot position and the target position at time t, respectively; It is obtained by integrating the target velocity over time.

[0385] Table 3

[0386] As can be seen in Table 3, after offline fine-tuning, all errors were reduced by more than half. Figure 15 vividly illustrates the velocity tracking of trajectory 1110 on a real robot under the original strategy 1510 and the adapted strategy 1520. While the original strategy lags behind the target unseen velocity, the trained control strategy obtained by the embodiment of the present application can effectively track them, with an average linear velocity error of approximately 0.05 m / s.

[0387] Figure 16 shows the actual trajectories of the commanded follow-up paths at different unseen target speeds, including the long circle trajectory 1610 that participated in the training, as well as the lemniscate trajectory 1620, U-shaped trajectory 1630, and star-shaped trajectory 1640 that did not participate in the training. As can be seen from Figure 16, the original strategy (preset training strategy) lags significantly behind the reference trajectory, while the trained control strategy obtained by the embodiment of the present application can effectively follow it, and even perform faster at higher speeds. In short, the experimental results show that the trained control strategy can successfully handle unseen commands and follow unfamiliar paths, highlighting the generalization ability of the embodiment of the present application.

[0388] Optionally, the motion control technology of the trained quadruped robot can be applied to at least one of the following scenarios.

[0389] (1) Exploration and rescue: Quadruped robots can operate in various harsh and complex environments, such as disaster sites, fires, earthquakes, etc., to provide assistance to rescue workers.

[0390] (2) Agriculture: Quadruped robots can walk in farmland and help farmers complete tasks such as sowing and harvesting.

[0391] (3) Industrial production: Quadruped robots can carry heavy objects in factories and help workers complete repetitive tasks.

[0392] (4) Medical field: Quadruped robots can help disabled people walk and provide support. They can also be used for rehabilitation treatment, guiding the blind, etc.

[0393] (5) Entertainment and education: Quadruped robots can be used as toys or educational tools to help children learn scientific and technological knowledge and skills.

[0394] It is worth noting that the above scenarios are merely illustrative examples and are not limited to the embodiments of the present application.

[0395] To sum up, with the help of the collected first observation data, we can make full use of the robot's operation in the physical world to carry out an accurate supervised learning process. Under the limitation of the reference action data, we can obtain predicted action data through the control strategy, so that we can use the predicted action data and the second state data at the moment after the first state data is predicted. Through the second state data and the reference action data, we can make targeted adjustments to the control strategy, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable operation capabilities during operation.

[0396] In an embodiment of the present application, the robot control method introduced trains the world model and control strategy in a supervised manner, thereby significantly improving sample efficiency; a two-stage approach can also be adopted, involving training the strategy in simulation and fine-tuning it in a real machine, and then fine-tuning it in the physical world using less data. This process significantly reduces the amount of real-world data required and makes it possible to learn more complex motion skills.

[0397] FIG17 is a block diagram of a robot control device according to an exemplary embodiment of the present application. As shown in FIG17 , the device includes the following parts:

[0398] Data acquisition module 1710 is configured to acquire first state data and reference motion data corresponding to the robot at at least two moments in time, wherein the first state data is data converted from first observation data, the first observation data is data collected by sensors within the robot's operating environment, and the reference motion data is used to represent the robot's expected posture within the operating environment;

[0399] a motion prediction module 1720 for predicting predicted motion data of the robot at time t based on the first observation data at time t of the at least two time moments and the reference motion data at time t under a control strategy, wherein the control strategy is used to guide the motion of the robot, and t is a positive number;

[0400] a state prediction module 1730 for predicting the state of the robot at a kth time based on the predicted motion data at the tth time and the first state data at the tth time, to obtain second state data at the kth time, where the kth time is a time after the tth time among the at least two time points;

[0401] The strategy training module 1740 is used to train the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy; wherein the trained control strategy is used to control the action of the robot.

[0402] In an optional embodiment, the state prediction module 1730 is also used to obtain a world model, which is used to predict the state of the robot. The world model is a model trained based on the first state data and the first observation data; state prediction is performed on the predicted action data at the t-th moment and the first state data at the t-th moment through the world model to obtain the second state data at the k-th moment.

[0403] In an optional embodiment, the state prediction module 1730 is also used to obtain an environmental simulation model, which is a model to be trained to obtain the world model. The environmental simulation model is used to predict the predicted state data at the jth moment based on the first state data at the i-th moment and the first observation data at the i-th moment, where the j-th moment is a moment after the i-th moment among at least two moments, and i and j are positive numbers; based on the predicted state data at the j-th moment and the first state data at the j-th moment, the predicted loss value is used to indicate the predicted state data at the j-th moment and the first state data at the j-th moment, to obtain a predicted loss value, and the predicted loss value is used to indicate the difference between the predicted state data at the j-th moment and the first state data at the j-th moment; the environmental simulation model is trained by the predicted loss value to obtain the world model.

[0404] In an optional embodiment, the state prediction module 1730 is also used to obtain a state loss value corresponding to the jth moment based on the predicted state data at the jth moment and the first state data at the jth moment, and the state loss value is used to indicate the difference between the predicted state data at the jth moment and the first state data at the jth moment; the state loss values ​​corresponding to the at least two moments are summed to obtain the predicted loss value.

[0405] In an optional embodiment, the action prediction module 1720 is also used to encode the first observation data at the tth moment and the reference action data at the tth moment under the control strategy to obtain a coded feature representation; based on decoding the coded feature representation, predict the predicted action data of the robot at the tth moment.

[0406] In an optional embodiment, the action prediction module 1720 is also used to perform a first encoding on the first observation data at the t-th moment and the reference action data at the t-th moment through a first encoder to obtain a first feature representation, and the first encoder is used to implement imitation learning of the robot's action based on the reference action data; perform a second encoding on the first observation data at the t-th moment through a second encoder to obtain a second feature representation, and the second encoder is used to predict and analyze the robot's action with the help of prior knowledge, and the prior knowledge is the knowledge learned during the training process to obtain the second encoder; the control strategy includes the first encoder and the second encoder; the first feature representation and the second feature representation are fused to obtain the encoded feature representation.

[0407] In an optional embodiment, the action prediction module 1720 is also used to obtain motion instruction data corresponding to the at least two moments respectively, and the motion instruction data is used to represent the data guiding the robot to perform the motion process; under the control strategy, the motion instruction data is third-encoded by a third encoder to obtain a third feature representation, and the third encoder is used to analyze the motion situation of the robot following the instruction based on the motion instruction data; the first observation data at the tth moment is second-encoded by a second encoder to obtain a second feature representation, and the second encoder is used to predict and analyze the robot's action with the help of prior knowledge, and the prior knowledge is the knowledge learned in the process of training to obtain the second encoder; based on the second feature representation and the third feature representation, the encoded feature representation is obtained.

[0408] In an optional embodiment, the action prediction module 1720 is also used to fuse the second feature representation and the third feature representation to obtain the encoded feature representation, and the control strategy includes the second encoder and the third encoder; or, the first encoder performs a first encoding on the first observation data at the t-th moment and the reference action data at the t-th moment to obtain a first feature representation, and the first encoder is used to implement imitation learning of the robot's action based on the reference action data; the first feature representation, the second feature representation and the third feature representation are fused to obtain the encoded feature representation, and the control strategy includes the first encoder, the second encoder and the third encoder.

[0409] In an optional embodiment, the action prediction module 1720 is further configured to decode the encoded feature representation through a decoder in the control strategy, and output the predicted action data of the robot at the t-th moment.

[0410] In an optional embodiment, the strategy training module 1740 is also used to obtain the loss value corresponding to the kth moment based on the difference between the second state data at the kth moment and the reference action data at the kth moment; and adjust the strategy parameters in the control strategy with the loss value to obtain the trained control strategy.

[0411] In an optional embodiment, the strategy training module 1740 is also used to obtain the loss values ​​corresponding to the at least two moments respectively; iteratively adjust the strategy parameters in the control strategy through the loss values ​​corresponding to the at least two moments respectively, to obtain the trained control strategy.

[0412] In an optional embodiment, the data acquisition module 1710 is also used to obtain the first observation data collected by the robot at the at least two moments respectively; perform state processing on the first observation data corresponding to the at least two moments respectively to obtain the first state data corresponding to the at least two moments respectively; obtain a reference action sequence, the reference action sequence is used to represent the expected posture sequence of the robot in the operating environment, and the reference action sequence includes reference action data corresponding to the at least two moments respectively.

[0413] It should be noted that the robot control device provided in the above embodiment is merely an example of the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the robot control device provided in the above embodiment and the robot control method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0414] Figure 18 shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. The server 1800 includes a central processing unit (CPU) 1801, a system memory 1804 including a random access memory (RAM) 1802 and a read-only memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the CPU 1801. The server 1800 also includes a mass storage device 1806 for storing an operating system 1813, application programs 1814, and other program modules 1815.

[0415] The mass storage device 1806 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1806 and its associated computer-readable media provide non-volatile storage for the server 1800.

[0416] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. The system memory 1804 and mass storage device 1806 described above may be collectively referred to as memory.

[0417] According to various embodiments of the present application, the server 1800 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 1800 may be connected to a network 1812 via a network interface unit 1811 connected to the system bus 1805, or the network interface unit 1811 may be used to connect to other types of networks or remote computer systems (not shown).

[0418] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0419] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the robot control method provided by the above-mentioned method embodiments.

[0420] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the robot control method provided by the above-mentioned method embodiments.

[0421] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robot control method described in any of the above embodiments.

[0422] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A robot control method, the method being executed by a computing device, the method comprising: Obtaining first state data and reference motion data corresponding to the robot at at least two moments, respectively, where the first state data is data converted from first observation data, the first observation data is data collected by sensors in an operating environment of the robot, and the reference motion data is used to represent an expected posture of the robot in the operating environment; Under the control strategy, based on the first observation data at the tth moment of the at least two moments and the reference motion data at the tth moment, predicting the predicted motion data of the robot at the tth moment, the control strategy being used to guide the motion of the robot, where t is a positive number; Based on the predicted motion data at the t-th time and the first state data at the t-th time, predicting the state of the robot at the k-th time to obtain second state data at the k-th time, where the k-th time is a time that is later than the t-th time among the at least two time points; Based on the second state data at the k-th moment and the reference action data at the k-th moment, the control strategy is trained to obtain a trained control strategy; wherein the trained control strategy is used to control the action of the robot.

2. The method according to claim 1, wherein The step of predicting the state of the robot at the kth moment based on the predicted action data at the tth moment and the first state data at the tth moment to obtain the second state data at the kth moment includes: Acquire a world model, where the world model is used to predict the state of the robot, and the world model is a model trained based on the first state data and the first observation data; State prediction is performed on the predicted action data at the t-th moment and the first state data at the t-th moment through the world model to obtain the second state data at the k-th moment.

3. The method according to claim 1 or 2, wherein: The obtaining of the world model includes: Obtaining an environment simulation model, where the environment simulation model is a model to be trained to obtain the world model, the environment simulation model being used to predict predicted state data at a jth moment based on first state data at an i-th moment and first observation data at the i-th moment, where the j-th moment is a moment that is later than the i-th moment among at least two moments, and i and j are positive numbers; Obtaining a predicted loss value based on the predicted state data at the j-th moment and the first state data at the j-th moment, the predicted loss value being used to indicate a difference between the predicted state data at the j-th moment and the first state data at the j-th moment; The environment simulation model is trained using the predicted loss value to obtain the world model.

4. The method according to any one of claims 1 to 3, wherein: The obtaining of the predicted loss value based on the predicted state data at the j-th moment and the first state data at the j-th moment includes: Based on the predicted state data at the j-th moment and the first state data at the j-th moment, obtaining a state loss value corresponding to the j-th moment, the state loss value being used to indicate a difference between the predicted state data at the j-th moment and the first state data at the j-th moment; The state loss values corresponding to the at least two moments are summed to obtain the predicted loss value.

5. The method according to any one of claims 1 to 4, wherein: The method of predicting the predicted motion data of the robot at time t based on the first observation data at time t of the at least two time moments and the reference motion data at time t under the control strategy includes: Under the control strategy, encoding the first observation data at the t-th time and the reference action data at the t-th time to obtain an encoded feature representation; Based on the decoding of the encoded feature representation, the predicted action data of the robot at the t-th moment is predicted.

6. The method according to any one of claims 1 to 5, wherein: The method of encoding the first observation data at the t-th time and the reference action data at the t-th time under the control strategy to obtain an encoded feature representation includes: performing a first encoding on the first observation data at the t-th time and the reference motion data at the t-th time by a first encoder to obtain a first feature representation, wherein the first encoder is used to implement imitation learning of the robot's motion based on the reference motion data; performing a second encoding on the first observation data at time t by a second encoder to obtain a second feature representation, wherein the second encoder is used to predict and analyze the robot's motion with the help of prior knowledge, the prior knowledge being knowledge learned during the training process of obtaining the second encoder; the control strategy includes the first encoder and the second encoder; The first feature representation and the second feature representation are fused to obtain the encoded feature representation.

7. The method according to any one of claims 1 to 6, wherein: The method of encoding the first observation data at the t-th time and the reference action data at the t-th time under the control strategy to obtain an encoded feature representation includes: Acquire motion instruction data corresponding to the at least two moments, wherein the motion instruction data is used to represent data for guiding the robot to perform a motion process; Under the control strategy, a third encoder is used to perform a third encoding on the motion instruction data to obtain a third feature representation, and the third encoder is used to analyze the motion of the robot executing the instruction following based on the motion instruction data; performing a second encoding on the first observation data at time t by a second encoder to obtain a second feature representation, wherein the second encoder is used to predict and analyze the robot's motion by using prior knowledge, where the prior knowledge is knowledge learned during the training process of obtaining the second encoder; The encoded feature representation is obtained based on the second feature representation and the third feature representation.

8. The method according to any one of claims 1 to 7, wherein: The obtaining the encoded feature representation based on the second feature representation and the third feature representation includes: fusing the second feature representation and the third feature representation to obtain the encoded feature representation, wherein the control strategy includes the second encoder and the third encoder; or A first encoder is used to perform a first encoding on the first observation data at the t-th moment and the reference action data at the t-th moment to obtain a first feature representation, and the first encoder is used to implement imitation learning of the robot's action based on the reference action data; the first feature representation, the second feature representation and the third feature representation are integrated to obtain the encoded feature representation, and the control strategy includes the first encoder, the second encoder and the third encoder.

9. The method according to any one of claims 1 to 8, wherein: The predicting and obtaining the predicted motion data of the robot at the time t based on decoding the encoded feature representation includes: The decoder in the control strategy decodes the encoded feature representation and outputs the predicted action data of the robot at the tth moment.

10. The method according to any one of claims 1 to 9, wherein: The step of training the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain the trained control strategy includes: Obtaining a loss value corresponding to the k-th moment based on a difference between the second state data at the k-th moment and the reference action data at the k-th moment; The strategy parameters in the control strategy are adjusted using the loss value to obtain the trained control strategy.

11. The method according to any one of claims 1 to 10, wherein: The step of adjusting the policy parameters in the control policy using the loss value to obtain the trained control policy includes: Obtaining loss values corresponding to the at least two moments respectively; The strategy parameters in the control strategy are iteratively adjusted using the loss values corresponding to the at least two moments to obtain the trained control strategy.

12. The method according to any one of claims 1 to 11, wherein: The obtaining of first state data and reference motion data corresponding to the robot at at least two moments includes: Acquire first observation data collected by the robot at the at least two moments; Performing state processing on the first observation data corresponding to the at least two moments respectively to obtain the first state data corresponding to the at least two moments respectively; A reference action sequence is obtained, where the reference action sequence is used to represent an expected posture sequence of the robot in the operating environment, and the reference action sequence includes reference action data corresponding to the at least two moments.

13. A robot control device, comprising: a data acquisition module, configured to acquire first state data and reference motion data corresponding to the robot at at least two moments, wherein the first state data is data converted from first observation data, the first observation data is data collected by sensors within the robot's operating environment, and the reference motion data is used to represent an expected posture of the robot within the operating environment; an action prediction module, configured to predict predicted action data of the robot at time t based on the first observation data at time t of the at least two time moments and the reference action data at time t under a control strategy, wherein the control strategy is used to guide the action of the robot, and t is a positive number; a state prediction module, configured to predict the state of the robot at a kth moment based on the predicted motion data at the tth moment and the first state data at the tth moment, to obtain second state data at the kth moment, wherein the kth moment is a moment that is later than the tth moment among the at least two moments; A strategy training module is used to train the control strategy based on the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy; wherein the trained control strategy is used to control the action of the robot.

14. A computer device comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the robot control method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the robot control method according to any one of claims 1 to 12.

16. A computer program product comprising computer instructions, wherein when the computer instructions are executed by a processor, the robot control method according to any one of claims 1 to 12 is implemented.

Images