Robot control method and device, equipment, storage medium and program product

Abstract

The invention discloses a robot control method and device, equipment, a storage medium and a program product, and relates to the field of machine learning. The method comprises the steps that first state data and reference action data corresponding to a robot at multiple moments are acquired; under the condition of a preset control strategy, predicting predicted action data of the robot based on the first observation data and the reference action data at the t-th moment; obtaining second state data at the k-th moment based on the predicted action data at the t-th moment and the first state data; and training a preset control strategy based on a loss value between the second state data and the reference action data at the kth moment. The operation condition of the robot in the physical world can be fully utilized to accurately supervise the learning process, and the action of the robot is controlled by means of the trained control strategy, so that the robot can have more accurate movement precision in the operation process. The method can be applied to various scenes such as cloud technology, artificial intelligence and intelligent transportation.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of machine learning, and particularly to a robot control method, device, equipment, storage medium, and program product. Background Art

[0002] With the continuous development of robot technology, the functions of robots are becoming increasingly powerful. Different types of robots can cope with various working environments and execute different operation tasks according to operation instructions.

[0003] In related technologies, a world model that simulates and predicts the physical world is usually used to provide a simulated environment for a robot, enabling the robot to learn and experiment in an internally generated, simulated environment. In this way, the learning efficiency and safety of the robot can be improved without directly contacting the physical world, and the robot can also have a high motion performance in the physical world.

[0004] In the above process, although the robot can learn efficiently with the help of the world model, when the robot needs to perform high-precision tasks (such as: precisely imitating animals, precisely moving on a specified route, etc.), due to the simulation characteristics of the world model, the process of training the robot only through the world model cannot fully utilize the advantages of the physical world, reducing the accuracy of the robot in performing high-precision tasks. Summary of the Invention

[0005] Embodiments of the present application provide a robot control method, device, equipment, storage medium, and program product, which can make full use of the running conditions of the robot in the physical world for an accurate supervised learning process, and control the actions of the robot with the trained control strategy, so that the robot can have more accurate motion accuracy and more stable running ability during operation. The technical solutions are as follows.

[0006] On the one hand, a robot control method is provided, and the method includes:

[0007] Obtain first state data and reference action data respectively corresponding to a robot at multiple moments. The first state data is data obtained by performing a preset state process on first observation data, and the first observation data is data collected by a sensor by the robot in the running environment where it is located. The reference action data is used to represent the expected posture of the robot in the running environment;

[0008] Under the condition of a preset control strategy, based on the first observation data at the t-th moment among the multiple moments and the reference action data at the t-th moment, predict the predicted action data of the robot at the t-th moment, where t is a positive number;

[0009] Based on the predicted action data at the t-th moment and the first state data at the t-th moment, predict the state of the robot at the k-th moment to obtain the second state data at the k-th moment, where the k-th moment is a moment after the t-th moment among the multiple moments;

[0010] Based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, train the preset control strategy to obtain a trained control strategy; wherein, the trained control strategy is used to control the actions of the robot.

[0011] On the other hand, a robot control device is provided, and the device includes:

[0012] A data acquisition module, configured to acquire the first state data and the reference action data respectively corresponding to the robot at multiple moments, where the first state data is data obtained by performing a preset state process on first observation data, and the first observation data is data collected by a sensor by the robot in the operating environment where it is located, and the reference action data is used to represent the desired posture of the robot in the operating environment;

[0013] An action prediction module, configured to, under the condition of a preset control strategy, based on the first observation data at the t-th moment and the reference action data at the t-th moment among the multiple moments, predict the predicted action data of the robot at the t-th moment, where t is a positive number;

[0014] A state prediction module, configured to, based on the predicted action data at the t-th moment and the first state data at the t-th moment, predict the state of the robot at the k-th moment to obtain the second state data at the k-th moment, where the k-th moment is a moment after the t-th moment among the multiple moments;

[0015] A strategy training module, configured to, based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, train the preset control strategy to obtain a trained control strategy; wherein, the trained control strategy is used to control the actions of the robot.

[0016] On the other hand, a computer device is provided, and the computer device includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the robot control method as described in any one of the embodiments of the present application above.

[0017] On the other hand, a computer-readable storage medium is provided. At least one instruction, at least one program, a code set, or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the robot control method according to any one of the embodiments of the present application as described above.

[0018] On the other hand, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the robot control method according to any one of the above embodiments.

[0019] The beneficial effects brought by the technical solutions provided by the embodiments of the present application at least include:

[0020] After obtaining the first state data and the reference action data respectively corresponding to multiple moments, under the condition of a preset control strategy, predict the predicted action data of the robot at the t-th moment, and then, according to the predicted action data and the first state data, predict the state of the robot at the k-th moment to obtain the second state data. Thus, train the trained control strategy according to the loss value between the second state data at the k-th moment and the reference action data at the k-th moment. With the collected first observation data, it is possible to make full use of the operating conditions of the robot in the physical world for an accurate supervised learning process. Under the limitation of the reference action data, obtain the predicted action data through the preset control strategy. Thus, it is possible to predict the second state data at a later moment with the predicted action data and the first state data. The preset control strategy can be adjusted specifically through the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, and thus enable the robot to have more accurate motion accuracy and more stable operating ability during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0023] Figure 2It is a flowchart of a robot control method provided by an exemplary embodiment of the present application;

[0024] Figure 3 It is a flowchart of a robot control method provided by another exemplary embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of obtaining a world model through training provided by an exemplary embodiment of the present application;

[0026] Figure 5 It is a flowchart of a robot control method provided by still another exemplary embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of training a preset control strategy provided by an exemplary embodiment of the present application;

[0028] Figure 7 It is a flowchart of a robot control method provided by yet another exemplary embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of training a preset control strategy provided by still another exemplary embodiment of the present application;

[0030] Figure 9 It is a schematic diagram of loss comparison between a robot control method provided by an exemplary embodiment of the present application and related technologies;

[0031] Figure 10 It is a schematic diagram of loss comparison after fine-tuning provided by an exemplary embodiment of the present application;

[0032] Figure 11 It is a schematic diagram of four kinds of trajectories provided by an exemplary embodiment of the present application;

[0033] Figure 12 It is a schematic diagram of comparison of following losses under different loads provided by an exemplary embodiment of the present application;

[0034] Figure 13 It is a schematic diagram of comparison of following losses under different speeds provided by an exemplary embodiment of the present application;

[0035] Figure 14 It is a schematic diagram of speed comparison under different iteration times provided by an exemplary embodiment of the present application;

[0036] Figure 15 It is a schematic diagram of speed comparison under different control strategies provided by an exemplary embodiment of the present application;

[0037] Figure 16 It is a schematic diagram of comparison of trajectory following corresponding to four different trajectories respectively provided by an exemplary embodiment of the present application;

[0038] Figure 17 It is a structural block diagram of a robot control device provided by an exemplary embodiment of the present application;

[0039] Figure 18 It is a structural block diagram of a server provided by an exemplary embodiment of the present application. Detailed implementation manners

[0040] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0041] First, a brief introduction is made to the terms involved in the embodiments of the present application.

[0042] Artificial Intelligence (AI): It is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include, for example, sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called the large model or the basic model, and can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0043] Machine Learning (ML): It is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration.

[0044] Robots: A robot is a mechanical or virtual device that automatically performs tasks. It is usually designed to complete specific human jobs or execute specific functions. Robots can possess the capabilities of perception, decision-making, and execution, enabling them to interact with the environment and complete complex tasks. Robots generally have learning capabilities (usually achieved through machine learning and deep learning technologies), a certain degree of autonomy, the ability to perceive the environment (achieved through sensors such as cameras, lidar, sonar, etc.), the ability to make decisions on perceived information (such as path planning, target recognition, task priority ranking), task execution capabilities, interaction capabilities, and other various capabilities. As a multi-functional engineering system, robots are widely used in various fields such as the industrial field, medical field, service field, exploration field, etc. With the progress of technology, the application expansion of robots will also continue to evolve.

[0045] In related technologies, a world model that simulates and predicts the physical world is usually used to provide a simulated environment for the robot, enabling the robot to learn and experiment in an internally generated, simulated environment. This can improve the learning efficiency and safety of the robot without directly contacting the physical world and also enable the robot to have a high level of motion performance in the physical world. In the above process, although the robot can learn efficiently with the help of the world model, when the robot needs to perform high-precision tasks, due to the simulation characteristics of the world model, the process of training the robot only through the world model cannot fully utilize the advantages of the physical world, reducing the accuracy of the robot in performing high-precision tasks.

[0046] In the embodiments of the present application, a robot control method is introduced, which can make full use of the operating conditions of the robot in the physical world for an accurate supervised learning process. The preset control strategy is specifically trained through the loss value between the predicted second state data and the reference action data obtained based on the first observation data, so as to control the actions of the robot with the trained control strategy, enabling the robot to have more accurate motion accuracy and more stable operating capabilities during operation. The robot control method provided by the embodiments of the present application can be applied to robot types such as quadruped robots, pushing robots, robotic arms, wheeled robots, etc., and can also be applied to various scenarios such as the industrial field, medical field, service field, exploration field, etc. The embodiments of the present application do not limit this.

[0047] In some embodiments, the robot control method applied to a robotic arm in the industrial field is taken as an example for illustration.

[0048] Schematically, robotic arms have flexibility, high precision, and programmability, making them widely used in the industrial field, such as performing high-precision assembly and disassembly tasks, performing welding tasks with higher risks, performing handling and loading / unloading tasks with high intensity, performing packaging tasks with high quality requirements, etc. To improve the motion accuracy of the robotic arm as fully as possible, various types of sensors deployed on the robotic arm can be used to collect the first observation data of the robotic arm in its operating environment, and then the first observation data is processed in a preset state to obtain the first state data. Under the condition of a preset control strategy, based on the first observation data and the reference action data, the predicted action data of the robotic arm can be predicted. Furthermore, based on the predicted action data and the first state data, the state of the robotic arm at a later time can be predicted to obtain the second state data. Thus, based on the loss value between the second state data and the reference action data, the preset control strategy is trained to obtain the trained control strategy. Through the trained control strategy, the actions of the robotic arm can be controlled more precisely, so that the robotic arm can provide more efficient operating capabilities in the industrial field according to the needs of the user, and improve the industrial operation efficiency by various motion types such as linear motion, rotational motion, arc motion, joint motion, grasping and releasing motion, etc.

[0049] In some embodiments, the robotic control method is illustrated by taking a quadruped robot in the service field as an example.

[0050] Schematically, the quadruped robot is implemented as a robot dog. In the service field, including the spiritual companionship service scenario, various types of sensors deployed on the robot dog can be used to collect the first observation data of the robot dog in its operating environment, and then the first observation data is processed in a preset state to obtain the first state data. The first state data can better avoid the influence of acquisition errors or noises and more comprehensively show the operating conditions of the robot dog. Under the condition of a preset control strategy, based on the first observation data and the reference action data, the predicted action data of the robot dog can be predicted. Furthermore, based on the predicted action data and the first state data, the state of the robot dog at a later time can be predicted to obtain the second state data. Thus, based on the loss value between the second state data and the reference action data, the preset control strategy is trained to obtain the trained control strategy. Through the trained control strategy, the actions of the robot dog can be controlled so that the robot dog can provide emotional companionship for the user according to the needs of the user through operations such as moving, jumping, running, sitting down, etc., and improve the quality of life of the user.

[0051] It should be noted that the above application scenarios are only schematic examples, and the robot types and robot application fields can be combined arbitrarily and are not limited here.

[0052] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the first observation data, the first state data, the reference action data, etc. involved in this application are all obtained under sufficient authorization.

[0053] Secondly, the implementation environment involved in the embodiments of this application is described. The robot control method provided in the embodiments of this application can be implemented by the robot alone or by the interaction between the robot and the server through data. The embodiments of this application do not limit this. Optionally, the interaction between the robot and the server to execute the robot control method is taken as an example for description.

[0054] Schematically, please refer to Figure 1 , in this implementation environment, there are involved a robot 110 and a server 120, and the robot 110 and the server 120 are connected through a communication network 130.

[0055] [[ID=\\(11\\)]]In some embodiments, the robot 110 has a data acquisition function. For example: A variety of sensors are deployed in the robot 110, such as: Inertial Measurement Unit (IMU), vision sensor, infrared sensor, contact sensor, pressure sensor, temperature sensor, etc., force / torque sensor, angle sensor, encoder for measuring joint angle and position, optical sensor, etc.

[0056] Schematically, with the help of various sensors deployed on the robot 110, the first observation data can be collected. That is: The first observation data is the data collected by the robot 110 through sensors in the operating environment where it is located. For example: including the linear velocity and angular velocity collected by the IMU deployed on the robot 110, including the joint position and joint velocity collected by the encoder, etc.

[0057] Optionally, taking the first observation data corresponding to multiple moments being collected respectively as an example, the reference action data corresponding to multiple moments can also be obtained to represent the expected posture of the robot 110 in the operating environment at the corresponding moments.

[0058] In some embodiments, in order to be able to more systematically express the system state of the robot 110 in the operating environment, the first observation data is subjected to a preset state process to obtain the first state data at the corresponding moment. For example: The first observation data at the t-th moment is subjected to a state process through a preset state process method to obtain the first state data at the t-th moment, where t is a positive number.

[0059] Optionally, the robot 110 sends the first state data corresponding to multiple moments and the reference action data corresponding to multiple moments to the server 120 through the communication network 130, so that the server 120 obtains the first state data and the reference action data corresponding to the robot 110 at multiple moments respectively.

[0060] Illustratively, the server 120 can inversely infer the first observation data based on the first state data; the server 120 can also receive the first observation data and the like sent by the robot 110 through the communication network 130.

[0061] In some embodiments, under the condition of a preset control strategy, the server 120 predicts the predicted action data of the robot 110 at the t-th moment based on the first observation data and the reference action data at the t-th moment among multiple moments.

[0062] Illustratively, the preset control strategy is a pre-set control strategy, which is used to predict the action that the robot needs to execute at the current moment according to the first observation data collected by the sensors on the robot 110 and the determined reference action data, that is, to obtain the predicted action data.

[0063] In some embodiments, the server 120 predicts the state of the robot 110 at the k-th moment based on the predicted action data and the first state data, and obtains the second state data.

[0064] Wherein, the k-th moment is a moment after the t-th moment among multiple moments.

[0065] Illustratively, after determining the predicted state data at the t-th moment and the first state data at the t-th moment, the server can predict the state of the robot 110 at the k-th moment after the t-th moment to obtain the second state data at the k-th moment. That is: the second state data is the predicted state result at the k-th moment.

[0066] In some embodiments, the server 120 trains the preset control strategy based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, and obtains the trained control strategy.

[0067] Wherein, the trained control strategy is used to control the actions of the robot 110.

[0068] Optionally, the server 120 sends the trained control strategy to the robot 110 through the communication network 130, so that the robot 110 controls its own actions based on the trained control strategy.

[0069] It should be noted that the above-mentioned robots include, but are not limited to, quadruped robots, pushing robots, robotic arm robots, wheeled robots, etc.; the above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0070] Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, application programs, and networks within a wide area network or local area network to achieve data calculation, storage, processing, and sharing. Cloud technology is the general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model, which can form a resource pool, be used on demand, and be flexible and convenient.

[0071] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system.

[0072] Combined with the above noun introduction and application scenarios, the robot control method provided by this application will be described. Taking this method applied to a server as an example, as Figure 2 shown, this method includes the following steps 210 to step 240.

[0073] Step 210, obtaining first state data and reference action data respectively corresponding to the robot at multiple moments.

[0074] Among them, the first state data is the data obtained after performing a preset state process on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment where it is located.

[0075] Schematically, a variety of sensors are deployed on the robot. The sensors can be deployed on the joints of the robot (such as: elbow joint, knee joint, shoulder joint, etc.), or at the position of the end effector (such as: the end of the robotic arm), or on multiple parts of the robot such as the head, torso, and contacts (such as: fingers, toes, etc.). The deployment position of the sensors usually depends on the design, task, and application requirements of the robot.

[0076] Optionally, at least one of the following multiple sensors can be deployed on the robot: inertial measurement unit, vision sensor, infrared sensor, contact sensor, pressure sensor, temperature sensor, etc., optical sensor, angle sensor, force / torque sensor, encoder for measuring joint angle and position, etc.

[0077] Among them, the inertial measurement unit, also known as the inertial sensor, is usually composed of a gyroscope and an accelerometer, and is usually deployed on the joints of the robot. The gyroscope is used to measure the angular velocity, and the accelerometer is used to measure the linear acceleration. The linear velocity can be obtained by integrating the linear acceleration.

[0078] Vision sensors such as cameras and cameras are used to capture image or video data for object recognition, scene interpretation, navigation, finding moving objects, etc.; infrared sensors are used to detect infrared light (thermal radiation) to measure temperature (non-contact thermometer), detect organisms or heat sources, etc.; contact sensors are used to detect physical contact to determine whether the robot components touch an object or a surface; pressure sensors are used to measure the pressure of gas or liquid to ensure that the grasped object is not damaged; temperature sensors are used to measure the temperature of the environment or object to maintain the thermal state and avoid overheating; optical sensors use light to detect objects, measure distances or sense the environment for precise distance measurement and environmental scanning; angle sensors are used to measure the rotation angle or position to precisely control the angles of the robot joints and rotating components; force / torque sensors are used to measure force and torque (rotational force), which are particularly important in robot arms and actuators, and can help the robot perform tasks with appropriate forces, such as precision assembly or object handling.

[0079] Encoders are used to measure joint angles and positions, thereby providing high-precision angle information and position information, which is crucial for the precise motion control of the robot. Encoders can be incremental (providing relative position changes) or absolute (providing absolute position information).

[0080] The above sensor types are only for illustrative introduction. Each sensor has its specific uses and advantages. Usually, in a complex robot system, multiple different types of sensors are used comprehensively to achieve higher functionality and adaptability, so as to achieve accurate motion control and perception.

[0081] In some embodiments, the first observation data is collected through the sensors deployed on the robot, and the first observation data is used to characterize the running situation of the robot in the running environment.

[0082] Schematically, the first observation data includes at least one of the data collected by sensors such as the robot, linear velocity (such as the value obtained by synthesizing the joint linear velocities corresponding to multiple joints on the robot), robot angular velocity (such as the value obtained by synthesizing the joint angular velocities corresponding to multiple joints on the robot), joint positions (the positions corresponding to multiple joints on the robot), and joint velocities (the motion velocities corresponding to multiple joints on the robot).

[0083] Optionally, the multiple moments are moments within a historical time period, and each of the multiple moments corresponds to first observation data. The first observation data collected at different moments is used to represent the data collected by the robot through sensors at the current moment. For example: at moment 1, the first observation data G1 is collected. The first observation data G1 is the data collected by the multiple sensors deployed on the robot at moment 1; at moment 2, the first observation data G2 is collected. The first observation data G2 is the data collected by the multiple sensors deployed on the robot at moment 2, and so on.

[0084] In some embodiments, after performing a preset state process on the first observation data, first state data is obtained.

[0085] Schematically, a preset state process method is used to perform a state process on the first observation data to obtain first state data. The preset state process method is a preset state process method for comprehensively analyzing the first observation data collected by sensors at the current moment to refine and obtain first state data representing the overall state of the robot relative to the operating environment.

[0086] For example: the first observation data, as the data directly collected by the robot, may contain a lot of noise and redundant information, making it difficult to accurately estimate the overall situation of the robot. By using at least one of multiple techniques such as filtering techniques, weight adjustment techniques, data model mapping techniques, etc. as the preset state process method to perform a state process on the first observation data, so that the first state data can better reflect the important information in the first observation data, filter out unnecessary noise information, simplify the analysis while reducing the difficulty of understanding the robot, and also improve the analysis efficiency of the robot, facilitating better understanding and control of the robot's behavior.

[0087] In some embodiments, the preset state process method for performing the preset state process is illustrated as follows.

[0088] (1) Filtering algorithm: Using a filtering algorithm (such as a Kalman filter, etc.) as the preset state process method can extract a relatively pure operating state of the robot from the first observation data measured by the sensors, while reducing the influence of noise, that is: obtaining first state data.

[0089] (2) Mathematical model: If the dynamic behavior of the robot can be represented by a mathematical model, then the mathematical model can be used as the preset state process method to map the first observation data measured by the sensors through the mathematical model to obtain first state data, where the data model may involve mathematical tools such as differential equations and integrals.

[0090] (3) Feature extraction: Use a feature extraction network as the preset state processing method to extract features from the sensor data with the feature extraction network to identify the key features of the robot state and obtain the first state data. This process can be achieved through technologies such as signal processing and pattern recognition.

[0091] (4) Machine learning: A machine learning algorithm can be used as the preset state processing method. The pre-trained machine learning model is used to learn the mapping relationship between the observed data and the state data corresponding to the robot from the sensor data, so that the first state data can be obtained based on the first observed data collected. This process is more useful for non-linear and complex systems, etc.

[0092] The preset state processing can also be referred to as state conversion, which is a key step in mapping the first observed data collected by the sensor to the robot state (the first state data), and helps to understand and control the behavior of the robot or other automated systems. The above preset state processing methods are only illustrative examples, and the embodiments of the present application are not limited thereto.

[0093] Optionally, the first state data includes at least one of the robot position, robot direction, robot linear velocity, robot angular velocity, joint position, and joint velocity; wherein, the first state data is obtained after the above state processing of the first observed data. For example, the robot includes multiple joints. Under the first state data, the joint position P corresponding to joint A1 among the multiple joints is obtained by adjusting the joint position p corresponding to joint A under the first observed data based on the joint positions corresponding to other joints, the robot linear velocity, the robot angular velocity, and other data under the first observed data.

[0094] Schematically, based on the first observed data corresponding to multiple moments, after performing the preset state processing on the first observed data corresponding to multiple moments, the first state data corresponding to multiple moments can be obtained.

[0095] For example: The first observed data G1 is collected at time 1. After performing the preset state conversion on the first observed data G1, the first state data Z1 corresponding to time 1 is obtained; the first observed data G2 is collected at time 2. After performing the preset state conversion on the first observed data G2, the first state data Z2 corresponding to time 2 is obtained, etc.

[0096] Among them, the reference action data is used to represent the expected posture of the robot in the operating environment.

[0097] Schematically, multiple moments each correspond to a reference action data set. Each reference action data set includes at least one of multiple action postures, such as robot motion (representing the robot's overall motion, such as standing, crawling, or raising its leg), joint motions (such as joint A bending 45°, joint B bending 90°, etc.). By combining the reference action data corresponding to multiple moments, a reference action sequence can be generated. For example, a reference action sequence is (Q1, Q2, Q3, ...), where moment 1 corresponds to reference action data Q1, moment 2 corresponds to reference action data Q2, and so on. Reference action sequences can be used to describe the movements a robot should take in a specific task or environment.

[0098] Optionally, a reference action sequence is obtained, and reference action data corresponding to multiple moments are obtained from the reference action sequence based on multiple moments. Each reference action data corresponds to one moment. By integrating the expected postures represented by multiple reference action data, multiple expected postures can be executed continuously to achieve the reference action.

[0099] Schematically, the reference action sequence is realized as a data set representing various continuous actions, such as a teaching motion sequence, a simulation-generated trajectory, a learning algorithm-generated action, and human motion capture.

[0100] A teachable motion sequence is typically a series of motions performed by a human operator, an animal, or other similar user. These motion trajectories can be used to represent the desired posture of the robot in similar situations. For example, if a robotic arm needs to grasp an object in space, a human operator can demonstrate manual grasping by manually operating the robotic arm. The recorded grasping trajectory serves as a reference motion sequence, where each moment corresponds to a reference motion data set.

[0101] Simulation-generated trajectories are robot motion trajectories generated using physical simulations or motion planning algorithms. These trajectories can serve as reference action sequences for performing similar tasks in real-world environments. For example, by planning a path for a robot to move to a target location in simulation, the generated path can be used as a reference action sequence for actual robot navigation.

[0102] Learning algorithms to generate actions, such as machine learning methods like reinforcement learning, can be used to allow robots to learn through trial and error to generate reference action sequences. In this case, the robot may form an optimized reference action sequence by repeatedly trying in the environment and adjusting its action strategy based on feedback.

[0103] Human motion capture, such as when the tasks of a robot involve collaborative work with humans or imitating human actions, can use human motion capture technology to record the motion trajectories generated when humans perform tasks, and convert the motion trajectories into a reference action sequence for the robot, in order to achieve the desired postures corresponding to different moments through multiple reference action data.

[0104] That is to say: With the help of reference action data, it can help the robot learn how to adjust its posture and execute actions in different scenarios, so as to help the robot better perform tasks in the physical world and improve the adaptability and flexibility of the robot.

[0105] Step 220, under the condition of a preset control strategy, based on the first observation data at the t-th moment and the reference action data at the t-th moment among multiple moments, predict the predicted action data of the robot at the t-th moment.

[0106] Where t is a positive number.

[0107] Schematically, the preset control strategy is a pre-set control strategy. The preset control strategy refers to the rules, algorithms or methods used to guide and regulate the behavior of the robot. The preset control strategy can predict the action situation of the robot at the current moment based on the first observation data and reference action data obtained at any moment, so as to execute appropriate motion behaviors according to the first observation data as much as possible under the limitation of the reference action data.

[0108] Optionally, the preset control strategy is implemented as at least one of multiple algorithms such as the A* algorithm, D* algorithm, Proportional-Integral-Differential (PID) algorithm, etc.; it can also be implemented as a model predictive control method (such as predicting future states through a data model); it can also be implemented as a reinforcement learning strategy (a strategy used to optimize the behavior of the robot through trial-and-error learning, which can adjust its actions through a reward signal in the interaction between the robot and the environment, so as to gradually learn the optimal strategy), such as: Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) are widely used as network implementation methods of the reinforcement learning strategy.

[0109] Schematically, the t-th moment is any moment among multiple moments except the last moment. Taking the prediction of the predicted motion data at the t-th moment by using the first observation data and the reference motion data at the t-th moment as an example, the first observation data and the reference motion data at the t-th moment are used as the independent variables of the preset control strategy, that is: the first observation data and the reference motion data at the t-th moment are substituted / input into the preset control strategy to predict the predicted motion data of the robot at the t-th moment.

[0110] That is: the predicted motion data is the result output by the preset control strategy and is used to predict the motion situation of the robot at the current moment.

[0111] Step 230: Based on the predicted motion data at the t-th moment and the first state data at the t-th moment, predict the state of the robot at the k-th moment to obtain the second state data at the k-th moment.

[0112] Among them, the k-th moment is a moment after the t-th moment among multiple moments.

[0113] Schematically, after obtaining the predicted motion data at the t-th moment and the first state data at the t-th moment, predict how the robot will change its state in a short time through the predicted motion data at the t-th moment to obtain the predicted state change. This process may involve the integration of the motion equation or the deduction using a mathematical model; then combine the first state data at the t-th moment with the predicted state change to achieve the purpose of transitioning from the first state data and obtain the second state data at the k-th moment.

[0114] Step 240: Based on the loss value between the second state data at the k-th moment and the reference motion data at the k-th moment, train the preset control strategy to obtain the trained control strategy.

[0115] Schematically, as a moment among multiple moments, when obtaining the reference motion data corresponding to each moment, the reference motion data at the k-th moment is included.

[0116] After predicting the second state data at the k-th moment, based on the first observation data collected at the k-th moment, it can be determined that the reference motion data at the k-th moment has strong purposefulness. Therefore, the predicted second state data at the k-th moment and the reference motion data at the k-th moment can be compared to determine the loss value between the second state data at the k-th moment and the reference motion data at the k-th moment.

[0117] Optionally, taking supervised learning as an example, a loss value between the reference action data and the predicted second state data can be determined. The loss value can be calculated by the Mean Squared Error (MSE), or by the Cross-Entropy Loss, or by using a custom loss function, etc. The choice of the relevant loss function depends on the nature of the task and the definition of the problem.

[0118] In some embodiments, the preset control strategy is trained by the loss value to obtain a trained control strategy.

[0119] Schematically, the preset control strategy is implemented as an algorithm, and the preset control strategy includes multiple algorithm parameters. When training the preset control strategy by the loss value, the algorithm parameters are changed to optimize the preset control strategy.

[0120] Optionally, calculate the rate of change of the loss function with respect to the algorithm parameters when determining the loss value to obtain a gradient value, and then use an optimization algorithm (such as the gradient descent algorithm) to update the algorithm parameters of the preset control strategy according to the gradient value to reduce the loss value until the number of training times is reached or the loss value drops to a preset threshold, obtaining a trained control strategy.

[0121] Among them, the trained control strategy is used to control the actions of the robot.

[0122] Optionally, after obtaining the trained control strategy, under the conditions of the trained control strategy, based on the first observation data at any moment and the reference action data at that moment, the predicted action data of the robot at the current moment can be predicted, so that the robot can execute the motion process at the current moment through the predicted action data.

[0123] It should be noted that the above is only a schematic example, and the embodiments of the present application are not limited thereto.

[0124] In summary, by means of the collected first observation data, an accurate supervised learning process can be fully utilized based on the operation of the robot in the physical world. Under the limitation of the reference action data, the predicted action data is obtained through the preset control strategy. Thus, the second state data at a subsequent moment can be predicted by means of the predicted action data and the first state data. The preset control strategy can be adjusted specifically through the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, and thus enable the robot to have more accurate motion accuracy and more stable operation ability during the operation process.

[0125] In an alternative embodiment, a state prediction is performed based on predicted action data and first state data with the aid of a world model to obtain second state data, and the world model is a model trained based on the first state data and first observation data. Schematically, as Figure 3 shown, the above Figure 2 shown embodiment can also be implemented as the following steps 310 to step 350; wherein, step 230 can also be implemented as the following steps 330 to step 340.

[0126] Step 310, obtain the first state data and reference action data respectively corresponding to the robot at multiple moments.

[0127] Among them, the first state data is the data obtained after performing a preset state process on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment where the robot is located; the reference action data is used to represent the desired posture of the robot in the operating environment.

[0128] Schematically, a variety of sensors are deployed on the robot. When the robot is operating in the operating environment where it is located, the multiple sensors deployed on it are in an operating state and can collect data in real time or periodically to obtain the first observation data respectively corresponding to multiple moments. For example: collecting data at a periodic interval of each second, then multiple moments represent multiple seconds, and each second corresponds to a first observation data, and the first observation data includes at least one of the data collected by sensors such as the robot, linear velocity, robot angular velocity, joint position, and joint velocity.

[0129] Optionally, after performing a preset state process on the first observation data respectively corresponding to multiple moments, the first state data respectively corresponding to multiple moments is obtained.

[0130] For example: the first observation data G1 is collected at moment 1, and after performing a preset state conversion on the first observation data G1, the first state data Z1 corresponding to moment 1 is obtained; the first observation data G2 is collected at moment 2, and after performing a preset state conversion on the first observation data G2, the first state data Z2 corresponding to moment 2 is obtained, and so on.

[0131] Schematically, the first state data includes at least one of the robot position, robot direction, robot linear velocity, robot angular velocity, joint position, and joint velocity.

[0132] Among them, the reference action data is used to represent the desired posture of the robot in the operating environment.

[0133] In some embodiments, the first observation data respectively collected by the robot at multiple moments is obtained; a preset state conversion is performed on the first observation data respectively corresponding to multiple moments to obtain the first state data respectively corresponding to multiple moments.

[0134] In some embodiments, a reference action sequence is obtained.

[0135] The reference action sequence is used to characterize the expected posture sequence of the robot in the operating environment. The reference action sequence includes reference action data corresponding to multiple moments respectively. That is, each of the multiple moments corresponds to one reference action data, and the multiple reference action data form the reference action sequence, which is used to represent the expected posture change of the robot at multiple moments.

[0136] Step 320, under the condition of a preset control strategy, based on the first observation data at the t-th moment among multiple moments and the reference action data at the t-th moment, predict the predicted action data of the robot at the t-th moment.

[0137] The t-th moment is any moment except the last moment among the multiple moments, and t is a positive number.

[0138] Schematically, the preset control strategy is a pre-set control strategy.

[0139] Optionally, the preset control strategy can be implemented as an algorithm for guiding and regulating the behavior of the robot; under the condition of the preset control strategy, the first observation data at the t-th moment and the reference action data at the t-th moment can be substituted into the preset control strategy to calculate the predicted action data of the robot at the t-th moment.

[0140] Optionally, the preset control strategy can be implemented as a machine learning model for guiding and regulating the behavior of the robot; under the condition of the preset control strategy, the first observation data at the t-th moment and the reference action data at the t-th moment can be input into the preset control strategy to learn the deep information therein through the machine learning model and predict the predicted action data of the robot at the t-th moment, etc.

[0141] Step 330, obtain a world model.

[0142] The world model is used to predict the state of the robot, and the world model is a model trained based on the first state data and the first observation data.

[0143] Optionally, the world model is regarded as a component or application deployed on the robot; in the field of robotics, the world model is usually designed as an internal representation for simulating and understanding the physical environment where the robot is located.

[0144] Schematically, the world model is the robot's internal representation of its operating environment, usually an abstract expression of the physical world. The world model is constructed by selectively capturing information relevant to the robot's task execution. As a generative model that attempts to learn the robot's interaction with the operating environment, the world model is the basis for the robot to understand and act in the physical world and is a dynamic and continuously updated model.

[0145] Optionally, the world model is used to estimate the state of the robot within the operating environment so that the robot can predict future states, such as predicting the robot's position, speed changes, or other behaviors. Additionally, based on the state prediction process of the world model, the robot can also perform task planning such as path planning and motion planning, that is, determining the state required to reach a specific position.

[0146] Among them, the world model can be continuously updated so that the robot can learn new knowledge from experience, continuously adapt to changes in the environment, and improve performance. Furthermore, it is beneficial for the robot to be able to understand and interact with complex environments. By enhancing complexity and precision, the robot's ability to perform tasks in the physical world can be improved.

[0147] In an optional embodiment, an environment simulation model is obtained.

[0148] Among them, the environment simulation model is the model to be trained to obtain the world model.

[0149] Among them, the environment simulation model is used to predict the predicted state data at the j-th moment based on the first state data at the i-th moment and the first observation data at the i-th moment. The i-th moment is one of multiple moments, and the j-th moment is a moment after the i-th moment among multiple moments. i is a positive number, and j is a positive number.

[0150] Schematically, the environment simulation model can be regarded as an initialized world model, and the environment simulation model has a certain state prediction function based on the model structure.

[0151] In some embodiments, the environment simulation model performs a state prediction process based on the first state data and the first observation data.

[0152] Schematically, the i-th moment is any one of multiple moments, and the j-th moment is a moment after the i-th moment among multiple moments. The first state data at the i-th moment is determined from the first state data corresponding to multiple moments, and the first observation data at the i-th moment can also be determined from the first observation data corresponding to multiple moments. Then, the first state data at the i-th moment and the first observation data at the i-th moment are input into the environment simulation model, and the predicted state data at the j-th moment after the i-th moment is output.

[0153] In some embodiments, first motion data corresponding to a plurality of moments are acquired, where the first motion data is used to describe the posture data generated when the robot moves in the motion environment.

[0154] Schematically, the first motion data includes at least one information describing the motion state of the robot, such as the position, direction, speed, acceleration, joint target angle, etc.; the first motion data is data collected by the motion sensor deployed by the robot.

[0155] The multiple moments correspond to a first action data respectively, for example, moment 1 corresponds to the first action data D1, moment 2 corresponds to the first action data D2, and so on.

[0156] In some embodiments, the environment simulation model performs a state prediction process based on the first action data, the first state data, and the first observation data.

[0157] Illustratively, the environment simulation model predicts the predicted state data at the jth moment based on the first action data at the i-th moment, the first state data at the i-th moment, and the first observation data at the i-th moment.

[0158] Optionally, the environmental simulation model predicts the state change difference at the jth moment based on the first action data at the i-th moment and the first observation data at the i-th moment; the state change difference at the j-th moment is added to the first state data at the i-th moment to obtain the predicted state data at the j-th moment.

[0159] Schematically, as shown in Formula 1 below, it is a formula for obtaining predicted state data based on the environmental simulation model.

[0160] Formula 1:

[0161]

[0162] Among them, s t is the first state data at time t; is the predicted state data at time t+1 obtained by predicting the state at time t+1 after time t; f w is the environmental simulation model; f w (δs t |o t ,a t ,π) is the neural network expression of the environmental simulation model, parameterized as θ w , express input w including o t and a t , o t represents the first observation data at time t, a trepresents the first action data at the t-th moment, and π represents the preset control strategy (in a fixed form); δs t is the state change difference at the j-th moment; thus, f w (δs t |o t ,a t ,π) expresses that under the preset control strategy π, by inputting the first observation data o w at the t-th moment and the first action data a t at the t-th moment into the environmental simulation model f t , the state change difference δs t at the (t + 1)-th moment can be predicted; furthermore, based on the state change difference δs t at the (t + 1)-th moment and the first state data at the t-th moment, the predicted state data at the (t + 1)-th moment can be obtained.

[0163] In an alternative embodiment, a prediction loss value is obtained based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment.

[0164] Schematically, the first state data at the j-th moment is determined from the first state data corresponding to multiple moments, and the predicted state data at the j-th moment is compared with the first state data at the j-th moment obtained based on the first observation data, thereby obtaining the prediction loss value.

[0165] In some embodiments, a state loss value corresponding to the j-th moment is obtained based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment.

[0166] Optionally, among multiple moments, except the first moment, the corresponding predicted state data can be obtained based on the prediction process, and each of the multiple moments corresponds to a first state data. Therefore, according to the first state data and the corresponding predicted state data corresponding to multiple moments respectively, the state loss values corresponding to multiple moments can be obtained.

[0167] Schematically, the first state data at the j-th moment is determined from the first state data corresponding to multiple moments, and the difference between the predicted state data at the j-th moment and the first state data at the j-th moment obtained based on the first observation data is determined to obtain the prediction loss value representing the difference between the first state data at the j-th moment and the first state data at the j-th moment.

[0168] In some embodiments, the prediction loss value is obtained by summing the state loss values corresponding to multiple moments.

[0169] Optionally, state loss values corresponding to multiple moments are determined according to the above process, and then the multiple state loss values are summed to obtain a prediction loss value.

[0170] Schematically, as shown in Formula Two below, it is a formula for obtaining a prediction loss value by integrating state loss values corresponding to multiple moments respectively.

[0171] Formula Two:

[0172]

[0173] Wherein, is the prediction loss value; n is the number of loss training steps, which can be regarded as the number of moments of multiple moments; is the predicted state data at the t-th moment; s t is the first state data at the t-th moment.

[0174] In an optional embodiment, the environment simulation model is trained with the prediction loss value to obtain a world model.

[0175] Schematically, the environment simulation model is trained for a preset number of times with the prediction loss value to obtain a world model; or, the environment simulation model is trained for a preset number of times with the prediction loss value until the loss value no longer decreases, so as to obtain a world model, etc.

[0176] As Figure 4 shown, it is a schematic diagram of training to obtain a world model.

[0177] Based on the fact that the world model is a trained model and has the same network structure as the environment simulation model, the environment simulation model before training can also be called the world model; the input of the world model 410 includes the state-action sequence τ = {s0, a0, s1, a1, …, s n , a n}, where there are the first state data s t (such as s0 at moment 0, s1 at moment 1, etc.) and the first action data a t (such as a0 at moment 0, a1 at moment 1, etc.); it also includes the first observation data o t (such as o0 at moment 0, o1 at moment 1, etc.), etc.

[0178] The state-action prediction sequence is predicted through the world model 410, which includes predicted state data and predicted action data corresponding to multiple moments respectively; through the state-action prediction sequence and the state-action sequence the prediction loss value can be calculated, and thus the world model 410 can be trained with the prediction loss value.

[0179] By using the state loss values corresponding to multiple moments respectively, a predicted loss value after comprehensively analyzing multiple moments is obtained, so that the world model can be centrally trained through the predicted loss value, and the training efficiency of the world model can be improved.

[0180] Step 340: Perform state prediction on the predicted action data at the t-th moment and the first state data at the t-th moment through the world model to obtain the second state data at the k-th moment.

[0181] Wherein, the k-th moment is a moment after the t-th moment among multiple moments.

[0182] Schematically, after obtaining the predicted action data at the t-th moment and the first state data at the t-th moment, a state prediction process is performed through the world model. Since the world model is obtained by training an environmental simulation model, the world model and the environmental simulation model have the same neural network structure, but the network parameters may vary due to training.

[0183] As shown in Formula 1 above, f w can represent the world model obtained after training; f w (δs t |o t , a t , π) can represent the neural network expression of the world model. When performing state prediction on the k-th moment through the world model, the predicted action data at the t-th moment and the first action data at the t-th moment can be obtained. By inputting the predicted action data at the t-th moment and the first action data at the t-th moment into the world model f w and then outputting the state change difference when transforming from the t-th moment to the k-th moment, that is, obtaining the state change difference corresponding to the k-th moment. Furthermore, a summation operation is performed on the first state data at the t-th moment and the state change difference corresponding to the k-th moment to obtain the second state data corresponding to the k-th moment.

[0184] Step 350: Train a preset control strategy based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment to obtain a trained control strategy.

[0185] In some embodiments, taking the process of training the control strategy as supervised learning as an example, the purpose of supervised learning is to enable the robot to learn to map states to corresponding actions. Then, the loss value is usually realized as the loss between the reference action data and the predicted second state data, and the action represented by the predicted second state data is made as close as possible to the reference action represented by the reference action data.

[0186] Schematically, after predicting the second state data at the k-th moment, based on the first observation data collected at the k-th moment, it can be determined that the reference action data at the k-th moment has strong purposefulness. Therefore, the predicted second state data at the k-th moment and the reference action data at the k-th moment can be compared to determine the loss value between the second state data at the k-th moment and the reference action data at the k-th moment.

[0187] In some embodiments, taking the process of training a control policy as an example of reinforcement learning, the purpose of reinforcement learning is to enable the robot to learn the value of taking a certain action in a certain state. Then, the loss value is usually realized as the difference between the obtained first state data and the predicted second state data.

[0188] Schematically, after predicting the second state data at the k-th moment, based on the first observation data collected at the k-th moment, it can be determined that the first state data at the k-th moment has strong authenticity. Therefore, the predicted second state data at the k-th moment and the first state data at the k-th moment can also be compared to determine the loss value between the second state data at the k-th moment and the reference action data at the k-th moment.

[0189] In an alternative embodiment, the preset control policy is trained with the loss value to obtain the trained control policy.

[0190] In some embodiments, based on the difference between the second state data at the k-th moment and the reference action data at the k-th moment, the loss value corresponding to the k-th moment is obtained; the policy parameters in the preset control policy are adjusted with the loss value to obtain the trained control policy.

[0191] Optionally, the loss values corresponding to multiple moments are obtained; the policy parameters in the preset control policy are iteratively adjusted with the multiple loss values to obtain the trained control policy.

[0192] Schematically, the preset control policy includes an encoder and a decoder, and the policy parameters are realized as the network parameters corresponding to the encoder; and / or, the policy parameters are realized as the network parameters corresponding to the decoder, etc.

[0193] Among them, the trained control policy is used to control the actions of the robot.

[0194] Schematically, after a planned path is given, the robot can efficiently control the actions of the robot according to the trained control policy. For example: more accurately obtain the predicted action data through the trained control policy, and then perform the action process based on the predicted action data.

[0195] In an alternative embodiment, the environmental simulation model and the preset control strategy are co-deployed on the robot to jointly control the robot and assist the robot in the motion process. In addition to being deployed on the robot, the environmental simulation model and the preset control strategy can be collaboratively optimized so that the robot can adapt to different environments and tasks through the trained world model and the trained control strategy, thereby improving the ability of autonomous decision-making and execution.

[0196] Optionally, the training processes corresponding to the above environmental simulation model and preset control strategy can be implemented as a sequential execution process or an alternating execution process. The following content describes the training process.

[0197] (1) Sequential execution process

[0198] Schematically, after obtaining the preset control strategy first, under the condition of keeping the predictive control strategy unchanged, the environmental simulation model is trained with the first state data and the first observation data corresponding to multiple moments respectively to obtain the world model. Then, under the condition of keeping the world model unchanged, the preset control strategy is trained with the first state data, the first observation data and the reference action data corresponding to multiple moments respectively, and finally the trained control strategy is obtained.

[0199] That is: first control the preset control strategy to train the world model; then control the world model unchanged to train the trained control strategy.

[0200] (2) Alternating execution process

[0201] Schematically, after obtaining the preset control strategy first, under the condition of keeping the predictive control strategy unchanged, the environmental simulation model is trained with the first state data and the first observation data corresponding to a part of multiple moments respectively to obtain the first trained simulation model. Then, under the condition of keeping the first trained simulation model unchanged, the preset control strategy is first trained with the first state data, the first observation data and the reference action data corresponding to a part of moments respectively to obtain the first trained control strategy;

[0202] After that, the first trained simulation model is trained with the first state data and the first observation data corresponding to another part of multiple moments respectively to obtain the world model. Then, under the condition of keeping the world model unchanged, the first trained control strategy is secondarily trained with the first state data, the first observation data and the reference action data corresponding to another part respectively until the trained control strategy is obtained.

[0203] That is: first, control the preset control strategy and train the environment simulation model with data at some moments; then keep the simulation model after the first training unchanged and train the control strategy after the first training with data at some moments; then control the preset control strategy after the first training and train the simulation model after the first training with data at another part of the moments; then keep the simulation model after the first training unchanged and train the control strategy after training with data at another part of the moments, and so on.

[0204] The division of the above-mentioned some moments and another part of the moments is only for illustrative examples. Multiple moments can also be divided into more moment groups, so as to execute the above-mentioned alternating training process through the data corresponding to the moments in multiple moment groups. The embodiments of the present application do not limit this.

[0205] In some embodiments, the world model and the trained control strategy are jointly used to assist the robot in the action control process.

[0206] Illustratively, in the software architecture of the robot, the world model and the trained control strategy are integrated. The world model provides the state information of the environment, such as: obstacle position, target position, etc.; the trained control strategy uses the information of the world model to more accurately generate the actions of the robot to respond to the current environment and task requirements. Thus, through the integration process, the robot becomes more flexible and adaptive, and can execute various tasks in complex and dynamic environments. It can also continuously learn and optimize the world model and the trained control strategy through data such as future collected data and reference action data, so as to fully improve the effect of the robot performing tasks in the physical world.

[0207] It should be noted that the above are only illustrative examples, and the embodiments of the present application do not limit this.

[0208] In summary, by means of the first observed data collected, it is possible to make full use of the running situation of the robot in the physical world for an accurate supervised learning process. Under the limitation of the reference action data, the predicted action data is obtained through the preset control strategy. Thus, it is possible to predict the second state data at subsequent moments by means of the predicted action data and the first state data. The preset control strategy can be adjusted specifically through the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, and thus enable the robot to have more accurate motion accuracy and more stable running ability during the running process.

[0209] In the embodiments of the present application, the content of predicting the second state data based on the world model is introduced. The world model is a model trained based on the first state data and the first observation data. The trained world model is used as the model for state prediction, so as to fully combine the world model with the preset control strategy to cope with the complex environment during the movement of the robot, improve the adaptability and generalization performance of the robot, facilitate the robot to perform more efficient online planning and decision-making, and improve the efficiency and performance of the robot.

[0210] In an alternative embodiment, during the process of predicting the predicted state data, the first observation data at the t-th moment and the reference action data at the t-th moment are mapped into the latent space to obtain the encoded feature representation. Then, under the condition of the preset control strategy, the predicted action data of the robot at the t-th moment is obtained according to the encoded feature representation. Schematically, as Figure 5 shown, the above Figure 2 shown embodiment can also be implemented as steps 510 to 550 below; among them, step 220 can also be implemented as steps 520 to 530 below.

[0211] Step 510, obtain the first state data and the reference action data respectively corresponding to the robot at multiple moments.

[0212] Among them, the first state data is the data obtained by performing preset state processing on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment; the reference action data is used to characterize the expected posture of the robot in the operating environment.

[0213] The content of step 510 has been described in the above steps 210 and 310, and will not be elaborated here.

[0214] Step 520, under the condition of the preset control strategy, perform encoding processing on the first observation data at the t-th moment and the reference action data at the t-th moment to obtain the encoded feature representation.

[0215] Schematically, the purpose of obtaining the reference action data is to enable the robot to better imitate the movement sequence from real organisms. Therefore, the process of the robot imitating the reference action data can be transformed into an encoder-decoder architecture, that is: the execution process of the preset control strategy is realized through the encoder-decoder architecture.

[0216] Among them, the encoder is used to map the input data (such as: the first observation data, the reference action data, etc.) into the latent space for representation to capture the key features of the input data, and the encoded feature representation output by it contains the abstract expression of the input data.

[0217] Among them, the decoder is used to map the encoded feature representation output by the encoder back to the original data space to generate a robot action similar to the reference action, that is, to output predicted action data for implementing the robot action.

[0218] In an optional embodiment, the first encoder performs first encoding processing on the first observation data at the t-th moment and the reference action data at the t-th moment to obtain a first feature representation.

[0219] Among them, the first encoder is used to perform imitation learning on the actions of the robot based on the reference action data.

[0220] Schematically, the first encoder is implemented as an Imitation Learning (IL) encoder. Imitation learning is a learning strategy. The IL encoder attempts to imitate or replicate the behavior observed from the reference action data. Usually, the IL encoder is responsible for mapping the first observation data and the reference action data to a latent representation, so that the output generated through this latent representation is similar to the behavior demonstrated by the reference action data.

[0221] Optionally, when analyzing the first observation data at the t-th moment and the reference action data at the t-th moment, the first observation data at the t-th moment and the reference action data at the t-th moment are input into the first encoder, so as to generate a first feature representation under the benchmark of the reference action data at the t-th moment and on the condition of imitating the behavior characterized by the reference action data at the t-th moment through the first observation data at the t-th moment.

[0222] In an optional embodiment, the second encoder performs second encoding processing on the first observation data at the t-th moment within a historical time period to obtain a second feature representation.

[0223] Among them, the second encoder is used to perform predictive analysis on the actions of the robot by means of prior knowledge, and the prior knowledge is the knowledge learned during the process of training the second encoder.

[0224] Optionally, the prior knowledge includes at least one of knowledge such as task structure knowledge, domain characteristic knowledge, or model prior expectations, etc.; in many cases, introducing prior knowledge through a prior encoder during the learning process can perform the learning task more effectively.

[0225] Schematically, the second encoder is implemented as a Prior encoder. Prior usually refers to the prior knowledge or expectation of the model for certain information. The Prior encoder is an encoder used to capture this prior knowledge; by inputting the first observation data into the Prior encoder, a latent representation can be mapped, and this latent representation contains prior information about the input data, thereby helping the preset control strategy to better utilize the prior knowledge to perform the task.

[0226] Optionally, when analyzing the first observation data at the t-th moment and the reference action data at the t-th moment, the first observation data at the t-th moment is input into the second encoder, so as to analyze the first observation data at the t-th moment with the help of prior knowledge and generate a second feature representation.

[0227] In an alternative embodiment, the first feature representation and the second feature representation are subjected to feature fusion to obtain an encoded feature representation.

[0228] Schematically, the first feature representation and the second feature representation are subjected to feature concatenation to obtain an encoded feature representation.

[0229] That is: the imitation information expressed by the first feature representation and the prior information expressed by the second feature representation can be concatenated to obtain an encoded feature representation that expresses deeper and more accurate information.

[0230] Step 530, based on the decoding process of the encoded feature representation, the predicted action data of the robot at the t-th moment is predicted.

[0231] Schematically, the encoded feature representation is decoded by a decoder to obtain the predicted action data of the robot at the t-th moment.

[0232] In an alternative embodiment, based on the encoded feature representation and the first observation data at the t-th moment, the predicted action data of the robot at the t-th moment is predicted.

[0233] Schematically, the encoded feature representation and the first observation data at the t-th moment are input into a decoder for decoding processing, and the predicted action data of the robot at the t-th moment is predicted.

[0234] Step 540, based on the predicted action data at the t-th moment and the first state data at the t-th moment, the state of the robot at the k-th moment is predicted to obtain second state data.

[0235] Wherein, the k-th moment is a moment after the t-th moment.

[0236] In an alternative embodiment, the state prediction process is performed by a trained world model, and the predicted action data at the t-th moment and the first state data at the t-th moment are input into the world model to obtain the second state data of the robot at the k-th moment.

[0237] Schematically, for the sequential execution process introduced above, after obtaining the world model through data corresponding to multiple moments, the predicted action data at the t-th moment and the first state data at the t-th moment are input into the world model to obtain the second state data at the k-th moment; or, for the alternating execution process introduced above, when determining the second state data at the k-th moment, determine the partial moments to which the t-th moment belongs. For example, if the t-th moment is a moment within the first partial moment, after obtaining the world model after the first training through the data corresponding to the first partial moment, the predicted action data at the t-th moment and the first state data at the t-th moment are input into the world model after the first training to obtain the second state data at the k-th moment, and so on.

[0238] It should be noted that the above is only a schematic example, and the embodiments of the present application are not limited thereto.

[0239] Step 550: Based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, train the preset control strategy to obtain the trained control strategy.

[0240] Among them, the trained control strategy is used to control the actions of the robot.

[0241] As Figure 6 shown, it is a schematic diagram for training the preset control strategy and obtaining the trained control strategy.

[0242] Based on the trained control strategy being a trained strategy, if it is implemented as a model or algorithm, it has the same network structure or algorithm expression as the preset control strategy. Therefore, the strategies before and after training can be collectively referred to as the control strategy. The control strategy is implemented as a combination of the IL encoder 610, the Prior encoder 620, and the motor decoder 630. Therefore, the process of training the control strategy can be regarded as a process of optimizing and adjusting the network parameters corresponding to the IL encoder 610, the Prior encoder 620, and the motor decoder 630 respectively.

[0243] Schematically, the input of the IL encoder 610 includes the first state data s t corresponding first observation data o t , and also includes the reference action data q t . The output of the IL encoder 610 is the first feature representation; the input of the Prior encoder 620 is the first observation data o t . The output of the Prior encoder 620 is the second feature representation; after fusing the first feature representation and the second feature representation, the encoded feature representation z t is obtained; by using the encoded feature representation z tInput the electric decoder 630 to decode and obtain the predicted action data a t ; in addition, use the world model 640 to perform state prediction on the predicted action data a t and the first state data s t to obtain the predicted second state data Finally, train the control strategy based on the loss value between the predicted second state data and the reference action data, that is, optimize and adjust the network parameters corresponding to the IL encoder 610, the Prior encoder 620, and the electric decoder 630 respectively until the trained control strategy is obtained.

[0244] Among them, model the prior distribution p(z t |o t ) and the posterior distribution q(z t |o t , q t ) as Gaussian distributions, as shown in Formula 3 and Formula 4 below.

[0245] Formula 3:

[0246] p(z t |o t ) ~ N(π prior (z t |o t ), σ 2 I)

[0247] Formula 4:

[0248] q(z t |o t , q t ) ~ N(π IL (z t |o t , q t ) + π prior (z t |o t ), σ 2 I)

[0249] Among them, N() represents the Gaussian distribution; π prior (z t |o t ) represents the parameterization of the neural network θ prior characterized by the Prior encoder 620; π IL (z t |o t , q t ) represents the parameterization of the neural network θ IL characterized by the IL encoder 610; σ represents the fixed standard deviation; I is the identity matrix.

[0250] Among them, the predicted second state data The loss function of the loss value between the reference action data is implemented as Formula Five below.

[0251] Formula Five:

[0252]

[0253] Among them, is the loss value at the t-th moment; is the joint position loss; is the joint velocity loss; is the robot position loss; is the robot velocity loss.

[0254] Joint position loss As shown in Formula Six below.

[0255] Formula Six:

[0256]

[0257] Among them, is the predicted joint position in the second state data at the t-th moment; is the joint position in the reference action data at the t-th moment.

[0258] Joint velocity loss As shown in Formula Seven below.

[0259] Formula Seven:

[0260]

[0261] Among them, is the predicted joint velocity in the second state data at the t-th moment;<00007??>is the joint velocity in the reference action data at the t-th moment.

[0262] Robot position loss As shown in Formula Eight below.

[0263] Formula Eight:

[0264]

[0265] Among them, is the predicted robot body position (e.g., the mass point position of the robot) in the second state data at the t-th moment; is the robot body position in the reference action data at the t-th moment; It should be noted that there seems to be an error in the original text where the number in is incorrect. It is assumed to be for the translation.is the predicted robot body direction in the second state data at the t-th moment (e.g., the direction of the mass point of the robot); is the robot body direction in the reference action data at the t-th moment.

[0266] Robot speed loss As shown in Formula Nine below.

[0267] Formula Nine:

[0268]

[0269] Wherein, is the predicted robot body speed in the second state data at the t-th moment (e.g., the speed of the mass point of the robot); is the robot body speed in the reference action data at the t-th moment; is the predicted robot body angular velocity in the second state data at the t-th moment (e.g., the angular velocity of the mass point of the robot); is the robot body angular velocity in the reference action data at the t-th moment.

[0270] It should be noted that the above are only illustrative examples, and the embodiments of the present application are not limited thereto.

[0271] In summary, by means of the collected first observation data, the operation situation of the robot in the physical world can be fully utilized for an accurate supervised learning process. Under the limitation of the reference action data, the predicted action data is obtained through a preset control strategy. Thus, the second state data at a later moment can be predicted by means of the predicted action data and the first state data. The preset control strategy can be adjusted specifically through the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, and thus enable the robot to have more accurate motion accuracy and more stable operation ability during the operation process.

[0272] In the embodiments of the present application, the content of determining the predicted action data by means of the way of obtaining the encoded feature representation through mapping is introduced. With the preset control strategy implemented as an encoder-decoder architecture, the data obtained is deeply analyzed by the encoder to improve the acquisition accuracy of the encoded feature representation, and the data information of the robot can be obtained more comprehensively and deeply. Thus, the predicted action data is obtained through the decoding process of the decoder, the standardization of action prediction is improved, the flexibility of the robot learning process is improved, and the performance and adaptability of the control strategy are also enhanced.

[0273] In an optional embodiment, when obtaining predicted action data through a preset control strategy, motion instruction data can also be obtained so that the robot can learn information about following instructions to move from it, thereby enriching the diversity when obtaining predicted motion data and facilitating the application of the robot in multiple motion scenarios. Schematically, as Figure 7 shown, step 220 shown above Figure 2 can also be implemented as steps 710 to 750 as follows.

[0274] Step 710, obtain motion instruction data corresponding to multiple moments respectively.

[0275] Among them, the motion instruction data is used to represent the data for guiding the robot to execute the motion process.

[0276] Optionally, the motion instruction data is implemented as at least one of multiple data values such as linear velocity, angular velocity, position, direction, etc. The motion instruction data can provide a general direction and speed for the robot to move. If the motion instruction data is implemented as the values of linear velocity and angular velocity, the robot's movement, rotation, etc. in the environment can be known through the motion instruction data.

[0277] Schematically, the motion instruction data is the guiding information transmitted from the outside to the robot, which is convenient for knowing the robot to execute specific actions. For example: the motion instruction data comes from a human operator, an upper decision-making system, a remote controller or other automation systems, and to a certain extent represents the expectation for the robot to execute a certain action.

[0278] Among them, each of the multiple moments corresponds to a motion instruction data, thereby representing the situation of guiding the robot to move at the current moment.

[0279] In some embodiments, the desired linear velocity and the desired angular velocity are randomly obtained to form the motion instruction data.

[0280] Schematically, a random motion instruction data is given as where represents the desired linear velocity at any moment t, represents the desired angular velocity at any moment t. That is: in the process of training the preset control strategy, the strategy training process can be executed with random motion instruction data.

[0281] Step 720, under the condition of the preset control strategy, perform third encoding processing on the motion instruction data through a third encoder to obtain a third feature representation.

[0282] Schematically, the preset control strategy is implemented as an encoder-decoder architecture. The encoder is used to map the input data input therein into a latent space for representation to capture the key features of the input data, and the encoded feature representation output by it contains the abstract expression of the input data; the decoder is used to map the encoded feature representation output by the encoder back to the original data space to generate a robot action similar to the reference action, that is, to output the predicted action data for implementing the robot action.

[0283] Optionally, the preset control strategy includes a third encoder. The third encoder is used to analyze the motion situation of the robot following the motion instruction data based on the motion instruction data.

[0284] Schematically, the third encoder is implemented as a Command Following (CF) encoder. The CF encoder is responsible for encoding the given motion instruction data (for example: linear velocity, angular velocity) into a third feature representation. The purpose of this encoding is to map the motion instruction data into a latent space, which contains the motion instruction data and information that may be the first observation data; afterwards, through the third feature representation generated by the CF encoder, the motion situation of the robot can be analyzed to analyze whether the robot has successfully complied with the motion instruction characterized by the motion instruction data, and whether there are deviations or errors during the execution process.

[0285] That is: the third feature representation can be used to evaluate the understanding degree and execution accuracy of the robot for the motion instruction data.

[0286] Optionally, when predicting the predicted action data at the t-th moment, the motion instruction data at the t-th moment is input into the third encoder to generate the third feature representation at the t-th moment.

[0287] Introducing the third encoder to analyze and learn the motion situation of the robot following the motion instruction data based on the motion instruction data, so as to further improve the acquisition accuracy of the encoded feature representation, has a certain authorization prospect.

[0288] In an optional embodiment, when the preset control strategy includes a second encoder and a third encoder, the following steps 731 to 732 are executed.

[0289] Step 731, perform a second encoding process on the first observation data at the t-th moment through the second encoder to obtain a second feature representation.

[0290] Schematically, the preset control strategy includes a second encoder. The second encoder is used to predict and analyze the action with the help of prior knowledge, and the prior knowledge is the knowledge learned during the process of training the second encoder.

[0291] Optionally, the second encoder is implemented as a Prior encoder, and the first observation data at the t-th moment is input into the second encoder to analyze the first observation data at the t-th moment with the help of prior knowledge and generate a second feature representation.

[0292] Step 732: Perform feature fusion on the second feature representation and the third feature representation to obtain an encoded feature representation.

[0293] Schematically, the third feature representation and the second feature representation are concatenated to obtain an encoded feature representation.

[0294] In an optional embodiment, when the preset control strategy includes a first encoder, a second encoder, and a third encoder, the following steps 741 to 742 are executed.

[0295] Step 741: Under the conditions of the preset control strategy, perform a first encoding process on the first observation data at the t-th moment through the first encoder to obtain a first feature representation; perform a second encoding process on the first observation data at the t-th moment through the second encoder to obtain a second feature representation.

[0296] Schematically, the content of step 741 has been described in step 520 above and will not be elaborated here.

[0297] Step 742: Perform feature fusion on the first feature representation, the second feature representation, and the third feature representation to obtain an encoded feature representation.

[0298] Schematically, the first feature representation, the second feature representation, and the third feature representation are concatenated to obtain an encoded feature representation. This encoded feature representation can fully combine the instruction following information, imitation learning information, and prior knowledge information, and has more accurate characteristics.

[0299] Step 750: Based on the decoding process of the encoded feature representation, predict the predicted action data of the robot at the t-th moment.

[0300] Schematically, use the decoder in the preset control strategy to perform the decoding process on the encoded feature representation, so as to predict the predicted action data of the robot at the t-th moment.

[0301] In an optional embodiment, the world model is used to perform state analysis on the predicted action data at the t-th moment to obtain the second state data at the k-th moment, and then the preset control strategy is trained according to the second state data at the k-th moment and the reference state data at the k-th moment.

[0302] As Figure 8 shown, it is a schematic diagram for training the preset control strategy and obtaining the trained control strategy.

[0303] Based on the trained control policy being the trained policy, if implemented as a model or algorithm, it is the same as the network structure or algorithm expression of the preset control policy. Therefore, the policies before and after training can be collectively referred to as the control policy.

[0304] Taking the control policy as being implemented as a combination of the CF encoder 810, the Prior encoder 820, and the motor decoder 830, the process of training the control policy can be regarded as a process of optimizing and adjusting the network parameters corresponding to the CF encoder 810, the Prior encoder 820, and the motor decoder 830 respectively.

[0305] Schematically, the input of the CF encoder 810 is the motion command data c t , and the output is the third feature representation; the input of the Prior encoder 820 is the first state data s t and the corresponding first observation data o t , and the output of the Prior encoder 820 is the second feature representation; after fusing the third feature representation and the second feature representation, the encoded feature representation z t is obtained; by inputting the encoded feature representation z t into the motor decoder 830, the predicted action data a t is decoded; in addition, with the help of the world model 840, state prediction is performed on the predicted action data a t and the first state data s t to obtain the predicted second state data Finally, the control policy is trained through the following loss value between the predicted second state data and the motion command data, that is, the network parameters corresponding to the CF encoder 810, the Prior encoder 820, and the motor decoder 830 are optimized and adjusted until the trained control policy is obtained.

[0306] Among them, the posterior distribution q(z t |o t , q t ) is modeled as a Gaussian distribution, as shown in Equation Ten below.

[0307] Equation Ten:

[0308] q(z t |o t , c t ) ~ N(π prior (z t |o t ) + π CF (z t |o t , c t ), σ2 I)

[0309] where N() represents a Gaussian distribution; π prior (z t |o t ) represents the parameterization of the neural network θ represented by the Prior encoder 820; π prior ; π CF (z t |o t , c t ) represents the parameterization of the neural network θ represented by the CF encoder 810; σ represents a fixed standard deviation; I is the identity matrix. CF Taking the action instruction data of angular velocity and linear velocity as an example, since the training objective is to make the robot follow the action instruction represented by the action instruction data, the following loss value includes the linear velocity loss

[0310] and the angular velocity loss and the angular velocity loss

[0311] where the following loss value, the linear velocity loss and the angular velocity loss are related as shown in Equation (11) below.

[0312] Equation (11):

[0313]

[0314] where the linear velocity loss is as shown in Equation (12) below.

[0315] Equation (12):

[0316]

[0317] where represents the linear velocity in the motion following data at the t-th moment; is the predicted linear velocity in the second state data at the t-th moment.

[0318] where the angular velocity loss is as shown in Equation (13) below.

[0319] Equation (13):

[0320]

[0321] where represents the angular velocity in the motion following data at the t-th moment; is the predicted angular velocity in the second state data at the t-th moment.

[0322] In some embodiments,Figure 6 and Figure 8 The schematic diagrams respectively showing the training of a preset control strategy and obtaining the trained control strategy can be used in combination. That is: The preset control strategy includes an IL encoder, a Prior encoder, a CF encoder, and an electric decoder.

[0323] That is: When obtaining the encoded feature representation z t a first feature representation is output by the IL encoder, a second feature representation is output by the Prior encoder, and a third feature representation is output by the CF encoder, so as to fuse and obtain the encoded feature representation z t When; and then based on Figure 6 the obtained loss value (which can also be called the imitation loss value), the IL encoder, the Prior encoder, and the electric decoder are trained, and based on Figure 8 the obtained following loss value, the CF encoder, the Prior encoder, and the electric decoder are trained, and the trained control strategy and the like are obtained.

[0324] It should be noted that the above is only a schematic example, and the embodiments of the present application are not limited thereto.

[0325] In summary, by means of the collected first observation data, the accurate supervised learning process can be fully utilized based on the running situation of the robot in the physical world. Under the limitation of the reference action data, the predicted action data is obtained through the preset control strategy, so that the second state data at the subsequent moment can be predicted by means of the predicted action data and the first state data. The preset control strategy can be adjusted specifically through the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, so that the robot can have more accurate motion accuracy and more stable running ability during the running process.

[0326] In the embodiments of the present application, during the process of obtaining the motion instruction data to participate in generating the encoded feature representation, the third encoder analyzes the motion situation of the robot following the instruction based on the motion instruction data, so that when the robot moves through the control strategy, it can learn the following information of following the instruction, which is convenient for the robot to adapt to various motion scenarios and improve the motion adaptation flexibility of the robot.

[0327] In an alternative embodiment, the above robot control method can be applied to the usage scenario of a quadruped robot, and the above robot control method can also be called a quadruped robot control method for effectively learning agile motion skills based on a model.

[0328] First, a brief introduction to the overall framework of the method is given. The overall framework consists of two parts, namely: the world model and the control strategy. The world model learns to approximate the unknown dynamics of simulation and reality, and can predict the next state (e.g., the second state data) given the current robot state (e.g., the first state data) and action (e.g., the first action data in the above training, or the reference action data during application). The control strategy learns flexible behaviors by mimicking the movements of real animals, and it can directly collect and analyze the samples predicted by the well-trained world model.

[0329] Both the world model and the control strategy are updated and iteratively trained in a supervised manner: First, state-action pairs can be collected under a fixed control strategy to adapt to the system dynamics, and the world model is used for fitting (i.e., training the world model); then the control strategy is updated and trained by interacting with the fixed world model. The whole process is repeated until the control strategy converges.

[0330] Schematically, the following several parts are used to illustrate the robot control method.

[0331] (1) World Model

[0332] Schematically, starting from training the world model f w it predicts the next state based on the current state and action, using the residual form as shown in Formula 1 above.

[0333] When training the world model, the robot collects state-action sequences under a preset control strategy; the training of the world model is a supervised learning method with an n-step prediction loss, which is beneficial for long-term prediction, as shown in Formula 2 above.

[0334] (2) Imitation Learning

[0335] In the context of the imitation task, the goal of the embodiment of this application is to imitate the motion sequences collected from real animals. The preset control strategy can be transformed into an encoder-decoder architecture, for example: the architecture of Variational Auto-Encoders (VAE) is used for the analysis process. As shown in Formulas 3 to 9 above.

[0336] Optionally, in order to ensure that the latent space is well formed so that suitable encoded feature representations can be further found in the downstream command following task, the relative entropy (Kullback-Leible, KL) divergence regularization loss as shown in Formula 14 below can be added.

[0337] Formula 14:

[0338]

[0339] Where, Represents the divergence loss value; D KL Represents the prior distribution p(z t |o t ) and the difference between the posterior distribution q(z t |o t , q t ); π IL (o t , q t ) represents the parameterization of the neural network θ IL of the IL encoder representation; σ represents a fixed standard deviation.

[0340] (3) Instruction following task

[0341] Schematically, a policy that follows the linear and angular velocities specified by the user can be trained. By introducing the instruction following encoder π CF (z t |o t , c t ), the action instruction data is encoded into the latent space.

[0342] In some embodiments, in order to maintain the naturalness of the robot's motion behavior, only the network parameters corresponding to the instruction following encoder can be updated during training, while keeping the network parameters of the prior network and the motor decoder unchanged.

[0343] (4) Fine-tuning on a real robot

[0344] Due to the gap between simulation and reality, the policy learned from simulation may fail when deployed to a real robot. Therefore, the instruction following encoder and the motor decoder can be fine-tuned on the real robot to follow the desired path. To maintain the natural behavior of the original motor encoder , a regularization term can be introduced for regularization, as shown in Formula XV below.

[0345] Formula XV:

[0346]

[0347] Wherein, represents the regularization term; represents the parameterization of the neural network of the motor encoder representation; π M (a t |o t , z t ) represents the adjusted network parameters.

[0348] It should be noted that the above is only a schematic example, and the embodiments of the present application are not limited thereto.

[0349] In an optional embodiment, to evaluate the effectiveness of the above-mentioned robot control method, a comparative experiment was conducted in the reinforcement learning environment Isaac Gym and on a real quadruped robot. The purpose of the experiment was to answer the following key questions.

[0350] (1) The improvement in sample efficiency of the robot control method compared with the reinforcement learning method.

[0351] (2) The effect of the fine-tuning process conducted on the real robot in reducing the simulation-to-reality gap.

[0352] (3) The generalization ability of the fine-tuned policy on tasks not involved in previous training.

[0353] Among them, the following three experimental processes were conducted for the above three questions.

[0354] (1) Experiments were conducted both in the simulation world and the physical world to compare the sample efficiency of the robot control method provided by the embodiments of the present application with the benchmark method based on reinforcement learning;

[0355] (2) In the physical world experiment, a fine-tuning process was conducted on a real quadruped robot to demonstrate the real gap effect;

[0356] (3) To further demonstrate its generalization ability, the robot control method provided by the embodiments of the present application was also used to perform a path following task on four unseen paths.

[0357] Schematically, the following content describes the simulation experiment and the physical world experiment.

[0358] (1) Simulation Experiment

[0359] (1) Sample efficiency in the imitation learning task.

[0360] Schematically, to solve the first question regarding sample efficiency, the participating comparative models were trained from scratch for the imitation task using the reinforcement learning environment Isaac Gym. Isaac Gym is a high-performance physics simulator based on a Graphics Processing Unit (GPU) for robot learning that can simulate a batch of robots simultaneously. Optionally, 128 agents can be used for training in this task.

[0361] Optionally, the robot control method provided by the embodiments of the present application was compared with the Proximal Policy Optimization (PPO) algorithm in terms of the number of samples collected from Isaac Gym. The reward function of the PPO algorithm is defined as r t is the reward function at time step t; is the calculated loss value.

[0362] Schematically, the same policy network structure is maintained for two methods (the method of the embodiment of the present application and the PPO algorithm) for meaningful comparison.

[0363] As Figure 9 shown, for the average reward during training, curve 910 shows the method of the embodiment of the present application, where an average reward of 0.8 is achieved with approximately 5 million samples — Figure 9 as shown by the dashed line in; in contrast, curve 920 shows the PPO algorithm, where more than 70 million samples are required to achieve a similar result. This shows that the method of the embodiment of the present application is more than ten times more sample-efficient than the PPO algorithm. Among them, the gray area distributed in the weekly test represents the discrete data distribution.

[0364] (2) Comparison of sample efficiency in adapting to a new environment.

[0365] Schematically, directly training the PPO algorithm on a real robot is dangerous and may easily damage the robot. Therefore, a method of changing physical parameters and fine-tuning in simulation can be introduced for training.

[0366] Optionally, some physical parameters can be changed for the imitation task. As shown in Table 1 below, for the physical parameter sides of the original environment (the environment used when training the robot) and the test environment (which may be the same as or different from the original environment), the test environment shows Environment 1, Environment 2, Environment 3, and Environment 4.

[0367] Table 1

[0368]

[0369]

[0370] For example: in Environment 1, the mass of the robot is increased from 5.74 kg to 14 kg. Since significantly changing the mass of the robot itself may make it impossible for the robot to apply the original policy, that is: making the original policy extremely difficult in the new environment.

[0371] In some embodiments, in order to simulate a situation similar to robot data collection in the physical world, two robots can be used in the simulation environment.

[0372] Schematically, each training iteration accumulates 3000 samples, which is equivalent to collecting data for 1 minute at a control frequency of 50 Hz. For the PPO algorithm, the policy is updated every 32 steps.

[0373] As Figure 10 shown, the training curve is presented. Figure 10 In curve 1010 in [reference], it highlights that the method of the embodiment of the present application can achieve an average reward of 0.8 using approximately 50,000 samples (equivalent to about 17 minutes of data) in this challenging environment; in contrast, Figure 10 the PPO algorithm represented by curve 1020 in [reference] still performs poorly even with ten times the sample size. Among them, the gray area distributed in the weekly test represents the discrete data distribution.

[0374] In some embodiments, to further study the performance of instruction following, the task can be extended to path following, where the robot aims to travel along a predefined path.

[0375] As Figure 11 shown, it is a schematic diagram of four desired trajectories, including trajectory 1110 (Oblong), trajectory 1120 (Lemniscate), trajectory 1130 (U - shape), and trajectory 1140 (Star); among them, the arrow 1150 on each trajectory represents the initial position.

[0376] Optionally, a Purely Reactive Algorithm (PR algorithm) is adopted to convert path information into instructions (motion instruction data).

[0377] Schematically, taking the example of following the rectangular trajectory shown by trajectory 1110 at a target speed of 0.9 m / s, and trajectory 1110 is the trajectory involved in training, the motion analysis can be carried out using the three environments shown in Table 1 above.

[0378] Optionally, to simulate the fine - tuning process in the physical world, each training iteration involves collecting 1500 samples (30 - second data).

[0379] As Figure 12 shown, the training curve of loss 1210 is described. It can be observed from Figure 12 that under workloads of 3 kg, 5 kg, and 7 kg, the method of the embodiment of the present application requires approximately 4 iterations (2 minutes), 6 iterations (3 minutes), and 8 iterations (4 minutes) of data to achieve a loss below 0.6. These results indicate relatively good performance at these speeds. Among them, there may still be some discrete data distributions, which are not shown in the figure; in contrast, the loss of the PPO algorithm remains almost unchanged with such a limited sample size, so the results are not drawn.

[0380] That is: through the method of the embodiments of the present application, the high sample efficiency and adaptability of the method in imitation learning and path following tasks for different environments can be demonstrated.

[0381] (2) Real Machine Experiment

[0382] (1) Adaptation from simulation to reality.

[0383] To answer the second question above, physical experiments can be conducted using real robots. Due to the gap between simulation and reality, the strategies trained in simulation may not be able to follow the path at the desired speed and may exhibit significant speed lag at high target speeds, which emphasizes the necessity of fine-tuning in the physical world.

[0384] Optionally, three adaptation experiments were conducted on the trajectory 1110 as Figure 11 shown, with target speeds of 0.6 m / s, 0.9 m / s, and 1.2 m / s respectively; to fine-tune the strategy in the real world, 30 seconds of data (1500 samples) need to be collected for each iteration to train the world model, and then the strategy network is updated using the data predicted by the adapted world model.

[0385] As Figure 13 shown, it is a change graph of the command following loss 1310 for four iterations (2 minutes of data) on a real robot with target speeds of 0.6 m / s, 0.9 m / s, and 1.2 m / s.

[0386] As shown in Table 2 below, for the average linear velocity error calculated in a 30-second trajectory and the angular velocity loss after each iteration of adaptation in the physical world, Policy 0 is the original policy, and Policies 1 to 4 are other comparison policies.

[0387] Table 2

[0388]

[0389] As shown in Table 2, the loss decreased significantly after the first iteration. Especially for the case of a speed of 1.2 m / s, the speed error decreased by more than 0.26 m / s. After four iterations, the loss converged, and the final performance was very effective in terms of command following.

[0390] As Figure 14As shown, it shows following at a speed of 1.2 m / s for real-world adaptation on a real robot. Obviously, in the original policy (iteration 0), the actual speed lags significantly behind the target speed. After the first iteration, the actual speed can follow the target to some extent, but there are significant fluctuations. In the fourth iteration, the policy effectively accompanies the target speed with minimal vibration. To avoid data confounding, Figure 14 The statistics of 0 iterations, 1 iteration, and 4 iterations are shown in Figure 14 , where the data fluctuations are smaller after 4 iterations compared to the larger data fluctuations in the case of 1 iteration.

[0391] That is: After real-world adaptation on a real robot, it moves along the path of trajectory 1110 at a speed of 1.2 meters per second.

[0392] (2) Generalization on unknown terrains.

[0393] Schematically, to answer the last question above, the performance of the speed and path of the policy of the embodiment of the present application being evaluated on unseen motion instruction data is evaluated. In the previous experiment, real robot data was collected, a total of 7.5 minutes, with target speeds of 0.6 m / s, 0.9 m / s, and 1.2 m / s.

[0394] Optionally, these data are used for offline fine-tuning to obtain an adaptive policy.

[0395] As Figure 11 shown, the performance on all paths is tested, including the unseen paths of trajectory 1120, trajectory 1130, and trajectory 1140, and also including the generalization ability for unseen target speeds of 0.7 m / s, 0.8 m / s, and 1.0 m / s.

[0396] As shown in Table 3 below, the average linear velocity error (e v ), angular velocity error (e ω ), and distance error (e p ) calculated on four paths each lasting 30 seconds are shown; where the distance error is defined as p t , where p and p

[0397] are the position of the robot and the target position at time t respectively;

[0398]

[0399] It can be seen from Table 3 that after offline fine-tuning, all errors are reduced by more than half. Figure 15Vividly shows the speed accompaniment of the real robot along trajectory 1110 under the original strategy 1510 and the adapted strategy 1520. The original strategy lags behind the target unseen speed, while the trained control strategy obtained by the embodiments of the present application can effectively accompany them, with an average linear velocity error of about 0.05 m / s.

[0400] Figure 16 Shows the real trajectories of the commanded following paths at different unseen target speeds, including the long circle trajectory 1610 involved in training, and also including the lemniscate trajectory 1620, U-shaped trajectory 1630, and star trajectory 1640 that are not involved in training. From Figure 16 it can be seen that the original strategy (preset training strategy) significantly lags behind the reference trajectory, while the trained control strategy obtained by the embodiments of the present application can effectively follow it and even perform faster at higher speeds. In summary, the experimental results show that the trained control strategy can successfully handle unseen commands and follow unfamiliar paths, highlighting the generalization ability of the embodiments of the present application.

[0401] Optionally, the above-mentioned motion control technology of the trained quadruped robot can be applied to at least one of the following scenarios.

[0402] (1) Exploration and rescue: The quadruped robot can operate in various harsh and complex environments, such as disaster sites, fires, earthquakes, etc., to provide assistance to rescue workers.

[0403] (2) Agriculture: The quadruped robot can walk in the farmland to help farmers complete tasks such as sowing and harvesting.

[0404] (3) Industrial production: The quadruped robot can carry heavy objects in the factory to help workers complete repetitive tasks.

[0405] (4) Medical field: The quadruped robot can help disabled people walk and provide support, and can also be used for rehabilitation therapy, guiding the blind, etc.

[0406] (5) Entertainment and education: The quadruped robot can be used as a toy or an educational tool to help children learn scientific and technological knowledge and skills.

[0407] It should be noted that the above scenarios are only illustrative examples, and the embodiments of the present application are not limited thereto.

[0408] In summary, by leveraging the first observation data obtained through collection, it is possible to fully utilize the operating conditions of the robot in the physical world for an accurate supervised learning process. Under the limitation of the reference action data, the predicted action data can be obtained through a preset control strategy. Thus, it is possible to predict the second state data at a subsequent moment by means of the predicted action data and the first state data. The preset control strategy can be adjusted specifically based on the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control strategy, improve the strategy stability and strategy application accuracy of the trained control strategy, and thereby enable the robot to have more accurate motion accuracy and more stable operating ability during operation.

[0409] In the embodiments of the present application, the introduced robot control method trains the world model and the control strategy in a supervised manner, thereby significantly improving the sample efficiency; a two-stage method can also be adopted, which involves training the strategy in simulation and fine-tuning on the real machine, and then less data can be used for fine-tuning in the physical world. This process significantly reduces the required amount of real-world data and makes it possible to learn more complex motion skills.

[0410] Figure 17 It is a structural block diagram of a robot control device provided by an exemplary embodiment of the present application, as Figure 17 shown. The device includes the following parts:

[0411] A data acquisition module 1710, configured to acquire the first state data and the reference action data respectively corresponding to the robot at multiple moments. The first state data is the data obtained by performing a preset state process on the first observation data, and the first observation data is the data collected by the robot through sensors in the operating environment where it is located. The reference action data is used to represent the expected posture of the robot in the operating environment.

[0412] An action prediction module 1720, configured to predict the predicted action data of the robot at the t-th moment based on the first observation data at the t-th moment and the reference action data at the t-th moment among the multiple moments under the condition of a preset control strategy, where t is a positive number.

[0413] A state prediction module 1730, configured to predict the state of the robot at the k-th moment based on the predicted action data at the t-th moment and the first state data at the t-th moment, so as to obtain the second state data at the k-th moment, where the k-th moment is a moment after the t-th moment among the multiple moments.

[0414] A policy training module 1740, configured to train the preset control policy based on a loss value between the second state data at the k-th moment and the reference action data at the k-th moment, so as to obtain a trained control policy; wherein, the trained control policy is used to control the actions of the robot.

[0415] In an alternative embodiment, the state prediction module 1730 is further configured to obtain a world model, which is used to predict the state of the robot and is a model trained based on the first state data and the first observation data; perform state prediction on the predicted action data and the first state data at the t-th moment through the world model to obtain the second state data at the k-th moment.

[0416] In an alternative embodiment, the state prediction module 1730 is further configured to obtain an environment simulation model, which is a model to be trained to obtain the world model and is used to predict the predicted state data at the j-th moment according to the first state data at the i-th moment and the first observation data at the i-th moment, where the j-th moment is a moment after the i-th moment among multiple moments, and i and j are positive numbers; obtain a prediction loss value based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment; train the environment simulation model through the prediction loss value to obtain the world model.

[0417] In an alternative embodiment, the state prediction module 1730 is further configured to obtain a state loss value corresponding to the j-th moment based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment; sum the state loss values corresponding to the multiple moments respectively to obtain the prediction loss value.

[0418] In an alternative embodiment, the action prediction module 1720 is further configured to perform encoding processing on the first observation data at the t-th moment and the reference action data at the t-th moment under the condition of a preset control policy to obtain an encoded feature representation; predict the predicted action data of the robot at the t-th moment based on decoding processing of the encoded feature representation.

[0419] In an alternative embodiment, the action prediction module 1720 is further configured to perform a first encoding process on the first observation data at the t-th moment and the reference action data at the t-th moment through a first encoder to obtain a first feature representation, where the first encoder is used to perform imitation learning on the actions of the robot based on the reference action data; perform a second encoding process on the first observation data at the t-th moment through a second encoder to obtain a second feature representation, where the second encoder is used to perform predictive analysis on the actions of the robot by means of prior knowledge, and the prior knowledge is the knowledge learned during the training of the second encoder; the preset control strategy includes the first encoder and the second encoder; perform feature fusion on the first feature representation and the second feature representation to obtain the encoded feature representation.

[0420] In an alternative embodiment, the action prediction module 1720 is further configured to obtain the motion instruction data respectively corresponding to the multiple moments, where the motion instruction data is used to characterize the data for guiding the robot to execute a motion process; under the condition of the preset control strategy, perform a third encoding process on the motion instruction data through a third encoder to obtain a third feature representation, where the third encoder is used to analyze the motion situation of the robot following the instruction based on the motion instruction data; perform a second encoding process on the first observation data at the t-th moment through a second encoder to obtain a second feature representation, where the second encoder is used to perform predictive analysis on the actions of the robot by means of prior knowledge, and the prior knowledge is the knowledge learned during the training of the second encoder; based on the second feature representation and the third feature representation, obtain the encoded feature representation.

[0421] In an alternative embodiment, the action prediction module 1720 is further configured to perform feature fusion on the second feature representation and the third feature representation to obtain the encoded feature representation, and the preset control strategy includes the second encoder and the third encoder; or, perform a first encoding process on the first observation data at the t-th moment and the reference action data at the t-th moment through a first encoder to obtain a first feature representation, where the first encoder is used to perform imitation learning on the actions of the robot based on the reference action data; perform feature fusion on the first feature representation, the second feature representation, and the third feature representation to obtain the encoded feature representation, and the preset control strategy includes the first encoder, the second encoder, and the third encoder.

[0422] In an alternative embodiment, the action prediction module 1720 is further configured to perform decoding processing on the encoded feature representation through a decoder in the preset control strategy, and output the predicted action data of the robot at the t-th moment.

[0423] In an alternative embodiment, the policy training module 1740 is further configured to obtain the loss value corresponding to the k-th moment based on the difference between the second state data at the k-th moment and the reference action data at the k-th moment; and adjust the policy parameters in the preset control policy with the loss value to obtain the trained control policy.

[0424] In an alternative embodiment, the policy training module 1740 is further configured to obtain the loss values corresponding to the multiple moments respectively; and iteratively adjust the policy parameters in the preset control policy with the multiple loss values to obtain the trained control policy.

[0425] In an alternative embodiment, the data acquisition module 1710 is further configured to obtain the first observation data collected by the robot at the multiple moments respectively; perform a preset state transformation on the first observation data corresponding to the multiple moments respectively to obtain the first state data corresponding to the multiple moments respectively; and obtain a reference action sequence, where the reference action sequence is used to represent the expected pose sequence of the robot in the operating environment, and the reference action sequence includes the reference action data corresponding to the multiple moments respectively.

[0426] In summary, by means of the collected first observation data, an accurate supervised learning process can be fully utilized based on the running conditions of the robot in the physical world. Under the limitation of the reference action data, the predicted action data can be obtained through the preset control policy, so that the second state data at the subsequent moment can be predicted with the predicted action data and the first state data. The preset control policy can be adjusted specifically with the loss value between the second state data and the reference action data, so as to obtain more accurate predicted action data through the trained control policy, improve the policy stability and policy application accuracy of the trained control policy, and thus enable the robot to have more accurate motion accuracy and more stable running ability during the running process.

[0427] It should be noted that: for the robot control device provided in the above embodiment, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the robot control device provided in the above embodiment and the embodiment of the robot control method belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0428] Figure 18The structural schematic diagram of a server provided by an exemplary embodiment of the present application is shown. The server 1800 includes a Central Processing Unit (CPU) 1801, a system memory 1804 including a Random Access Memory (RAM) 1802 and a Read Only Memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the central processing unit 1801. The server 1800 further includes a mass storage device 1806 for storing an operating system 1813, application programs 1814, and other program modules 1815.

[0429] The mass storage device 1806 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1806 and its associated computer-readable medium provide non-volatile storage for the server 1800. That is to say, the mass storage device 1806 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read Only Memory (CD-ROM) drive.

[0430] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The above-mentioned system memory 1804 and mass storage device 1806 may be collectively referred to as memory.

[0431] According to various embodiments of the present application, the server 1800 may also be run by connecting to a remote computer on a network such as the Internet. That is, the server 1800 may be connected to the network 1812 through a network interface unit 1811 connected to the system bus 1805, or in other words, the network interface unit 1811 may also be used to connect to other types of networks or remote computer systems (not shown).

[0432] The above-mentioned memory further includes one or more programs, and one or more programs are stored in the memory and are configured to be executed by the CPU.

[0433] An embodiment of the present application further provides a computer device, which includes a processor and a memory. At least one instruction, at least one segment of program, code set, or instruction set is stored in the memory, and the at least one instruction, at least one segment of program, code set, or instruction set is loaded and executed by the processor to implement the robot control method provided by the above-mentioned method embodiments.

[0434] An embodiment of the present application further provides a computer-readable storage medium, on which at least one instruction, at least one program segment, a code set or an instruction set is stored. The at least one instruction, at least one program segment, the code set or the instruction set is loaded and executed by a processor to implement the robot control method provided in each of the above method embodiments.

[0435] An embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the robot control method described in any one of the above embodiments.

[0436] The foregoing are only optional embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method, characterized in that, The method includes: Obtaining first state data and reference action data respectively corresponding to a robot at multiple moments. The first state data is data obtained by performing preset state processing on first observation data, and the first observation data is data collected by a sensor by the robot in its operating environment. The reference action data is used to represent the desired posture of the robot in the operating environment; Under the condition of a preset control strategy, predicting the predicted action data of the robot at the t-th moment based on the first observation data at the t-th moment and the reference action data at the t-th moment among the multiple moments, where t is a positive number; Predicting the state of the robot at the k-th moment based on the predicted action data at the t-th moment and the first state data at the t-th moment, to obtain second state data at the k-th moment. The k-th moment is a moment after the t-th moment among the multiple moments; Training the preset control strategy based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, to obtain a trained control strategy; wherein, the trained control strategy is used to control the actions of the robot.

2. The method according to claim 1, characterized in that, The predicting the state of the robot at the k-th moment based on the predicted action data at the t-th moment and the first state data at the t-th moment, to obtain second state data at the k-th moment, includes: Obtaining a world model, where the world model is used to predict the state of the robot, and the world model is a model trained based on the first state data and the first observation data; Performing state prediction on the predicted action data and the first state data at the t-th moment through the world model, to obtain the second state data at the k-th moment.

3. The method according to claim 2, characterized in that, The obtaining the world model includes: Obtaining an environment simulation model, where the environment simulation model is a model to be trained to obtain the world model, and the environment simulation model is used to predict predicted state data at the j-th moment according to the first state data at the i-th moment and the first observation data at the i-th moment. The j-th moment is a moment after the i-th moment among the multiple moments, and i and j are positive numbers; Obtaining a prediction loss value based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment; Training the environment simulation model through the prediction loss value to obtain the world model.

4. The method according to claim 3, characterized in that, The obtaining a prediction loss value based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment includes: Obtaining a state loss value corresponding to the j-th moment based on the difference between the predicted state data at the j-th moment and the first state data at the j-th moment; Summing the state loss values respectively corresponding to the multiple moments to obtain the prediction loss value.

5. The method according to any one of claims 1 to 4, characterized in that, Under the condition of the preset control strategy, predicting the predicted action data of the robot at the t-th moment based on the first observation data and the reference action data at the t-th moment among the multiple moments includes: Under the condition of the preset control strategy, encoding the first observation data and the reference action data at the t-th moment to obtain an encoded feature representation; Based on the decoding process of the encoded feature representation, predicting the predicted action data of the robot at the t-th moment.

6. The method according to claim 5, wherein Under the condition of the preset control strategy, encoding the first observation data and the reference action data at the t-th moment to obtain an encoded feature representation, including: Performing a first encoding process on the first observation data and the reference action data at the t-th moment through a first encoder to obtain a first feature representation, where the first encoder is used to perform imitation learning on the actions of the robot based on the reference action data; Performing a second encoding process on the first observation data at the t-th moment through a second encoder to obtain a second feature representation, where the second encoder is used to perform predictive analysis on the actions of the robot by means of prior knowledge, and the prior knowledge is the knowledge learned during the training of the second encoder; the preset control strategy includes the first encoder and the second encoder; Performing feature fusion on the first feature representation and the second feature representation to obtain the encoded feature representation.

7. The method according to claim 5, characterized in that Under the condition of the preset control strategy, encoding the first observation data and the reference action data at the t-th moment to obtain an encoded feature representation, including: Obtaining the motion instruction data corresponding to each of the multiple moments, where the motion instruction data is used to characterize the data for guiding the robot to execute the motion process; Under the condition of the preset control strategy, performing a third encoding process on the motion instruction data through a third encoder to obtain a third feature representation, where the third encoder is used to analyze the motion situation of the robot following the instruction based on the motion instruction data; Performing a second encoding process on the first observation data at the t-th moment through a second encoder to obtain a second feature representation, where the second encoder is used to perform predictive analysis on the actions of the robot by means of prior knowledge, and the prior knowledge is the knowledge learned during the training of the second encoder; Based on the second feature representation and the third feature representation, obtaining the encoded feature representation.

8. The method according to claim 5, wherein The obtaining the encoded feature representation based on the second feature representation and the third feature representation includes: Performing feature fusion on the second feature representation and the third feature representation to obtain the encoded feature representation, and the preset control strategy includes the second encoder and the third encoder; or, The first encoder performs first encoding processing on the first observation data at the t-th moment and the reference action data at the t-th moment to obtain a first feature representation. The first encoder is used to perform imitation learning on the actions of the robot based on the reference action data; the first feature representation, the second feature representation, and the third feature representation are subjected to feature fusion to obtain the encoded feature representation. The preset control strategy includes the first encoder, the second encoder, and the third encoder.

9. The method according to claim 5, wherein The predicting of the predicted action data of the robot at the t-th moment based on the decoding processing of the encoded feature representation includes: The decoder in the preset control strategy performs decoding processing on the encoded feature representation and outputs the predicted action data of the robot at the t-th moment.

10. The method according to any one of claims 1 to 4, characterized in that, The training of the preset control strategy based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment to obtain the trained control strategy includes: Based on the difference between the second state data at the k-th moment and the reference action data at the k-th moment, the loss value corresponding to the k-th moment is obtained; The policy parameters in the preset control strategy are adjusted with the loss value to obtain the trained control strategy.

11. The method according to claim 10, wherein The adjusting of the policy parameters in the preset control strategy with the loss value to obtain the trained control strategy includes: Obtaining the loss values respectively corresponding to the multiple moments; The policy parameters in the preset control strategy are iteratively adjusted with multiple loss values to obtain the trained control strategy.

12. The method according to any one of claims 1 to 4, characterized in that The obtaining of the first state data and the reference action data respectively corresponding to the robot at multiple moments includes: Obtaining the first observation data collected by the robot at the multiple moments respectively; Performing a preset state conversion on the first observation data respectively corresponding to the multiple moments to obtain the first state data respectively corresponding to the multiple moments; Obtaining a reference action sequence, where the reference action sequence is used to represent the expected posture sequence of the robot in the operating environment, and the reference action sequence includes the reference action data respectively corresponding to the multiple moments.

13. A robot control device, characterized in that, The device includes: A data acquisition module, configured to acquire the first state data and the reference action data respectively corresponding to the robot at multiple moments. The first state data is data obtained after performing a preset state process on the first observation data, and the first observation data is data collected by the robot in the operating environment through a sensor. The reference action data is used to represent the expected posture of the robot in the operating environment; An action prediction module, configured to predict the predicted action data of the robot at the t-th moment based on the first observation data at the t-th moment and the reference action data at the t-th moment among the multiple moments under the condition of a preset control strategy, where t is a positive number; A state prediction module, configured to predict the state of the robot at the k-th moment based on the predicted action data at the t-th moment and the first state data at the t-th moment, so as to obtain the second state data at the k-th moment, where the k-th moment is a moment after the t-th moment among the multiple moments; A policy training module, configured to train the preset control policy based on the loss value between the second state data at the k-th moment and the reference action data at the k-th moment, so as to obtain a trained control policy; wherein, the trained control policy is used to control the actions of the robot.

14. A computer device, characterized in that, The computer device includes a processor and a memory, and at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the robot control method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, At least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the robot control method according to any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes computer instructions, and when the computer instructions are executed by a processor, the robot control method according to any one of claims 1 to 12 is implemented.

Images