Method and device for determining robot control strategy model, readable medium and program product

Abstract

The invention provides a method and equipment for determining a robot control strategy model, a method and equipment for determining a robot control strategy, a computer readable storage medium and a program product. A specific embodiment of the application comprises the following steps: training in a simulation environment to obtain a reinforcement learning strategy model for determining a control strategy of a simulation robot; and for a plurality of links involved in the execution process of the control strategy of the reinforcement learning strategy model on the simulation robot, adopting migration algorithms corresponding to the plurality of links to adapt the reinforcement learning strategy model to a real environment to obtain a robot control strategy model. According to the method, a large number of training samples can be efficiently generated based on a simulation environment, and the training efficiency, accuracy and generalization of the reinforcement learning strategy model are improved; based on the strategy migration algorithms corresponding to the multiple links, the adaptation degree between the robot control strategy model and the real environment is improved, and the accuracy of the robot control strategy model in the real environment is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for determining a robot control strategy model, a method and device for determining a robot control strategy, a computer-readable medium, and a program product. Background Art

[0002] Embodied intelligence refers to the process in which a physical robot learns how to solve specific tasks in the process of continuous interaction with the environment. Data-driven robot control strategy models based on reinforcement learning generally require a large number of training samples to ensure the generalization and accuracy of their control strategies. However, real-world training samples are often scarce, and it is difficult to collect training samples, which limits the application scope of the trained robot control strategy model and poor intelligent performance in real scenarios.

[0003] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section. Summary of the invention

[0004] Multiple aspects of the present application provide a method and device for determining a robot control strategy model, a method and device for determining a robot control strategy, a computer-readable storage medium and a program product, so as to solve the problem of poor intelligent performance caused by insufficient adaptability of the trained robot control strategy model to real application scenarios due to a lack of training samples.

[0005] In the first aspect of the present application, a method for determining a robot control strategy model is provided, comprising: training in a simulation environment a reinforcement learning strategy model for determining the control strategy of a simulated robot; for multiple links involved in the execution of the control strategy of the simulated robot by the reinforcement learning strategy model, adapting the reinforcement learning strategy model to the real environment using migration algorithms corresponding to each of the multiple links to obtain a robot control strategy model.

[0006] The second aspect of the present application provides a method for determining a robot control strategy, comprising: determining the robot control strategy through a robot control strategy model based on the robot's perception data of the real environment and the robot's state data, wherein the robot control strategy model is obtained through the method described in the first aspect above; controlling the robot to operate according to the control strategy.

[0007] The third aspect of the present application provides a device for determining a robot control strategy model, comprising: a reinforcement learning module, configured to train in a simulation environment to obtain a reinforcement learning model for determining the control strategy of a simulated robot; a strategy migration module, configured to adapt the reinforcement learning strategy model to a real environment using migration algorithms corresponding to multiple links involved in the execution process of the control strategy of the simulated robot by the reinforcement learning model, so as to obtain a robot control strategy model.

[0008] The fourth aspect of the present application provides a device for determining a robot control strategy, comprising: a strategy determination module, configured to determine the robot control strategy through a robot control strategy model, based on the robot's real environment perception data of the real environment and the robot's state data, wherein the robot control strategy model is obtained through the device described in the third aspect above; a control strategy module, configured to control the robot to operate according to the control strategy.

[0009] In a fifth aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods described in the first and second aspects above.

[0010] In a sixth aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the methods described in the first and second aspects above.

[0011] In a seventh aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first and second aspects above is implemented.

[0012] In the solution provided in the embodiment of the present application, first, a reinforcement learning strategy model for controlling the execution of simulated robot tasks is trained in a simulation environment. A large number of training samples can be efficiently generated based on the simulation environment, thereby improving the training efficiency, accuracy and generalization of the reinforcement learning strategy model. Then, for multiple links involved in the control strategy execution process of the simulated robot by the reinforcement learning strategy model, the reinforcement learning strategy model is adapted to the real environment using migration algorithms corresponding to each of the multiple links to obtain a robot control strategy model. Based on the migration algorithms corresponding to each of the multiple links, the adaptability between the robot control strategy model and the real environment is improved, thereby ensuring the accuracy of the robot control strategy model in the real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0014] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0015] Figure 1 A schematic diagram of a flow chart of a method for determining a robot control strategy model provided in one embodiment of the present application;

[0016] Figure 2 Schematic diagram of the knowledge distillation process from the teacher strategy model to the student strategy model for this application;

[0017] Figure 3 A schematic diagram of the migration algorithm corresponding to the motion control link of this application;

[0018] Figure 4 A schematic diagram of a flow chart of a robot control strategy provided in one embodiment of the present application;

[0019] Figure 5 A schematic diagram of the structure of a device for determining a robot control strategy model provided in one embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of a device for determining a robot control strategy provided in an embodiment of the present application;

[0021] Figure 7 This is a structural schematic diagram of a device suitable for implementing the solution in the embodiment of the present application. The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0024] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0025] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0026] An embodiment of the present application provides a method for determining a robot control strategy model. The method first trains in a simulation environment to obtain a reinforcement learning strategy model for controlling the execution of simulated robot tasks. Based on the simulation environment, a large number of training samples can be efficiently generated, thereby improving the training efficiency, accuracy, and generalization of the reinforcement learning strategy model. Then, for multiple links involved in the control strategy execution process of the simulated robot by the reinforcement learning strategy model, the reinforcement learning strategy model is adapted to the real environment using migration algorithms corresponding to each of the multiple links to obtain a robot control strategy model. Based on the migration algorithms corresponding to each of the multiple links, the adaptability between the robot control strategy model and the real environment is improved, thereby ensuring the accuracy of the robot control strategy model in the real environment.

[0027] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0028] Figure 1The processing flow of a method provided by an embodiment of the present application is shown, and the method at least includes the following processing steps:

[0029] Step S101, training in a simulation environment to obtain a reinforcement learning strategy model for determining a control strategy for a simulation robot.

[0030] A simulation environment is a virtual environment created by computer technology and software tools to simulate physical processes, system behaviors, or environmental conditions in the real world so that models can be trained and tested. The simulation environment allows relevant personnel to iterate and optimize the model without actually building or operating the real system.

[0031] Compared with the model training process in the real environment, the training process in the simulation environment has the following advantages:

[0032] Data generation: The simulation environment can efficiently generate a large amount of training data that can be used for model training. The training data covers various situations and scenarios that the model may encounter, which can not only enrich the model's training set, but also improve the model's generalization ability.

[0033] Condition Control: In a simulation environment, personnel can precisely control various parameters and conditions, such as light, temperature, pressure, etc., to simulate different working environments and test scenarios. This control capability enables personnel to systematically evaluate the performance of the model under different conditions.

[0034] Safety: Compared with model training in a real environment, the simulation environment is safer. In a simulation environment, even if the model has errors or abnormal behavior, it will not cause harm to the actual system or personnel.

[0035] Reproducibility: The simulation environment provides reproducible experimental conditions, allowing researchers to run the same experiment multiple times and compare the results. This reproducibility helps researchers more accurately evaluate the performance and stability of the model.

[0036] Cost-effectiveness: Compared with model training in a real environment, the training process in a simulation environment does not require the purchase and maintenance of expensive hardware equipment, nor does it need to bear the losses caused by experimental failures, allowing relevant personnel to conduct more experiments and iterations at a lower cost.

[0037] In the present embodiment, according to the task scenario of the robot required to perform tasks in the real environment, a simulation environment corresponding to the task scenario can be created in a targeted manner. By modeling the real robot, a simulation robot can be obtained. The modeling process mainly imports the STL (Stereolithography, standard triangle language file) file containing the three-dimensional model of the robot, the URDF (Unified Robot Description Format, unified robot description format) or XML (xtensible Markup Language, extensible markup language) file used to define the robot dynamics and kinematic relationship into the engine, and according to the simulation engine rules, the XML model is modified into an interactive simulation environment with interactive attributes. The URDF file usually contains the link (link), joint (joint), sensor (sensor), kinematic chain (chain) and collision (collision), visual (visual) attributes of the embodied intelligent body.

[0038] Reinforcement Learning (RL) is a machine learning method, the purpose of which is to allow a simulated intelligent agent to learn how to obtain the maximum cumulative reward in a specific task through its interaction with a simulated environment. The simulated intelligent agent in this embodiment is specifically manifested as a simulated robot and a reinforcement learning strategy model that controls the simulated robot. The simulated robot interacts directly with the simulated environment and collects a sequence of simulated environment perception data and robot state data. The reinforcement learning strategy model determines the robot's control strategy based on the sequence of simulated environment perception data and the robot's current state data to control the robot to interact with the simulated environment, and so on. The simulated intelligent agent gradually learns how to achieve a specific goal by selecting the best sequence of actions and obtaining a trained reinforcement learning strategy model by trying different actions and observing the feedback (i.e., reward or punishment) of the simulated environment on its actions.

[0039] In order to further improve the accuracy of the trained reinforcement learning strategy model, a deep reinforcement learning method can be used in this embodiment. Deep reinforcement learning (DRL) combines the techniques of deep learning and reinforcement learning, introduces high-dimensional perception data, and uses deep neural networks as the initial reinforcement learning strategy model to learn the mapping relationship between the simulation environment perception data sequence and the state data of the simulation robot to the control strategy, which is used to solve decision-making problems in complex environments.

[0040] Rewards in the reinforcement learning process are feedback signals used in deep reinforcement learning to guide the agent to learn and optimize behavior, indicating the quality of a certain action or strategy. The reward function defines the function of each state-action pair. It reflects the goal of performing the task. For example, in this application, the correct direction of moving over obstacles and safe movement will give positive rewards, while collisions with obstacles will give negative rewards.

[0041] In this embodiment, the rewards in the reinforcement learning process mainly include but are not limited to the following categories:

[0042] Instruction following reward R inst_follow : Reward the robot for following the predetermined speed and direction instructions to ensure that its behavior meets the control goals;

[0043] Movement Efficiency Bonus R motion_effciency : Rewards robots for energy efficiency and motion smoothness when performing tasks, including reducing unnecessary joint motion and torque consumption;

[0044] Stability Reward R stability : Reward the robot for maintaining its balance and gait stability, avoiding falls or unstable movements;

[0045] Collision Avoidance Reward - R collision_avoidance : Penalize unnecessary contact between the robot and the environment, such as collisions between the robot’s arms, feet, legs, or knees and obstacles, to protect the robot from damage;

[0046] Gait smoothness reward R gait_smoothnes : Reward the robot to generate smooth and coherent gaits, which improves the naturalness and comfort of movement by reducing sudden changes in gait;

[0047] Environmental adaptability reward R environment_suitablity : Reward the robot to adjust its behavior according to environmental changes, such as adjusting gait and speed on different terrains;

[0048] Behavior consistency reward R consistency : Reward the robot for maintaining consistent behavior in different environments and conditions to ensure the generalization ability of the strategy.

[0049] Combining all of the above factors can construct a comprehensive reward function, such as:

[0050] R=R inst_follow ×W inst_follow +R motion_effciency ×W motion_effciency

[0051] +R stability ×W stability -R collision_avoidance ×W collision_avoidance

[0052] +R gait_smoothness ×W gait_smppthness +R environment_suitablity

[0053] ×W environment_suitablity +R consistency ×W consistency

[0054] Where W inst_follow , W motion_effciency , W stability , W collision_avoidance , W gait_smoothness , W environment_suitablity , W consistency It is the weight coefficient corresponding to each category, which is used to adjust the relative importance of each part.

[0055] Through these rewards, the robot is guided to achieve efficient, stable and adaptable motion control in complex environments while reducing energy consumption and avoiding unnecessary damage. This design enables the robot to show better performance and robustness in a variety of tasks and environments.

[0056] In some optional implementations of this embodiment, the execution subject may perform step 101 in the following manner:

[0057] The first step is to disassemble the simulation task corresponding to the simulation robot to obtain multiple subtasks.

[0058] According to the set simulation task goals, specific actions and constraints, the high-complexity task is decomposed into multiple basic subtasks with low complexity. For example, the simulation task is decomposed into basic subtasks such as the movement of the simulation robot body, object manipulation, and object grasping.

[0059] In the second step, based on multiple subtasks, the initial reinforcement learning strategy model for controlling the operation of the simulation robot in the simulation environment is trained in parallel to obtain a reinforcement learning strategy model.

[0060] GPU (graphics processing unit) has the ability to process tens of thousands of parallel instructions. In this implementation, the simulation engine based on parallel GPU computing can achieve efficient parallel collection of policy learning data by simultaneously generating and managing multiple (e.g., thousands) robots in the same physically accurate simulation environment. According to different basic subtasks, corresponding parallel training environments will be generated.

[0061] For the parallel training environment of the simulated robot mobile obstacle crossing task, multiple robots can be placed together in the same terrain grid environment. The grid does not change as a whole at each training reset, but is achieved by arranging different terrain types and difficulty levels in parallel to form a comprehensive terrain grid, assigning specific terrain types and corresponding difficulty levels to each simulated robot. The simulated robot is placed in the center of a simple grid and moves along a trajectory of gradually increasing challenging terrain. When the simulated robot successfully crosses the current terrain, its difficulty level will automatically increase and start from a more difficult terrain at the next training reset. Conversely, if the robot does not move half of its target distance in a round of training, its difficulty level will be reduced accordingly. In order to increase training diversity and prevent forgetting obstacle crossing skills for low-level terrain, the simulated robot that reaches the highest difficulty level will randomly select the training difficulty at reset. The advantage of this adaptive training method is that it can dynamically adjust the training difficulty based on the actual performance of the robot without external intervention. In addition, it can independently adapt to the difficulty of each terrain type and provide intuitive and quantitative feedback on the training progress.

[0062] Ultimately, when the simulated robots are able to evenly distribute across all terrains and successfully complete the obstacle-crossing task, they have mastered the task.

[0063] For tasks such as object manipulation and object grasping, multiple sub-environments are generated simultaneously. In each sub-environment, the robot interacts with the precisely modeled target object and trains the reinforcement learning policy model that controls the robot. The multiple interacting objects can be interactive objects with the same parameters or interactive objects with the same parameter type but different parameter distributions.

[0064] In the training process of the reinforcement learning policy model, the data collected by each simulated robot is crucial for updating the policy gradient. Assume that the amount of data collected before each batch iteration is given by the formula N episode =N robot ×N steps Determine, where N robot Represents the number of simulated robots trained in parallel, N steps Represents the maximum number of iterations that each robot can perform in one round of training.

[0065] Therefore, finding the optimal balance point of the number of simulation robots is the key to achieving efficient training. When the number of robots is too large, the amount of data contributed by each robot to the policy update is reduced, which may cause the gradient estimation to become unstable and increase noise, thereby affecting the stability and efficiency of the learning process. On the contrary, if the number of simulation robots is small, each simulation robot needs to perform more iteration steps in each policy update. Although this maintains the consistency of samples in quantity, due to the proximity of samples in time, the diversity of sample data will be reduced, violating the assumption of independent and identical distribution, and having a negative impact on the training effect. In this embodiment, the optimal number of robots is found through experiments and adjustments, and the training process is parallelized according to the optimal data to ensure that while maintaining data diversity, accurate gradient estimation can also be obtained, thereby improving learning efficiency and model performance.

[0066] In this implementation, combining task decomposition and parallel training helps to improve the training efficiency and accuracy of the reinforcement learning model.

[0067] Step S102, for multiple links involved in the policy control execution process of the simulation robot by the reinforcement learning policy model, the reinforcement learning policy model is adapted to the real environment by using the migration algorithms corresponding to the multiple links to obtain the robot control policy model.

[0068] In this embodiment, the correspondence between the links and the migration algorithm is determined in advance. Then, for each link involved in the execution process of the control strategy of the simulation robot by the reinforcement learning strategy model, the migration algorithm corresponding to the link is used to adapt the link corresponding to the reinforcement learning strategy model to the real environment to obtain the robot control strategy model.

[0069] Among them, the Sim2Real (Simulation to Reality) migration algorithm aims to solve the problem of migrating a model trained in a simulation environment to the real world. This type of algorithm achieves cross-domain migration of models by reducing the difference between the simulation environment and the real environment, including but not limited to domain adaptation algorithms, domain randomization algorithms, inverse dynamics model algorithms, and cross-model migration algorithms. Based on the rich experience of technicians, the correspondence between the links and the migration algorithm can be determined.

[0070] In this embodiment, the timing relationship between different nodes in the entire control strategy execution process is divided into multiple links with a chronological relationship, such as an environment perception link, a strategy control link, and a task execution link.

[0071] In some optional implementations of the present embodiment, the execution entity may perform step 102 in the following manner: in the process of iteratively training the student strategy model using the reinforcement learning strategy model as the teacher strategy model, a migration algorithm corresponding to each of the multiple links is used to gradually adapt the student strategy model in the training process to the real environment to obtain a robot control strategy model.

[0072] In this implementation, the knowledge distillation process from the teacher strategy model to the student strategy model and the migration process of the student strategy model to the real environment are carried out simultaneously.

[0073] Continue to refer Figure 2 , which shows a schematic diagram of the knowledge distillation process from the teacher strategy model to the student strategy model.

[0074] Firstly, the current state data of the simulation robot and the simulation environment perception data sequence collected by the simulation robot are input into the trained reinforcement learning strategy model to obtain the control strategy; then, the control strategy is executed by the simulation robot to interact with the simulation environment; finally, the reward data is determined based on the changed simulation environment and reward function.

[0075] The student strategy model also takes the current state data of the simulation robot and the simulation environment perception data sequence collected by the simulation robot as input, outputs prediction instructions, and calculates the loss between the prediction instructions and the control strategy output by the reinforcement learning strategy model to update the parameters of the student strategy model through the loss.

[0076] By iteratively executing the above process, in response to reaching a preset end condition, a trained student strategy model is obtained, wherein the preset end condition is, for example, that the training time exceeds a preset time threshold, the number of training times exceeds a preset number threshold, and the training loss region converges.

[0077] In each training process of the student strategy model, for each link in the training process, the corresponding migration algorithm is used for data processing, so that the student strategy model gradually adapts to the real environment during the training process. In this way, as the teacher strategy model knowledge is distilled to obtain the learning strategy model, the obtained student strategy model realizes the migration from the simulation environment to the real environment, and the final student strategy model is used as the robot control strategy model.

[0078] In this implementation, through the knowledge distillation process, a student strategy model that is easy to deploy, has high reasoning speed and generalization ability can be obtained, and the model migration process for the real environment is carried out simultaneously during the knowledge distillation process, which helps to improve the adaptability between the robot control strategy model and the real environment while improving the overall processing speed.

[0079] In some optional implementations of this embodiment, the execution subject may perform the migration operation in the following manner:

[0080] First, for the motion control link of the simulated robot by the student strategy model, the credibility of the state data of the simulated robot and the simulated environment perception data sequence is determined through the state credibility recursive encoder to obtain credibility data. Then, the control strategy of the simulated robot is determined according to the credibility data through the student strategy model to gradually adapt the motion control algorithm of the student strategy model to the real environment.

[0081] Continue to refer Figure 3 , shows a schematic diagram of the migration algorithm corresponding to the strategy control link. Among them, the hidden state represents the hidden state of the student strategy model.

[0082] The motion control link refers to the link in which the student strategy model controls the simulation robot to perform tasks based on the control strategy. The key component of the migration method corresponding to the motion control link is a state credibility recursive encoder that integrates the state data of the simulation robot and the simulation environment perception data sequence. The state credibility recursive encoder needs to be trained in the simulation environment to determine the feasibility of the state data and the simulation environment perception data sequence. After end-to-end training, the state credibility encoder can integrate the state data of the simulation robot's own body and the simulation environment perception data sequence. Without relying on heuristic methods, it can learn to use the model prediction value to plan the support point and acceleration movement of the simulation robot in a forward-looking manner when the simulation environment perception data sequence is reliable, and seamlessly fall back to the motion planning that relies on the body's own state data when necessary. Among them, the simulation environment perception data sequence of the robot to the simulation environment includes but is not limited to the images, point clouds, sounds and other types of data collected by the camera, radar, and sound collector on the robot; the robot's state data is the robot's perception data of its own state, such as control instructions acting on the body; body speed and direction (linear speed and angular velocity); joint position, velocity and acceleration; phase information for gait generation.

[0083] Therefore, combined with the learning model of the state credibility recursive encoder, the robot can achieve higher speed and efficiency based on environmental perception data, and can also operate robustly based on its own state data.

[0084] In this implementation, for the motion control link, based on the state credibility recursive encoder, the credibility of the state data of the simulated robot and the simulated environment perception data sequence is judged, so that the student strategy model can judge the credibility of the input data in the real environment and achieve adaptation to the real environment.

[0085] In some optional implementations of the present embodiment, the above-mentioned execution subject may perform the above-mentioned migration operation in the following manner: for the multiple simulated environment perception data sequences obtained by the simulated robot in the environment perception link, the multiple simulated environment perception data sequences are processed using the preprocessing methods corresponding to the multiple environment perception data, so as to reduce the difference between the corresponding simulated environment perception data sequences and the real environment perception data, and gradually adapt the simulated environment perception data sequences required by the student strategy model to the real environment.

[0086] In the environmental perception link, the aim is to reduce the appearance difference between the simulated environmental perception data sequence of the student strategy model in the simulated environment and the real environmental perception data in the real environment.

[0087] The various simulation environment perception data sequences include but are not limited to perception data of the image, point cloud, sound, etc. Taking the simulation environment perception data sequence in the form of an image as an example, for RGB (Red, Green, Blue) images, preprocessing such as motion blur and Gaussian noise is performed; for depth images, preprocessing such as depth cropping, pixel-level Gaussian noise and random artifacts is performed.

[0088] Taking the simulated environment perception data sequence obtained by lidar as an example, preprocessing such as simulating real sensor characteristics and data enhancement is performed.

[0089] In this embodiment, in order to further reduce the appearance difference between the simulated environment perception data sequence of the student strategy model in the simulated environment and the real environment perception data in the real environment, the real environment perception data corresponding to the real environment can also be preprocessed. Taking the depth image in the real environment as an example, it can be preprocessed by depth cropping, hole filling, spatial smoothing and temporal smoothing.

[0090] In this implementation, the simulated environment perception data sequence obtained in the environment perception link is processed by a corresponding preprocessing method to reduce the appearance difference between the simulated environment perception data sequence of the student strategy model in the simulated environment and the real environment perception data in the real environment, so that the final robot control strategy model is adapted to the real environment in the environment perception dimension.

[0091] In some optional implementations of this embodiment, the execution subject may perform the migration operation in the following manner: for the operation link of the control instructions of the simulation robot based on the student strategy model, at least one of the following migration operations is performed to gradually adapt the control instructions of the student strategy model in the training process to the real environment:

[0092] 1. Randomly set the initial physical state of the simulated robot in the simulation environment.

[0093] In each training session, the masses, initial joint positions, and speeds of the simulated robot's body, arms, legs, and other parts, as well as the simulated robot's initial posture and speed, are randomized.

[0094] 2. Randomly set the interaction parameters between the simulation robot and the simulation object that interacts with the simulation robot.

[0095] The simulation object that interacts with the simulation robot is the simulation object that the simulation robot has contacted during the operation according to the control instruction, such as the ground, stairs, and objects that the robot arm needs to grasp. The interaction parameters are parameters such as the friction coefficient between the simulation robot and the interactive simulation object.

[0096] 3. Apply external force parameters to the simulation robot that affect the operation state of the simulation robot.

[0097] As an example, one can apply external forces and torques to the robot body, or occasionally set the coefficient of friction of the feet to a low value to introduce slippage.

[0098] 4. Alternately train the student strategy model in a simulation environment and a real environment.

[0099] For tasks such as manipulating objects and grasping objects, in order to reduce the problem of inconsistent parameter distribution between the real environment and the simulation environment, the student strategy model is trained alternately in the simulation environment and the real environment to adjust the distribution of simulation parameters. In this way, the parameter distribution of the simulation environment can be changed to achieve the migration of the student strategy model to the real environment by matching the simulation environment and the real environment.

[0100] In this implementation, the simulation robot adapts the student strategy model to the real environment by performing corresponding migration operations based on the task execution link of the control instructions of the student strategy model, thereby improving the adaptability of the student strategy model to the real environment during the operation link.

[0101] In some optional implementations of this embodiment, the above-mentioned execution subject may perform the process of iteratively training the student strategy model using the reinforcement learning strategy model as the teacher strategy model in the following manner:

[0102] The first step is to establish a simulation model of the real robot based on the kinematic characteristics of the real robot to be controlled by the robot control strategy model in the real environment.

[0103] There may be differences between the simulated robot and the real robot used in the training process of the reinforcement learning strategy model, resulting in the reinforcement learning strategy model not being fully adapted to the real robot. In this implementation, a simulation model of the real robot is established based on the kinematic characteristics of the real robot, aiming to conduct the training process of the student strategy model completely based on the kinematic characteristics of the real robot, so that the student strategy model can be adapted to the real robot.

[0104] For example, there are subtle differences in the dynamic characteristics (such as mass distribution, friction, etc.) of robot forms such as robotic arms and quadruped robots of the same type but different models, and physical-level modeling is required for real robots of the target model, where the target model is, for example, the model corresponding to the robot to be controlled in the real environment.

[0105] In the second step, the reinforcement learning strategy model is used as the teacher strategy model, and the simulation model is used as the simulation robot controlled by the student strategy model to iteratively train the student strategy model.

[0106] In this implementation, the simulation model executes the instructions output by the student strategy model to interact with the simulation environment. By modeling a real robot in a real environment, the student strategy model is gradually adapted to the real robot during the training process.

[0107] In some optional implementations of the present embodiment, the above-mentioned execution entity may perform the above-mentioned second step in the following manner: first, through the reinforcement learning strategy model, determine the control strategy of the simulated robot according to the state data of the simulated robot, the simulated environment perception data sequence of the simulated robot to the simulated environment, and the environmental interaction data between the simulated robot and the simulated environment; then, train the student strategy model with the state data and the simulated environment perception data sequence as input and the control strategy as the expected output.

[0108] The environmental interaction data between the simulated robot and the simulated environment, as an auxiliary training data, is used to train the teacher strategy. It is data that is easy to obtain in the simulation environment but may be unavailable or difficult to obtain in the real environment, such as accurate terrain height information and friction coefficient; detailed information on the contact state between the robot's feet and the ground, such as the contact force and the normal of the contact point; external forces and torques acting on the robot body; the contact state of the thigh and calf of the legged robot, etc.

[0109] The teacher strategy model can use three types of information: the simulated robot's body state data, simulated environment perception data sequence, and auxiliary training data. The purpose is to learn how to perform tasks under complex and changing conditions in the simulated environment. The student strategy model can only use the body state data and simulated environment perception data sequence. The purpose is to imitate the behavior of the teacher strategy model when there are only body state data and simulated environment perception data sequence, so as to achieve effective deployment of the student strategy model in the real environment.

[0110] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operation: adjusting the parameters of the robot control strategy model according to the difference between the real task in the real environment and the simulated task in the simulated environment.

[0111] As an example, the real task in the real environment is to carry an object of 10 kg, while the simulated task in the simulation environment is to carry an object of 5 kg. In this case, the parameters of the robot control strategy model for the carrying force need to be adjusted.

[0112] In this implementation, based on the difference between the real task and the simulated task, the parameters of the robot control strategy model are adjusted, so that the robot control strategy model can be further adapted to the real task in the real environment.

[0113] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operation: tailor the robot control strategy model according to the computing capability of the deployment device corresponding to the robot control strategy model in the real environment.

[0114] For example, when the computing power of the deployment device used to deploy the robot control strategy model can meet the computing requirements of the robot control strategy model, the complete robot policy control model is directly deployed on the deployment device; for another example, when the computing power of the deployment device used to deploy the robot control strategy model cannot meet the computing requirements of the robot control strategy model, the robot control strategy model needs to be trimmed to reduce the parameter scale of the robot control strategy model.

[0115] In this implementation, the robot control strategy model is flexibly tailored based on the computing power of the deployed device, so that the robot control strategy model is further adapted to the real environment.

[0116] Continue to refer Figure 4 , shows a processing flow of a method provided by an embodiment of the present application, the method at least comprising the following processing steps:

[0117] Step 401, determining the control strategy of the robot through the robot control strategy model according to the real environment perception data of the robot to the real environment and the state data of the robot.

[0118] The robot control strategy model is obtained by any implementation method of the robot control strategy model determination method.

[0119] As an example, after receiving the target task, the robot control strategy model is used to determine the control instructions required for the robot to complete the target task based on the robot's real environment perception data of the real environment and the robot's state data.

[0120] The robot's real environment perception data of the real environment includes but is not limited to images, point clouds, sounds and other types of data collected by the robot's cameras, radars, and sound collectors about the surrounding environment; the robot's state data is the robot's perception data of its own state, such as control instructions acting on the body; body speed and direction (linear speed and angular velocity); joint position, speed and acceleration; and phase information used for gait generation.

[0121] Step 402: Control the robot to operate according to the control strategy.

[0122] After determining the control instruction, the robot operates according to the control instruction to interact with the real environment. The above steps 401-402 are executed cyclically until the target task to be performed by the robot is completed.

[0123] In addition, the present application also provides a device for determining a robot control strategy model, the structure of which is as follows: Figure 5 shown.

[0124] A device for determining a robot control strategy model, comprising: a reinforcement learning module 501 configured to train in a simulation environment to obtain a reinforcement learning strategy model for determining the control strategy of a simulated robot; a strategy migration module 502 configured to adapt the reinforcement learning strategy model to a real environment using migration algorithms corresponding to multiple links involved in the execution process of the control strategy of the simulated robot by the reinforcement learning strategy model, so as to obtain a robot control strategy model.

[0125] In some optional implementations of this embodiment, the strategy transfer module 502 is further configured to: in the process of iteratively training the student strategy model using the reinforcement learning strategy model as the teacher strategy model, use migration algorithms corresponding to multiple links to gradually adapt the student strategy model in the training process to the real environment to obtain a robot control strategy model.

[0126] In some optional implementations of this embodiment, the strategy migration module 502 is further configured as follows: for the motion control link of the simulated robot by the student strategy model, the credibility of the state data of the simulated robot and the simulation environment perception data sequence is determined by a state credibility recursive encoder to obtain credibility data; through the student strategy model, the control strategy of the simulated robot is determined according to the credibility data to gradually adapt the motion control algorithm of the student strategy model to the real environment.

[0127] In some optional implementations of the present embodiment, the strategy migration module 502 is further configured to: for the multiple simulated environment perception data sequences obtained by the simulated robot in the environment perception link, the multiple simulated environment perception data sequences are processed using the preprocessing methods corresponding to the multiple environment perception data, so as to reduce the difference between the corresponding simulated environment perception data sequences and the real environment perception data, and gradually adapt the simulated environment perception data sequences required by the student strategy model to the real environment.

[0128] In some optional implementations of the present embodiment, the strategy migration module 502 is further configured to: for the operation link of the control instructions of the simulated robot based on the student strategy model, perform at least one of the following migration operations to gradually adapt the control instructions of the student strategy model in the training process to the real environment: randomly set the initial physical state of the simulated robot in the simulation environment; randomly set the interaction parameters between the simulated robot and the simulated object that interacts with the simulated robot; apply external force parameters that affect the operating state of the simulated robot to the simulated robot; alternately train the student strategy model in the simulation environment and the real environment.

[0129] In some optional implementations of this embodiment, the strategy migration module 502 is further configured to: establish a simulation model of the real robot based on the kinematic characteristics of the real robot to be controlled by the robot control strategy model in the real environment; use the reinforcement learning strategy model as the teacher strategy model and the simulation model as the simulated robot controlled by the student strategy model, and iteratively train the student strategy model.

[0130] In some optional implementations of this embodiment, the strategy migration module 502 is further configured to: determine the control strategy of the simulated robot according to the state data of the simulated robot, the simulated environment perception data sequence of the simulated robot to the simulated environment, and the environmental interaction data between the simulated robot and the simulated environment through the reinforcement learning strategy model; train the student strategy model with the state data and the simulated environment perception data sequence as input and the control strategy as the expected output.

[0131] In some optional implementations of this embodiment, the above-mentioned device also includes: an adjustment module (not shown in the figure), which is configured to adjust the parameters of the robot control strategy model according to the difference between the real task in the real environment and the simulated task in the simulation environment.

[0132] In some optional implementations of this embodiment, the above-mentioned device also includes: a tailoring module (not shown in the figure), which is configured to tailor the robot control strategy model according to the computing power of the deployment device corresponding to the robot control strategy model in the real environment.

[0133] In some optional implementations of this embodiment, the reinforcement learning module 501 is further configured to: disassemble the simulation task corresponding to the simulation robot to obtain multiple sub-tasks; and according to the multiple sub-tasks, parallelly train the initial reinforcement learning strategy model for controlling the operation of the simulation robot in the simulation environment to obtain a reinforcement learning strategy model.

[0134] In this embodiment, first, a reinforcement learning strategy model for determining the control strategy of a simulated robot is trained in a simulation environment. A large number of training samples can be efficiently generated based on the simulation environment, thereby improving the training efficiency, accuracy, and generalization of the reinforcement learning strategy model. Then, for multiple links involved in the execution of the control strategy of the simulated robot by the reinforcement learning strategy model, the reinforcement learning strategy model is adapted to the real environment using migration algorithms corresponding to each of the multiple links to obtain a robot control strategy model. Based on the migration algorithms corresponding to each of the multiple links, the adaptability between the robot control strategy model and the real environment is improved, thereby ensuring the accuracy of the robot control strategy model in the real environment.

[0135] In addition, the present application also provides a robot control device, the structure of which is as follows: Figure 6 shown.

[0136] A device for determining a robot control strategy comprises: a strategy determination module 601, configured to determine the robot control strategy through a robot control strategy model according to the robot's real environment perception data of the real environment and the robot's state data, wherein the robot control strategy model is obtained by a person in an embodiment of the robot control strategy model determination device in an implementation manner; a control strategy module 602, configured to control the robot to operate according to the control strategy.

[0137] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for determining the robot control strategy model and the robot control method in the aforementioned embodiment, and the principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0138] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0139] Figure 7 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 700 includes a central processing unit (CPU, Central Processing Unit) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 702 or the program loaded from the storage part 708 to the random access memory (RAM, Random Access Memory) 703. In RAM 703, various programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O, Input / Output) interface 705 is also connected to the bus 704.

[0140] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 708 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet.

[0141] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 701, the above functions defined in the method of the present application are executed.

[0142] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0143] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0144] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0146] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0152] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0154] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for determining a robot control model, wherein: The method comprises: A reinforcement learning strategy model is obtained by training in a simulation environment to determine a control strategy of a simulation robot; For the multiple links involved in the execution process of the control strategy of the simulation robot by the reinforcement learning strategy model, the reinforcement learning strategy model is adapted to the real environment using the migration algorithms corresponding to the multiple links to obtain the robot control strategy model.

2. The method according to claim 1, wherein: The method of adapting the reinforcement learning strategy model to the real environment by using the migration algorithms corresponding to the multiple links to obtain the robot control strategy model includes: In the process of iteratively training the student strategy model using the reinforcement learning strategy model as the teacher strategy model, the migration algorithms corresponding to the multiple links are used to gradually adapt the student strategy model in the training process to the real environment to obtain the robot control strategy model.

3. The method according to claim 2, wherein: The step of gradually adapting the student strategy model in the training process to the real environment by using the migration algorithms corresponding to the multiple links respectively includes: For the motion control link of the simulated robot by the student strategy model, the credibility of the state data of the simulated robot and the simulation environment perception data sequence is determined by a state credibility recursive encoder to obtain credibility data; The control strategy of the simulation robot is determined according to the credibility data through the student strategy model, so as to gradually adapt the motion control algorithm of the student strategy model to the real environment.

4. The method according to claim 2, wherein: The step of gradually adapting the student strategy model in the training process to the real environment by using the migration algorithms corresponding to the multiple links respectively includes: For the multiple simulated environment perception data sequences obtained by the simulated robot in the environment perception link, the multiple simulated environment perception data sequences are processed using the preprocessing methods corresponding to the multiple environment perception data to reduce the differences between the corresponding simulated environment perception data sequences and the real environment perception data, and gradually adapt the simulated environment perception data sequences required by the student strategy model to the real environment.

5. The method according to claim 2, wherein: The step of gradually adapting the student strategy model in the training process to the real environment by using the migration algorithms corresponding to the multiple links respectively includes: For the operation link of the simulation robot based on the control strategy of the student strategy model, at least one of the following migration operations is performed to gradually adapt the control strategy of the student strategy model in the training process to the real environment: Randomly setting an initial physical state of the simulation robot in the simulation environment; Randomly setting interaction parameters between the simulation robot and a simulation object that interacts with the simulation robot; Applying external force parameters affecting the running state of the simulation robot to the simulation robot; The student strategy model is trained alternately in the simulation environment and the real environment.

6. The method according to claim 2, wherein: The method of iteratively training the student strategy model using the reinforcement learning strategy model as the teacher strategy model includes: Based on the kinematic characteristics of the real robot to be controlled by the robot control strategy model in the real environment, establishing a simulation model of the real robot; The reinforcement learning strategy model is used as the teacher strategy model, and the simulation model is used as the simulation robot controlled by the student strategy model, and the student strategy model is iteratively trained.

7. The method according to claim 6, wherein: The method of taking the reinforcement learning strategy model as the teacher strategy model and the simulation model as the simulation robot controlled by the student strategy model, and iteratively training the student strategy model comprises: Determining the control strategy of the simulation robot according to the state data of the simulation robot, the simulation environment perception data sequence of the simulation robot to the simulation environment, and the environment interaction data between the simulation robot and the simulation environment through the reinforcement learning strategy model; The student strategy model is trained by taking the state data and the simulation environment perception data sequence as input and the control strategy as expected output.

8. The method according to any one of claims 1 to 7, wherein: Also includes: According to the difference between the real task in the real environment and the simulated task in the simulated environment, the parameters of the robot control strategy model are adjusted.

9. The method according to any one of claims 1 to 7, wherein: Also includes: The robot control strategy model is tailored according to the computing capability of the deployment device corresponding to the robot control strategy model in the real environment.

10. The method according to any one of claims 1 to 7, wherein: The reinforcement learning strategy model trained in the simulation environment to determine the control strategy of the simulation robot includes: Disassembling the simulation task corresponding to the simulation robot to obtain multiple subtasks; According to the multiple subtasks, an initial reinforcement learning strategy model for controlling the operation of the simulation robot in the simulation environment is trained in parallel to obtain the reinforcement learning strategy model.

11. A method for determining a robot control strategy, wherein: The method comprises: Determining the control strategy of the robot according to the real environment perception data of the robot to the real environment and the state data of the robot through a robot control strategy model, wherein the robot control strategy model is obtained according to any one of claims 1 to 10; The robot is controlled to operate according to the control strategy.

12. A device for determining a robot control strategy model, wherein: The device comprises: A reinforcement learning strategy module is configured to train in a simulation environment to obtain a reinforcement learning strategy model for determining a control strategy of a simulation robot; The strategy migration module is configured to adapt the reinforcement learning strategy model to the real environment by using the migration algorithms corresponding to the multiple links involved in the execution process of the control strategy of the simulation robot by the reinforcement learning strategy model, so as to obtain the robot control strategy model.

13. A device for determining a robot control strategy, wherein: The device comprises: a strategy determination module configured to determine the control strategy of the robot according to the real environment perception data of the robot to the real environment and the state data of the robot through a robot control strategy model, wherein the robot control strategy model is obtained according to claim 12; The control strategy module is configured to control the robot to operate according to the control strategy.

14. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

15. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 11.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images