A robot autonomous learning method, device, equipment and storage medium

Abstract

The present invention discloses a robot autonomous learning method, device, equipment and storage medium, the method comprising: based on a pre-built virtual environment, obtaining environmental parameters automatically generated by the virtual environment; generating training data through a deep learning model according to the environmental parameters; generating at least one robot skill according to the training data by using a hierarchical reinforcement learning framework; determining the state space of the robot according to the task to be performed, and based on the state space, using a variational reasoning method to predict the potential skill vector of the task to be performed, the potential skill vector has a one-to-one correspondence with one of the robot skills in at least one robot skill; according to the robot skill and the potential skill vector, using a reinforcement learning algorithm to obtain a robot control strategy for completing the task to be performed. When facing different types of tasks, the present invention does not need to repeatedly train the robot for different types of tasks, thereby improving the generalization of the robot's autonomous learning.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a robot autonomous learning method, device, equipment and storage medium. Background Art

[0002] Traditional industrial robots are highly dependent on humans, require too much human intervention in the learning process, lack the ability to learn autonomously, and are difficult to perform complex tasks. Intelligent robots that can learn autonomously have similar perception, movement, understanding, and coordination capabilities as humans. They can autonomously learn skills from perceived data information and quickly adapt to environmental changes, thus being able to perform complex tasks.

[0003] Existing robot autonomous learning methods usually rely on a large amount of training data to enable the robot to train the skill model for a single type of work task. When faced with different types of work tasks, the skill model often cannot effectively complete the work task and needs to be retrained for the current type of work task, resulting in poor generalization of the robot's autonomous learning. Summary of the invention

[0004] The present invention provides a robot autonomous learning method, device, equipment and storage medium, which can randomly generate training data in a virtual environment to achieve diversity of training data, and can perform skill training on the robot for different types of tasks according to the training data to generate robot skills. When faced with different types of tasks, a variational reasoning method is used to predict the potential skill vector of the task, and then a robot control strategy for completing the task is obtained based on the robot skills generated by the training, without the need to re-perform skill training for the current type of work task, thereby improving the generalization of the robot's autonomous learning.

[0005] In order to solve the above technical problems, a first aspect of an embodiment of the present invention provides a robot autonomous learning method, comprising:

[0006] Based on a pre-built virtual environment, obtaining environment parameters automatically generated by the virtual environment;

[0007] Generating training data through a deep learning model according to the environmental parameters;

[0008] Generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data;

[0009] Determine a state space of the robot according to the task to be performed, and based on the state space, use a variational inference method to predict a potential skill vector of the task to be performed, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skill;

[0010] According to the robot skills and the potential skill vector, a reinforcement learning algorithm is used to obtain a robot control strategy for completing the task to be performed.

[0011] As a preferred embodiment, the method further comprises:

[0012] Based on a predetermined source domain for representing a virtual environment and a target domain for representing a real environment, a preset neural network model is used to learn a mapping from the source domain to the target domain, and an error between the source domain and the target domain is obtained according to the mapping;

[0013] According to the mapping situation, the neural network model is used to update the state space of the source domain to reduce the error between the source domain and the target domain.

[0014] As a preferred embodiment, the method further comprises:

[0015] In response to an incremental learning instruction input by a user, the robot control strategy is updated according to the incremental learning information in the incremental learning instruction.

[0016] As a preferred solution, the deep learning model is a generative adversarial network model;

[0017] Then, generating training data through a deep learning model according to the environmental parameters specifically includes:

[0018] According to the environmental parameters, obtaining distribution characteristics of the environmental parameters through a generator in the generative adversarial network model;

[0019] The training data is randomly generated by the generator according to the distribution characteristics.

[0020] As a preferred solution, the hierarchical reinforcement learning framework includes an upper-level decision model and a lower-level decision model;

[0021] Then, generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data specifically includes:

[0022] According to the training data and based on the upper-level decision model, the task sequence at each moment is planned by the MPC algorithm;

[0023] Based on the lower-level decision model, motion planning is performed on the task sequence at each moment through the DQN algorithm to generate the at least one robot skill.

[0024] As a preferred solution, the method specifically obtains the neural network model in advance through the following steps:

[0025] Step S61, based on the LSTM neural network, obtaining the initial state of the target domain;

[0026] Step S62, setting the state of the source domain to be the same as the initial state of the target domain;

[0027] Step S63, according to the preset behavior strategy corresponding to each state, respectively, on the source domain and the target domain, perform action sampling at the current state of the source domain and the current state of the target domain, to obtain state transition information, wherein the state transition information includes the next state of the source domain and the next state of the target domain;

[0028] Step S64, setting the next state of the source domain to be the same as the next state of the target domain;

[0029] Repeat step S63 to step S64 until all states of the source domain are set, and use the LSTM neural network with all states set as the neural network model.

[0030] As a preferred solution, in response to the incremental learning instruction input by the user, updating the robot control strategy according to the incremental learning information in the incremental learning instruction specifically includes:

[0031] In response to the incremental learning instruction, acquiring incremental learning information, wherein the incremental learning information includes initial position information of the modified motion trajectory formed according to the user instruction, intersection position information of the original motion trajectory and the modified motion trajectory, and real-time position information of the modified motion trajectory;

[0032] According to the incremental learning information, the original motion trajectory is updated by the following formula:

[0033]

[0034] in, represents the intersection position information of the original motion trajectory and the modified motion trajectory, represents the position information of the corrected motion trajectory at time t, represents the initial position information of the corrected motion trajectory, Represents the position information of the end of the corrected new motion trajectory at time t;

[0035] In response to a stop learning instruction input by a user, the robot control strategy is updated according to the position information of the end of the new motion trajectory at the current moment.

[0036] A second aspect of an embodiment of the present invention provides a robot autonomous learning device, comprising:

[0037] An environment parameter acquisition module, used to acquire environment parameters automatically generated by a pre-built virtual environment based on the virtual environment;

[0038] A training data generation module, used to generate training data through a deep learning model according to the environmental parameters;

[0039] A robot skill generation module, configured to generate at least one robot skill using a hierarchical reinforcement learning framework based on the training data;

[0040] a potential skill vector prediction module, configured to determine a state space of the robot according to a task to be performed, and based on the state space, predict a potential skill vector of the task to be performed using a variational inference method, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skill;

[0041] A robot control strategy acquisition module is used to obtain a robot control strategy for completing the task to be performed using a reinforcement learning algorithm based on the robot skills and the potential skill vector.

[0042] A third aspect of an embodiment of the present invention provides a robot autonomous learning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot autonomous learning method as described in any one of the first aspects is implemented.

[0043] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the robot autonomous learning method as described in any one of the first aspects.

[0044] Compared with the prior art, the beneficial effect of the embodiments of the present invention lies in that training data can be randomly generated in a virtual environment to achieve diversity in training data, and the robot can be trained for different types of tasks based on the training data to generate robot skills. When faced with different types of tasks, a variational reasoning method is used to predict the potential skill vector of the task, and then a robot control strategy for completing the task is obtained based on the robot skills generated by the training. There is no need to re-train skills for the current type of work task, thereby improving the generalization of the robot's autonomous learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a robot autonomous learning method provided by an embodiment of the present invention;

[0046] Figure 2 It is a structural schematic diagram of a robot autonomous learning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] See also Figure 1 The first aspect of the embodiment of the present invention provides a robot autonomous learning method, including steps S1 to S5, which are as follows:

[0049] Step S1, based on a pre-built virtual environment, obtaining environment parameters automatically generated by the virtual environment.

[0050] It should be noted that the embodiment of the present invention pre-builds a virtual environment for robot training through simulation software such as Unity3D. Since the cost of training robots in real environments is very high and time-consuming, the robot is trained in a virtual environment. Based on the pre-built virtual environment, environmental parameters automatically generated by the virtual environment are obtained. The environmental parameters include physical parameters such as geometry, friction, gravity, various observation noises, gravity factors, joint friction, and the position and direction of observation.

[0051] Step S2: Generate training data through a deep learning model according to the environmental parameters.

[0052] Specifically, according to the environmental parameters, the potential distribution characteristics in the environmental parameters are captured by a deep learning model, and training data is randomly generated according to the distribution characteristics.

[0053] Step S3, generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data.

[0054] It should be noted that in order to solve the problem of sparse rewards, the present invention uses a hierarchical reinforcement learning framework to generate at least one robot skill based on the training data generated in step S2. The robot skill here can be understood as the low-level skill required for the robot to perform a single type of simple task.

[0055] Step S4, determining the state space of the robot according to the task to be performed, and based on the state space, using a variational inference method to predict a potential skill vector of the task to be performed, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skills.

[0056] It should be noted that the embodiment of the present invention uses state space to describe the robot's tasks to be performed, and the robot coordination process can be regarded as a mapping from state space S to action space A. In a multi-task scenario, a set of robot skill numbers is pre-defined using the set K = {1,...,I}, and the reward function of each robot skill is r(s,a). Since multiple robot skills are required to complete the tasks to be performed in a multi-task scenario, it is necessary to predict the potential skill vector of the tasks to be performed, so as to generate a robot control strategy for completing the tasks to be performed based on the potential skill vector. The state space of the robot is determined according to the tasks to be performed, and based on the state space, a variational inference method is used to predict the potential skill vector of the tasks to be performed, wherein the potential skill vector has a one-to-one correspondence with one of all robot skills.

[0057] It is worth noting that the variational reasoning method adopted in the embodiment of the present invention consists of a reinforcement learning part, a reasoning part and a model predictive control part of the policy gradient. For the training of the reinforcement learning part, a proximal policy optimization algorithm is used to train the policy and embedding network of the reinforcement learning, and a multivariate Gaussian distribution is used to represent the policy, embedding and reasoning functions, and then the average and diagonal covariance are realized by the output parameterization of the multilayer perceptron. Finally, the policy distribution and the embedding distribution are jointly optimized using the reinforcement learning algorithm, and the reasoning function distribution is trained using supervised learning and cross entropy loss functions, and finally a potential skill vector sequence is obtained. In order to assist in learning the potential skill vector of the task to be performed, the reasoning function adopts the idea of ​​the variational reasoning method, approximates it with a simple distribution function, and predicts the potential skill vector. In addition, the reasoning function can use cross entropy or KL divergence (Kullback-Leibler divergence) to measure the difference between the posterior distribution and the simple distribution function. Before the robot executes the robot control strategy to complete the task, the MPC (model predictive control) method is used to simulate the potential skill vector online to evaluate the potential skill vector in the simulation.

[0058] Step S5, using a reinforcement learning algorithm to obtain a robot control strategy for completing the task to be performed based on the robot skills and the potential skill vector.

[0059] A robot autonomous learning method provided by an embodiment of the present invention can randomly generate training data in a virtual environment to achieve diversity in the training data. The robot can be trained for different types of tasks based on the training data to generate robot skills. When faced with different types of tasks, a variational reasoning method is used to predict the potential skill vector of the task. Then, based on the robot skills generated by the training, a robot control strategy for completing the task is obtained. There is no need to re-train the skills for the current type of work task, thereby improving the generalization of the robot's autonomous learning.

[0060] As a preferred embodiment, the method further comprises:

[0061] Based on a predetermined source domain for representing a virtual environment and a target domain for representing a real environment, a preset neural network model is used to learn a mapping from the source domain to the target domain, and an error between the source domain and the target domain is obtained according to the mapping;

[0062] According to the mapping situation, the neural network model is used to update the state space of the source domain to reduce the error between the source domain and the target domain.

[0063] It should be noted that in order to solve the problem that the virtual environment has errors with the real environment due to modeling errors, the domain adaptation method can learn the mapping from the virtual environment to the real environment. The virtual environment is used as the source domain, denoted as D s , the real environment is taken as the target domain, denoted as D t , each domain can be represented by a Markov decision process<S,A,p,r,γ> Indicates, where S represents the state space, A represents the action space, p represents the environment transition probability, r represents the reward value, and γ represents the attenuation coefficient. Under the assumption that D s and D t Under the condition of sharing the same action space A, due to the error between the virtual environment and the real environment, even if the two domains are similar in structure, the rewards r generated may be different. Therefore, the embodiment of the present invention uses a preset neural network model to learn the mapping from the source domain to the target domain based on the predetermined source domain for representing the virtual environment and the target domain for representing the real environment, and obtains the error between the source domain and the target domain according to the mapping; according to the mapping, the neural network model is used to update the state space of the source domain to reduce the error between the source domain and the target domain, so that the motion trajectory of the simulated object in the source domain is close to the motion trajectory of the robot in the target domain, thereby improving the simulation quality in the source domain.

[0064] As a preferred embodiment, the method further comprises:

[0065] In response to an incremental learning instruction input by a user, the robot control strategy is updated according to the incremental learning information in the incremental learning instruction.

[0066] It should be noted that, since the robot control strategy generated after autonomous learning may still be unable to adapt to the current task to be performed, resulting in the inability to complete the task, the embodiment of the present invention can respond to the incremental learning instructions input by the user and update the robot control strategy through the user's intervention, thereby achieving the purpose of incremental learning. The user's instruction information, such as gesture information and voice information, is collected by sensors pre-set in the outside world, and the user's instruction information is processed and converted into incremental learning instructions. In response to the incremental learning instructions input by the user, the robot control strategy is updated according to the incremental learning information in the incremental learning instructions.

[0067] As a preferred solution, the deep learning model is a generative adversarial network model;

[0068] Then, generating training data through a deep learning model according to the environmental parameters specifically includes:

[0069] According to the environmental parameters, obtaining distribution characteristics of the environmental parameters through a generator in the generative adversarial network model;

[0070] The training data is randomly generated by the generator according to the distribution characteristics.

[0071] It should be noted that the generative adversarial network model is mainly composed of a generator and a discriminator. The generator can capture the potential distribution characteristics of environmental parameters and randomly generate training data based on the distribution characteristics; the discriminator is used to determine whether the input data is real collected data or generated virtual data. In the process of training the generative adversarial network model, the corresponding loss function can be set for the generator and the discriminator, and the back-propagation algorithm can be used to update the parameters of the model until it converges to the Nash equilibrium.

[0072] As a preferred solution, the hierarchical reinforcement learning framework includes an upper-level decision model and a lower-level decision model;

[0073] Then, generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data specifically includes:

[0074] According to the training data and based on the upper-level decision model, the task sequence at each moment is planned by the MPC algorithm;

[0075] Based on the lower-level decision model, motion planning is performed on the task sequence at each moment through the DQN algorithm to generate the at least one robot skill.

[0076] It should be noted that for each moment in the task process, the hierarchical reinforcement learning framework selects a robot action according to the task sequence in the current robot state, then switches to a new state, and obtains the reward of the action strategy, which is the probability distribution of the action sequence in a certain state. According to the training data generated by the generative adversarial network model, based on the upper-level decision model, the task sequence at each moment is planned through the MPC (Model Predictive Control) algorithm; based on the lower-level decision model, the motion planning of the task sequence at each moment is performed through the DQN (Deep Q-Learning Network) algorithm to generate at least one robot skill.

[0077] Taking the task of assembling parts as an example, for the upper-level decision-making process, through the idea of ​​the MPC (Model Predictive Control) algorithm, traverse every possible action in the current state, try to select every optional assembly part, perform forward simulation on each action, and plan the assembly parts until the end of the assembly part task; for the lower-level decision-making process, according to the assembly part sequence obtained by the upper-level decision, the DQN (Deep Q-Learning Network) algorithm is used to perform motion planning in the current state, and according to the image of the current state, the difference between the image of the current state and the target configuration is calculated to ensure that the corresponding position is reached quickly and safely to complete the assembly part task. Among them, the embodiment of the present invention pre-learns the objective function corresponding to each action in a specific state through the DQN (Deep Q-Learning Network) algorithm, and after the data of the assembly parts are collected, the gradient descent algorithm is used to optimize the objective function corresponding to each action.

[0078] As a preferred solution, the method specifically obtains the neural network model in advance through the following steps:

[0079] Step S61, based on the LSTM neural network, obtaining the initial state of the target domain;

[0080] Step S62, setting the state of the source domain to be the same as the initial state of the target domain;

[0081] Step S63, according to the preset behavior strategy corresponding to each state, respectively, on the source domain and the target domain, perform action sampling at the current state of the source domain and the current state of the target domain, to obtain state transition information, wherein the state transition information includes the next state of the source domain and the next state of the target domain;

[0082] Step S64, setting the next state of the source domain to be the same as the next state of the target domain;

[0083] Repeat step S63 to step S64 until all states of the source domain are set, and use the LSTM neural network with all states set as the neural network model.

[0084] As a preferred solution, in response to the incremental learning instruction input by the user, updating the robot control strategy according to the incremental learning information in the incremental learning instruction specifically includes:

[0085] In response to the incremental learning instruction, acquiring incremental learning information, wherein the incremental learning information includes initial position information of the modified motion trajectory formed according to the user instruction, intersection position information of the original motion trajectory and the modified motion trajectory, and real-time position information of the modified motion trajectory;

[0086] According to the incremental learning information, the original motion trajectory is updated by the following formula:

[0087]

[0088] in, represents the intersection position information of the original motion trajectory and the modified motion trajectory, represents the position information of the corrected motion trajectory at time t, represents the initial position information of the corrected motion trajectory, Represents the position information of the end of the corrected new motion trajectory at time t;

[0089] In response to a stop learning instruction input by a user, the robot control strategy is updated according to the position information of the end of the new motion trajectory at the current moment.

[0090] It is worth noting that when the user inputs the incremental learning instruction and starts to correct the original motion trajectory, the robot's end effector can sense the current user's correction of the original motion trajectory, thereby obtaining the intersection position information of the original motion trajectory and the corrected motion trajectory. The corrected motion trajectory is the motion trajectory formed by the user's demonstration. Furthermore, through the formula A new motion trajectory updated in real time is obtained, and the new motion trajectory is the motion trajectory of the original motion trajectory starting from the intersection point where the original motion trajectory intersects with the modified motion trajectory and connecting with the modified motion trajectory. When a stop learning instruction input by the user is received, in response to the stop learning instruction, the position information of the end of the new motion trajectory at the current moment is obtained through the end effector of the robot, which is the last position point of the new motion trajectory, and then starting from the last position point of the new motion trajectory, the remaining position points of the original motion trajectory are executed until the task is completed, thereby updating the robot control strategy.

[0091] A robot autonomous learning method provided by an embodiment of the present invention can randomly generate training data in a virtual environment to achieve diversity in training data. It can train the robot for different types of tasks based on the training data to generate robot skills. When facing different types of tasks, a variational reasoning method is used to predict the potential skill vector of the task, and then a robot control strategy for completing the task is obtained based on the robot skills generated by the training. There is no need to retrain skills for the current type of work task, which improves the generalization of the robot's autonomous learning. In addition, the error between the virtual environment and the real environment is reduced through a neural network model, thereby improving the effect of the robot's autonomous learning. The robot control strategy can be updated in response to the user's incremental learning instructions, further improving the generalization of the robot's autonomous learning.

[0092] See also Figure 2 A second aspect of an embodiment of the present invention provides a robot autonomous learning device, comprising:

[0093] The environment parameter acquisition module 201 is used to acquire the environment parameters automatically generated by the virtual environment based on the pre-built virtual environment;

[0094] A training data generation module 202, configured to generate training data through a deep learning model according to the environmental parameters;

[0095] A robot skill generation module 203, configured to generate at least one robot skill based on the training data using a hierarchical reinforcement learning framework;

[0096] a potential skill vector prediction module 204, configured to determine a state space of the robot according to the task to be performed, and based on the state space, predict a potential skill vector of the task to be performed using a variational inference method, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skill;

[0097] The robot control strategy acquisition module 205 is used to obtain the robot control strategy for completing the task to be performed by using a reinforcement learning algorithm according to the robot skills and the potential skill vector.

[0098] As a preferred solution, the device further includes a source domain status updating module 206, which is used to:

[0099] Based on a predetermined source domain for representing a virtual environment and a target domain for representing a real environment, a preset neural network model is used to learn a mapping from the source domain to the target domain, and an error between the source domain and the target domain is obtained according to the mapping;

[0100] According to the mapping situation, the neural network model is used to update the state space of the source domain to reduce the error between the source domain and the target domain.

[0101] As a preferred solution, the device further includes an incremental learning module 207, which is used to:

[0102] In response to an incremental learning instruction input by a user, the robot control strategy is updated according to the incremental learning information in the incremental learning instruction.

[0103] As a preferred solution, the deep learning model is a generative adversarial network model;

[0104] Then, the training data generation module 202 is used to generate training data through a deep learning model according to the environmental parameters, specifically including:

[0105] According to the environmental parameters, obtaining distribution characteristics of the environmental parameters through a generator in the generative adversarial network model;

[0106] The training data is randomly generated by the generator according to the distribution characteristics.

[0107] As a preferred solution, the hierarchical reinforcement learning framework includes an upper-level decision model and a lower-level decision model;

[0108] Then, the robot skill generation module 203 is used to generate at least one robot skill according to the training data using a hierarchical reinforcement learning framework, specifically including:

[0109] According to the training data and based on the upper-level decision model, the task sequence at each moment is planned by the MPC algorithm;

[0110] Based on the lower-level decision model, motion planning is performed on the task sequence at each moment through the DQN algorithm to generate the at least one robot skill.

[0111] As a preferred solution, the device further includes a neural network model acquisition module 208, which is specifically used for the following steps:

[0112] Step S61, based on the LSTM neural network, obtaining the initial state of the target domain;

[0113] Step S62, setting the state of the source domain to be the same as the initial state of the target domain;

[0114] Step S63, according to the preset behavior strategy corresponding to each state, respectively, on the source domain and the target domain, perform action sampling at the current state of the source domain and the current state of the target domain, to obtain state transition information, wherein the state transition information includes the next state of the source domain and the next state of the target domain;

[0115] Step S64, setting the next state of the source domain to be the same as the next state of the target domain;

[0116] Repeat step S63 to step S64 until all states of the source domain are set, and use the LSTM neural network with all states set as the neural network model.

[0117] As a preferred solution, the incremental learning module 207 is used to respond to the incremental learning instruction input by the user and update the robot control strategy according to the incremental learning information in the incremental learning instruction, specifically including:

[0118] In response to the incremental learning instruction, acquiring incremental learning information, wherein the incremental learning information includes initial position information of the modified motion trajectory formed according to the user instruction, intersection position information of the original motion trajectory and the modified motion trajectory, and real-time position information of the modified motion trajectory;

[0119] According to the incremental learning information, the original motion trajectory is updated by the following formula:

[0120]

[0121] in, represents the intersection position information of the original motion trajectory and the modified motion trajectory, represents the position information of the corrected motion trajectory at time t, represents the initial position information of the corrected motion trajectory, Represents the position information of the end of the corrected new motion trajectory at time t;

[0122] In response to a stop learning instruction input by a user, the robot control strategy is updated according to the position information of the end of the new motion trajectory at the current moment.

[0123] It should be noted that a robot autonomous learning device provided in an embodiment of the present invention can implement all the processes of the robot autonomous learning method described in any of the above embodiments. The functions of each module in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the robot autonomous learning method described in the above embodiments, and will not be repeated here.

[0124] A third aspect of an embodiment of the present invention provides a robot autonomous learning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot autonomous learning method as described in any embodiment of the first aspect is implemented.

[0125] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory. The terminal device may also include an input / output device, a network access device, a bus, etc.

[0126] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0127] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0128] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the robot autonomous learning method as described in any embodiment of the first aspect.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary hardware platform, and of course can also be implemented entirely by hardware. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0130] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A robot autonomous learning method, characterized in that: include: Based on a pre-built virtual environment, obtaining environment parameters automatically generated by the virtual environment; Generating training data through a deep learning model according to the environmental parameters; Generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data; Determine a state space of the robot according to the task to be performed, and based on the state space, use a variational inference method to predict a potential skill vector of the task to be performed, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skill; According to the robot skills and the potential skill vector, a reinforcement learning algorithm is used to obtain a robot control strategy for completing the task to be performed; Wherein, the hierarchical reinforcement learning framework includes an upper-level decision model and a lower-level decision model; Then, generating at least one robot skill using a hierarchical reinforcement learning framework based on the training data specifically includes: According to the training data and based on the upper-level decision model, the task sequence at each moment is planned by the MPC algorithm; Based on the lower-level decision model, motion planning is performed on the task sequence at each moment through the DQN algorithm to generate the at least one robot skill.

2. The robot autonomous learning method according to claim 1, characterized in that: The method further comprises: Based on a predetermined source domain for representing a virtual environment and a target domain for representing a real environment, a preset neural network model is used to learn a mapping from the source domain to the target domain, and an error between the source domain and the target domain is obtained according to the mapping; According to the mapping situation, the neural network model is used to update the state space of the source domain to reduce the error between the source domain and the target domain.

3. The robot autonomous learning method according to claim 2, characterized in that: The method further comprises: In response to an incremental learning instruction input by a user, the robot control strategy is updated according to the incremental learning information in the incremental learning instruction.

4. The robot autonomous learning method according to claim 3, characterized in that: The deep learning model is a generative adversarial network model; Then, generating training data through a deep learning model according to the environmental parameters specifically includes: According to the environmental parameters, obtaining distribution characteristics of the environmental parameters through a generator in the generative adversarial network model; The training data is randomly generated by the generator according to the distribution characteristics.

5. The robot autonomous learning method according to claim 4, characterized in that: The method specifically obtains the neural network model in advance through the following steps: Step S61, based on the LSTM neural network, obtaining the initial state of the target domain; Step S62, setting the state of the source domain to be the same as the initial state of the target domain; Step S63, according to the preset behavior strategy corresponding to each state, respectively, on the source domain and the target domain, perform action sampling at the current state of the source domain and the current state of the target domain, to obtain state transition information, wherein the state transition information includes the next state of the source domain and the next state of the target domain; Step S64, setting the next state of the source domain to be the same as the next state of the target domain; Repeat step S63 to step S64 until all states of the source domain are set, and use the LSTM neural network with all states set as the neural network model.

6. The robot autonomous learning method according to claim 5, characterized in that: The step of updating the robot control strategy in response to the incremental learning instruction input by the user according to the incremental learning information in the incremental learning instruction specifically includes: In response to the incremental learning instruction, acquiring incremental learning information, wherein the incremental learning information includes initial position information of the modified motion trajectory formed according to the user instruction, intersection position information of the original motion trajectory and the modified motion trajectory, and real-time position information of the modified motion trajectory; According to the incremental learning information, the original motion trajectory is updated by the following formula: in, represents the intersection position information of the original motion trajectory and the modified motion trajectory, represents the position information of the corrected motion trajectory at time t, represents the initial position information of the corrected motion trajectory, Represents the position information of the end of the corrected new motion trajectory at time t; In response to a stop learning instruction input by a user, the robot control strategy is updated according to the position information of the end of the new motion trajectory at the current moment.

7. A robot autonomous learning device, characterized in that: include: An environment parameter acquisition module, used to acquire environment parameters automatically generated by a pre-built virtual environment based on the virtual environment; A training data generation module, used to generate training data through a deep learning model according to the environmental parameters; A robot skill generation module, configured to generate at least one robot skill using a hierarchical reinforcement learning framework based on the training data; a potential skill vector prediction module, configured to determine a state space of the robot according to a task to be performed, and based on the state space, predict a potential skill vector of the task to be performed using a variational inference method, wherein the potential skill vector has a one-to-one correspondence with one of the at least one robot skill; A robot control strategy acquisition module, used to obtain a robot control strategy for completing the task to be performed using a reinforcement learning algorithm according to the robot skill and the potential skill vector; Wherein, the hierarchical reinforcement learning framework includes an upper-level decision model and a lower-level decision model; Then, the robot skill generation module is used to generate at least one robot skill according to the training data using a hierarchical reinforcement learning framework, specifically including: According to the training data and based on the upper-level decision model, the task sequence at each moment is planned by the MPC algorithm; Based on the lower-level decision model, motion planning is performed on the task sequence at each moment through the DQN algorithm to generate the at least one robot skill.

8. A robot autonomous learning device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the robot autonomous learning method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the robot autonomous learning method according to any one of claims 1 to 6.

Images