An Unsupervised Skill Learning System Based on Maximum Entropy Intrinsic Reward

Abstract

This invention provides an unsupervised skill learning system based on maximum entropy intrinsic reward, belonging to the field of robot autonomous action learning technology. It proposes a multi-object representation learning model that decouples objects in a scene from the background of an image in an unsupervised manner, obtaining compact multi-object representations. A maximum entropy intrinsic reward function centered on objects is constructed to measure the overall information entropy of the explored sample set. Task-independent reinforcement learning training is performed through intrinsic rewards to learn an efficient robot exploration strategy, driving the robot to explore as many new samples as possible in the environment. The pre-trained multi-object representation learning model and robot exploration strategy are used as initialization for the perception and control models of downstream task-specific learning, significantly improving the learning efficiency and effectiveness of downstream tasks.

Description

Technical Field

[0001] This invention belongs to the field of robot autonomous action learning technology, and in particular relates to an unsupervised skill learning system based on maximum entropy intrinsic reward. Background Technology

[0002] Benefiting from the rise of artificial intelligence algorithms and the rapid development of computing power in various hardware, robotics has played a vital role in numerous fields such as assembly, service, healthcare, security, rescue, and transportation. In daily life, service robots are required to perform a wide variety of tasks in unstructured environments. Therefore, enabling robots to efficiently and autonomously learn various skills has always been a goal we have been pursuing.

[0003] Traditional robot control methods can precisely control robots to complete specific tasks. However, these methods are highly dependent on the expertise and software programming skills of technicians, and require different robot trajectory distributions to be programmed for different task objectives. When faced with dynamically changing, unstructured scenarios, previously deployed robot skills may fail to generalize and adapt to new situations. Deep reinforcement learning methods can enable robots to autonomously learn task-related skills through exploration and exploitation; however, this learning approach requires the manual design of reward functions highly relevant to the specific target task. The efficiency and effectiveness of skill learning are highly dependent on the definition of the reward function and are very sensitive to small hyperparameters and constraints within it. Therefore, a good reward function definition requires the professional experience of technicians. Furthermore, the reward functions for many tasks are difficult to define accurately numerically. For example, to make a robot pour water naturally like a human, it is difficult to define whether the pouring process, posture, and trajectory are natural.

[0004] In daily life and work, when humans arrive at a new scene, they often explore the environment "aimlessly" and spontaneously, interacting with various objects to obtain relevant information. For robots, encouraging them to autonomously explore their environment and learn skills without a task-specific reward function is a significant challenge. Summary of the Invention

[0005] The purpose of this invention is to provide an unsupervised skill learning system based on maximum entropy intrinsic reward, characterized by the following steps:

[0006] S1: Build a virtual simulation environment for the robot, control the robot to interact with the environment through a random exploration strategy, and obtain interaction data after the interaction;

[0007] S2: Store the interaction data in the experience replay pool, and use the data in the experience replay pool to train a multi-object representation learning model. The multi-object representation learning model learns the structured representations of multiple objects from image observations with different backgrounds in an unsupervised manner.

[0008] S3: Based on the multi-object representation of learning, calculate the maximum entropy intrinsic reward function based on the multi-object representation, calculate the distance between the current observation representation and the k nearest neighbor observation representations in the experience replay pool through the maximum entropy intrinsic reward function, and then estimate the information entropy of the entire experience replay pool.

[0009] S4: Use the information entropy of the entire experience replay pool as the intrinsic reward function, train the robot's exploration strategy by maximizing the intrinsic reward function, and encourage the robot to interact with multiple objects in the environment to obtain interaction samples. Then, train the multi-object representation learning model further using the interaction samples.

[0010] S5: Based on the multi-object representation learning model and the pre-trained robot exploration strategy, specific skills are reinforced through the extrinsic reward function of the downstream task, thereby improving the sample efficiency and learning results of strategy learning.

[0011] Furthermore, in S1, the robot virtual simulation environment is implemented using OpenAlGym and mujoco simulators, including a Fetch robotic arm, an operating platform, and multiple operating objects of different colors. The robot simulation virtual environment controls the Fetch robotic arm to interact with multiple operating objects of different colors on the operating platform by sampling action instructions from a random exploration strategy, thereby obtaining a series of image sequences, i.e., interaction data.

[0012] Furthermore, in S2, the multi-object representation learning model includes an object branch and a background branch. The object encoder in the object branch extracts the object representation of each object from the image and generates the object mask and texture through the object decoder. The background encoder in the background branch extracts the background representation from the image and generates the background mask and texture through the background decoder.

[0013] Furthermore, the multi-object representation learning model is optimized solely through image reconstruction loss, without requiring labels for objects or background, and is represented as follows:

[0014]

[0015] Where I represents the original image, To reconstruct the image.

[0016] Further, in S3, the maximum entropy intrinsic reward function is used to estimate the information entropy of the entire experience replay pool. Specifically, based on the particle estimation method, each interaction sample is treated as a particle, and the average distance between each particle and the k nearest neighbor particles in the entire experience replay pool is calculated as the overall information entropy of the experience replay pool, thereby quantifying the diversity of the collected interaction samples. The overall information entropy is expressed as:

[0017]

[0018] Where b(k) is a correction term that depends on the hyperparameter k; It is the spatial distance between the i-th particle and its k-th nearest neighbor particle, expressed as:

[0019]

[0020] The overall information entropy is proportional to the distance between each instance and its k nearest neighbors. Therefore, the information entropy is expressed as:

[0021]

[0022] Where c is a constant; k is the current object representation z. i The k nearest neighbor particles; ||·|| is the Euclidean distance.

[0023] Furthermore, in S4, training the robot exploration strategy specifically involves: using information entropy as the intrinsic reward function and employing a proximal policy optimization algorithm in reinforcement learning to learn the robot exploration strategy. The input to the robot exploration strategy is a structured representation of multiple objects, and the output is an action used to directly control the robot.

[0024] The interaction sample specifically refers to the multi-object representation learning model being unsupervised trained using data from the continuous interaction between the robot and the environment, thereby adapting to the image distribution of objects in different postures and robustly extracting representations of multiple objects from the image.

[0025] Furthermore, in S5, the downstream task is a rearrangement task, specifically, the robot rearranges multiple objects on the operating table to their corresponding target positions, and the target positions are randomly generated on the operating table.

[0026] The extrinsic reward function is the reward function for a specific downstream task, defined as the distance between the currently observed multi-object representation and the multi-object representation of the target image.

[0027] Furthermore, in S5, during the reinforcement learning process of the specific skill, the pre-trained multi-object representation learning model and robot exploration strategy are used as parameters to initialize the perception model and control model. The multi-object representation learning model is further trained unsupervised using interactive data, and reinforcement learning is performed based on the task-specific external reward function, thereby improving the sample efficiency and generalization of robot reinforcement learning.

[0028] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in:

[0029] 1. This invention proposes a maximum entropy intrinsic reward function centered on objects. By maximizing the intrinsic reward function, i.e. the overall information entropy of the exploration samples, the robot is encouraged to conduct unsupervised exploration in unseen environments, thereby efficiently exploring various new states of objects and improving the quality of reinforcement learning training samples.

[0030] 2. This invention proposes an unsupervised multi-object decoupling algorithm that decouples multiple objects from different control scenarios without labeling. It learns compact object representations as structured representations for image observation, enabling the intrinsic reward function to efficiently calculate changes in objects within the scene while ignoring task-irrelevant factors such as the background. Furthermore, the learned structured representations can serve as policy input for reinforcement learning, significantly shortening the algorithm's training time and improving policy learning effectiveness.

[0031] 3. This invention proposes a robot skill learning paradigm of pre-training followed by transfer to a specific task. It uses a task-independent intrinsic reward function to train an exploration policy as policy initialization, and then uses a task-specific extrinsic reward function to learn specific skills. This can effectively reduce the interaction time and data requirements of downstream tasks by utilizing pre-training, and greatly improve the learning efficiency and performance of downstream specific skills. Attached Figure Description

[0032] Figure 1 The flowchart below shows an unsupervised skill learning system based on maximum entropy intrinsic reward according to the present invention.

[0033] Figure 2 This is a schematic diagram of the task environment for rearrangement in this invention;

[0034] Figure 3 This is a schematic diagram of the structure of the multi-object representation learning model in this invention. Detailed Implementation

[0035] The following will describe in more detail an unsupervised skill learning system based on maximum entropy intrinsic reward according to the present invention, with reference to the schematic diagrams, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving the advantageous effects of the invention. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0036] Example

[0037] This invention discloses an unsupervised skill learning system based on maximum entropy intrinsic reward. For example... Figure 1 As shown.

[0038] This embodiment uses the Fetch robotic arm as an example; its control workstation is equipped with Ubuntu 22.04, an Intel Core i7-13700K (16 cores, 24 threads, 5.4GHz turbo boost), an NVIDIA GTX 3070*2 GPU, and 32GB DDR4 memory. This patent also requires a general-purpose RGB camera to observe the overall scene of the control panel, including the following steps:

[0039] S1: This invention constructs a robot simulation environment based on the OpenAIGym and Mujoco simulation platforms. A schematic diagram of this environment is shown below. Figure 2 As shown, the environment includes a UR5 robotic arm, a control panel, and multiple wooden blocks of different colors, used to simulate a robot control scenario. This invention employs a random exploration strategy to generate action commands, enabling the robot to interact with multiple objects in the environment and collecting sequence data as training data.

[0040] Based on the collected dataset, a multi-object representation learning model is trained. The network structure diagram of the model is shown below. Figure 3 As shown. The model contains an object branch and a background branch, which model objects and the background from the image, respectively. The object encoder extracts the object representation of each object from the image and generates the object mask and texture through the object decoder. The background encoder in the background module extracts the background representation from the image and generates the background mask and texture using the background decoder. The encoder network consists of three convolutional layers with a size of 5, a stride of 2, and a ReLU activation function. The decoder network consists of three convolutional layers with a size of 5, a stride of 0.5, and a ReLU activation function, two fully connected layers, and an activation function layer. The notation is as follows:

[0041] [Mask i RGB i ] = Dec i (z i ) = Dec i(Enc i (I)), i = obj, bg

[0042]

[0043] The entire model is optimized solely through image reconstruction loss, without requiring object or background labels, as shown below:

[0044]

[0045] By optimizing the image reconstruction loss, the masks of objects and robots compete with each other without overlapping. During training, background and object features are represented using means and variances of the same dimension, with their respective variances fixed, and only the means are learned. Without the presence of objects and background, the multi-object representation learning model can clearly decouple the textures and masks of multiple objects and the background.

[0046] S2: Based on the multi-object representations learned above, the richness of the entire experience replay pool is estimated. During reinforcement learning training, interactive samples are stored in the experience replay pool. This invention uses information entropy to quantify the richness of the experience replay pool. Assuming there are n samples from an unknown distribution in the experience replay pool, this invention estimates the information entropy of all data using a particle-based nonparametric entropy estimation method. Treating each sample as a particle, the overall information entropy is expressed as:

[0047]

[0048] Where b(k) is a correction term dependent on the hyperparameter k. It is the spatial distance between the i-th particle and its k-th nearest neighbor particle, specifically expressed as:

[0049]

[0050] Therefore, the overall information entropy is proportional to the distance between each particle and its k nearest neighbors. Thus, we express the information entropy using Equation 1:

[0051]

[0052] Where c is a constant, and k represents the current object representation z. i The k nearest neighbor particles, where ||·|| is the Euclidean distance.

[0053] To encourage robots to continuously explore new states in the environment, we define the task-independent intrinsic reward as the information entropy of the overall experience replay pool, as shown in Formula 1.

[0054] S3: During training, after each set of interaction samples is collected, they are stored in the experience replay pool, and a portion of samples are randomly selected for training. These samples are first used to unsupervised train the multi-object representation learning model, enabling the perception model to adapt to the image distribution of objects in different poses and robustly extract multi-object representations from the image.

[0055] Furthermore, through the intrinsic reward function in step S2 above, a robot exploration policy is learned using the Proximal Policy Optimization (PPO) algorithm in reinforcement learning. The policy input is a structured representation of multiple objects, and the output is an action that can be used to directly control the robot.

[0056] S4: Through continuous unsupervised exploration and training, a pre-trained multi-object representation learning model and a robot exploration strategy were obtained. To test the effectiveness of the pre-trained exploration strategy, we used the pre-trained model as the initialization parameters for the perception and control models, and then transferred both models to a specific downstream task.

[0057] The downstream task in this invention is a repositioning task, where the robot needs to reposition multiple objects on the worktable to their corresponding target positions. The target placement positions are randomly generated on the worktable, and it's possible for one object to be stacked on top of another. Based on the task objective, this invention defines the extrinsic reward function as the distance between the current observation and the target image. In this invention, the extrinsic reward function can be defined as the distance between the currently observed multi-object representation and the multi-object representation of the target image.

[0058] Based on extrinsic rewards, a task-specific robot control policy is learned using the Proximal Policy Optimization (PPO) algorithm in reinforcement learning. Furthermore, during training, samples taken from the experience replay pool not only train the robot control policy but also further train a multi-object representation learning model in unsupervised mode. Through the "general knowledge" about environmental perception and dynamics learned by the pre-trained model, the robot can quickly learn specific skills, greatly improving the sample efficiency and final performance of robot skill learning.

[0059] A perception model based on multi-object representation, used as input to the control strategy and for calculating intrinsic and extrinsic reward functions, can highly efficiently represent changes in objects while eliminating background and other interfering information. Learning exploration strategies based on the maximum entropy intrinsic reward function encourages the robot to efficiently explore new states, significantly improving the efficiency and effectiveness of subsequent skill learning for specific downstream tasks.

[0060] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. An unsupervised skill learning system based on maximum entropy intrinsic reward, characterized in that, Includes the following steps: S1: Build a virtual simulation environment for the robot, control the robot to interact with the environment through a random exploration strategy, and obtain interaction data after the interaction; S2: Store the interaction data in the experience replay pool, and use the data in the experience replay pool to train a multi-object representation learning model. The multi-object representation learning model learns the structured representations of multiple objects from image observations with different backgrounds in an unsupervised manner. S3: Based on the multi-object representation of learning, calculate the maximum entropy intrinsic reward function based on the multi-object representation, calculate the distance between the current observation representation and the k nearest neighbor observation representations in the experience replay pool through the maximum entropy intrinsic reward function, and then estimate the information entropy of the entire experience replay pool. S4: Use the information entropy of the entire experience replay pool as the intrinsic reward function, train the robot's exploration strategy by maximizing the intrinsic reward function, and encourage the robot to interact with multiple objects in the environment to obtain interaction samples. Then, train the multi-object representation learning model further using the interaction samples. S5: Based on the multi-object representation learning model and the pre-trained robot exploration strategy, specific skills are reinforced through the external reward function of the downstream task, thereby improving the sample efficiency and learning results of strategy learning. In step S3, the maximum entropy intrinsic reward function is used to estimate the information entropy of the entire experience replay pool. Specifically, based on the particle estimation method, each interaction sample is treated as a particle, and the average distance between each particle and its k nearest neighbor particles in the entire experience replay pool is calculated as the overall information entropy of the experience replay pool, thereby quantifying the diversity of the collected interaction samples. The overall information entropy is expressed as: ； Where b(k) is a correction term that depends on the hyperparameter k; It is the spatial distance between the i-th particle and its k-th nearest neighbor particle, expressed as: ； The overall information entropy is proportional to the distance between each instance and its k nearest neighbors. Therefore, the information entropy is expressed as: ； in, It is a constant; Characterize z for the current object i of The nearest neighbor particle; It is a Euclidean distance.

2. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 1, characterized in that, In S1, the robot virtual simulation environment is implemented using OpenAI Gym and mujoco simulators, including a Fetch robotic arm, an operating platform, and multiple operating objects of different colors. The robot simulation virtual environment controls the Fetch robotic arm to interact with multiple operating objects of different colors on the operating platform by sampling action commands from a random exploration strategy, thereby obtaining a series of image sequences, i.e., interaction data.

3. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 2, characterized in that, In S2, the multi-object representation learning model includes an object branch and a background branch. The object encoder of the object branch extracts the object representation of each object from the image and generates the object mask and texture through the object decoder. The background encoder in the background branch extracts background representation from the image and generates a background mask and texture through the background decoder.

4. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 3, characterized in that, The multi-object representation learning model is optimized solely through image reconstruction loss, without requiring object or background labels, and is represented as follows: ； in, Represented as the original image, To reconstruct the image.

5. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 4, characterized in that, In S4, training the robot exploration strategy specifically involves: using information entropy as the intrinsic reward function and employing a short-range policy optimization algorithm in reinforcement learning to learn the robot exploration strategy. The input of the robot exploration strategy is a structured representation of multiple objects, and the output is the action used to directly control the robot. The interaction sample specifically refers to the multi-object representation learning model being unsupervised trained using data from the continuous interaction between the robot and the environment, thereby adapting to the image distribution of objects in different postures and robustly extracting representations of multiple objects from the image.

6. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 5, characterized in that, In S5, the downstream task is a rearrangement task, which specifically involves the robot rearranging multiple objects on the operating table to their corresponding target positions, with the target positions being randomly generated on the operating table. The extrinsic reward function is the reward function for a specific downstream task, defined as the distance between the currently observed multi-object representation and the multi-object representation of the target image.

7. The unsupervised skill learning system based on maximum entropy intrinsic reward according to claim 6, characterized in that, In step S5, during the reinforcement learning process of the specific skill, the pre-trained multi-object representation learning model and robot exploration strategy are used as parameters to initialize the perception model and control model. The multi-object representation learning model is further trained in unsupervised manner using interactive data, and reinforcement learning is performed based on the task-specific external reward function, thereby improving the sample efficiency and generalization of robot reinforcement learning.

Images