Intelligent voxel motion learning method based on joint grouping strategy

Abstract

An intelligent meta-action learning method based on joint grouping strategy is used for robot automatic control. It controls the robot to quickly learn new tasks through the spatiotemporal fusion of joint grouping and meta-action, thus avoiding a lot of time spent on training from scratch. An information theory objective is proposed to optimize the reinforcement learning algorithm to train the robot, so that a series of diverse meta-action strategies can be learned; the trajectory similarity between joints is calculated based on the dynamic time warping algorithm, and then the robot joints are grouped based on the hierarchical clustering algorithm; the meta-action control strategies of different joint groups are obtained according to the corresponding masks of the joint groups; a hierarchical robot overall control strategy network is constructed, and the meta-action fusion at the temporal and spatial levels is realized by linearly combining the weights of the meta-action strategies of each joint group, so that new tasks can be quickly learned; at the same time, the coordination and coherence of the robot's actions are enhanced, and better performance can be achieved on new tasks.

Description

Technical Field

[0001] The invention relates to an intelligent body action learning technology based on a joint grouping strategy, and belongs to the technical field of intelligent body autonomous action learning. Background Art

[0002] With the rise of artificial intelligence algorithms and the rapid development of various hardware devices, robotics has played an important role in many fields such as medical care, service, assembly, security, rescue, and transportation. The similarity between the robot skills required in various fields is low, and customized deployment of robots for a single task is time-consuming and laborious. Therefore, how to enable robots to learn a series of basic meta-actions from different tasks and further combine them into different skills is our goal. As the main executor of the task, it is particularly important for robots to study how to discover and utilize similarity knowledge under different action trajectories.

[0003] Traditional robot control methods are extremely dependent on the professional knowledge and software programming level of technicians, and different robot trajectory distributions need to be designed for specific task objectives. When the application scenario changes, the previously deployed robot skills cannot be reused. Deep reinforcement learning methods can enable robots to autonomously learn task-related skills through exploration and utilization. However, for some tasks, it is impossible to design a suitable reward function to clearly describe the task objectives. In addition, this learning method also requires manual design of corresponding reward functions for different tasks. The intelligent agent explores from scratch according to different reward functions until the training is completed to obtain a better control strategy. This learning method has low data utilization and may take hundreds or thousands of hours to collect data and train strategies in a simulation environment, which is impractical in real scenarios.

[0004] In daily life and work, humans are often able to decompose actions into different joint groups, such as arms, torsos, legs, etc., and then complete various tasks through different combinations of actions. How robots learn the required basic meta-actions to quickly learn new skills remains a challenge. Based on this, the present invention proposes an intelligent meta-action learning method based on a joint grouping strategy. Through this method, the robot can learn a series of distinguishable meta-actions based on an information-theoretic optimization objective, and then hierarchically combine meta-actions of different joint angles to complete new tasks, thereby ensuring that meta-actions are combined in time and space, avoiding reinforcement learning training from scratch. Summary of the invention

[0005] In view of the limitations and shortcomings of the prior art, the present invention proposes an intelligent meta-action learning method based on a joint grouping strategy. The idea is to model the robot's sequential decision-making problem as a Markol decision process, and train the robot based on an information theory objective to optimize the deep reinforcement learning algorithm, thereby establishing a meta-action library. After sampling the motion trajectory of each joint, the joints are grouped according to the dynamic time warping algorithm and the hierarchical clustering algorithm, and the meta-action is decomposed into multiple joint component actions. When the robot needs to perform a new task, the robot can combine the various joint component actions at the spatiotemporal level based on the hierarchical structure, so as to quickly perform reinforcement learning training and complete the new task.

[0006] The innovation of the present invention is:

[0007] (I) We propose an information theoretic objective to replace the reward function that is highly relevant to the task to optimize deep reinforcement learning. We learn meta-actions with high discrimination and low similarity through the maximum entropy strategy, explore the diversity of meta-actions, and thus establish a meta-action library for the robot.

[0008] (II) A robot joint grouping algorithm based on trajectory similarity is proposed. After reinforcement learning training of the basic task, the action trajectory of each joint is sampled. The similarity between joints is calculated based on the dynamic time warping algorithm, and finally the joint grouping is obtained based on the layer accumulation clustering algorithm, which reduces the difficulty of meta-action learning and makes the overall action after the meta-action of each joint group is combined smoother.

[0009] (III) A hierarchical control strategy based on spatiotemporal fusion of meta-actions is proposed, which can avoid training from scratch, greatly shorten the training time of deep reinforcement learning algorithms, and use prior knowledge of actions to learn new tasks faster and better.

[0010] Technical Solution

[0011] An intelligent meta-action learning method based on joint grouping strategy is characterized in that an information theory goal is proposed to optimize the reinforcement learning algorithm to train the robot, so that a series of diverse meta-action strategies are learned; a trajectory similarity calculation between joints is proposed based on a dynamic time warping algorithm, and then the grouping of robot joints is realized based on a hierarchical clustering algorithm; the meta-action control strategies of different joint groups are obtained according to the corresponding masks of the joint groups; a hierarchical robot overall control strategy network is constructed, and the weights of the meta-action strategies of each joint group are linearly combined to realize the meta-action fusion at the time level and the space level, so that new tasks can be learned quickly, avoiding the need to train the reinforcement learning algorithm from scratch, and greatly reducing the training time; at the same time, the coordination and coherence of the robot's actions are enhanced, so that better performance can be achieved on new tasks.

[0012] An intelligent voxel motion learning method based on joint grouping strategy, characterized in that the method is implemented by the following steps:

[0013] S1: First, we build a meta-action strategy network, and then propose an information-theoretic optimization objective to replace the reward function that is highly relevant to the task. We conduct deep reinforcement learning training on the robot in a simulation environment, and use the distinguishability of meta-actions to obtain a rich and diverse series of meta-actions.

[0014] S2: Use the existing meta-action strategy to sample a large number of trajectories, and use the time warping algorithm to calculate the similarity based on the trajectory of each joint, and then group the joints according to hierarchical clustering to obtain joint groups;

[0015] S3: Generate the corresponding joint group mask based on the joint group in S2, and combine the meta-action strategy obtained in step S1 to obtain the meta-action strategy network of different joint groups;

[0016] S4: In the new robot application scenario, build a hierarchical reinforcement learning control network to learn new tasks faster by combining the joint component actions obtained in step S3, and at the same time obtain a better reward function.

[0017] Furthermore, in step S1, the meta-action strategy network is a neural network, represented by π(A|S, Z), wherein the input is the state S, specifically the observation image of the environment and the robot itself taken by the monocular camera; the output is the action A of the robot, specifically a 56-dimensional vector composed of the joint angles and joint angular velocities of the robot, represented by Joint j＝[1∶56] In the network, the intermediate latent variable Z is constrained to a uniform distribution, expressed as Z~P(z). i The policy network is a meta-action, denoted as π i (A|S,z i ), i=0, 1, .., N.

[0018] Furthermore, in step S1, for deep reinforcement learning, a reward function of a specific task is not used for training, but an information theoretic optimization objective is designed to constrain the deep reinforcement learning algorithm. The information theoretic optimization objective includes three design principles, namely: (1) all states must be visited, and different states visited by different meta-actions can be distinguished, so the mutual information I(S; Z) between meta-actions and states needs to be maximized; (2) meta-actions are distinguished by states, because actions that do not change the environment and the robot's own state will not be observed, so the mutual information I(A; Z|S) between meta-actions and actions under the condition of states needs to be minimized; (3) skills that are as random as possible are learned to explore diversity, so that the entropy of meta-actions is as high as possible, so the entropy H(A|Z) of actions under the condition of meta-actions needs to be maximized. In summary, the information theoretic optimization objective proposed by the present invention is expressed as follows:

[0019] R=I(S;Z)+H(A|S)-I(A;Z|S)

[0020] =(H(Z)-H(Z|S))+H(A|S)-(H(A|S)-H(A|S,Z))

[0021] =H(Z)-H(Z|S)+H(A|S,Z)

[0022] The robot performs deep reinforcement learning training by maximizing the above information theory optimization objectives and ultimately obtains N meta-action strategies.

[0023] Furthermore, in step S2, the temporal dynamic warping algorithm is a typical optimization algorithm, which uses a temporal warping function that satisfies certain conditions to describe the temporal correspondence between the test template and the reference template, and solves the warping function corresponding to the minimum cumulative distance when the two templates are matched. First, the state is randomly initialized in the Mujoco simulation environment, and the meta-action strategy obtained in step S1 is used to generate K trajectories T = {τ1, τ2, .., τ K}, thus obtaining the trajectory of each joint The similarity of the trajectories of every two joints is calculated through a time warping algorithm, and hierarchical clustering is performed based on the similarity. The hierarchical clustering algorithm is a relatively intuitive algorithm that can merge and cluster small clusters from the bottom up. Specifically, each joint is first regarded as an independent cluster, and then the two clusters with the shortest similarity are found at each step, and then merged into a large cluster until all samples are merged into an overall cluster, and finally a tree structure is established. The advantage of clustering in this algorithm is that the entire clustering tree is obtained at one time, and the number of clusters can be changed according to the needs of clustering, and there is no need to recalculate the attribution of sample points due to the change in the number of clusters. The present invention can compare the effects of clustering results at different levels on the reinforcement learning algorithm to learn new skills through experiments, and select the clustering result with better overall effect as the final joint grouping, expressed as C m =cluster m =[Joint 1∶j ].

[0024] Furthermore, in step S3, the joint group mask is a 56-dimensional vector containing only 0 and 1. According to the joint clustering result obtained in step S2, each joint group generates a corresponding joint mask Mask m , where the joint position that does not belong to the joint group is 0, and the joint position that enters the joint group is 1. The joint group mask is bitwise multiplied by the meta-action strategy output obtained in step S1 to obtain the meta-action strategy for different joint groups, expressed as

[0025] Furthermore, in step S4, the hierarchical reinforcement learning control network is a linear combination of the joint component action strategies obtained in step S3. The specific expression of the overall control network is as follows:

[0026]

[0027] The present invention uses the joint component action strategy as the weight initialization of the overall control network. Under the constraint of the reward function of the new task, the overall control network is trained to output the appropriate element action coefficient α through reinforcement learning proximal policy optimization algorithm (PPO). i , so that new tasks can be completed faster and better. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of robot joint grouping and meta-action strategy according to an embodiment of the present invention;

[0029] Figure 2 This is a flow chart of intelligent voxel motion learning based on joint grouping strategy of the present invention;

[0030] Figure 3 This is a network structure diagram of the control model for controlling the robot in the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings and examples to clearly and completely describe the concept, specific method and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. Unless otherwise specified, the calculation method of each link in this example is consistent with the general method. It should be noted that this embodiment is implemented based on the technical solution of the invention, but the protection scope of the present invention is not limited to the following embodiments.

[0032] Robot joint grouping and meta-action strategies, such as Figure 1 As shown in the figure, the intelligent voxel motion learning method based on joint grouping strategy is used for robot automatic control. It can control the robot to quickly learn new tasks through the spatiotemporal fusion of joint grouping and meta-motion, thus avoiding spending a lot of time on training from scratch.

[0033] The method of the present invention consists of:

[0034] An information theoretic objective is proposed to optimize the reinforcement learning algorithm to train the robot, so that it can learn a series of diverse meta-action strategies; a dynamic time warping algorithm is proposed to calculate the trajectory similarity between joints, and then the robot joints are grouped based on a hierarchical clustering algorithm; the meta-action control strategies of different joint groups are obtained according to the corresponding masks of the joint groups; a hierarchical robot overall control strategy network is constructed, and the weights of the meta-action strategies of each joint group are linearly combined to achieve meta-action fusion at the temporal and spatial levels, so that new tasks can be learned quickly, avoiding training the reinforcement learning algorithm from scratch and greatly reducing the training time; at the same time, the coordination and coherence of the robot's actions are enhanced, so that better performance can be achieved on new tasks.

[0035] like Figure 2 As shown, the following steps are included:

[0036] S1: Build a robot simulation environment based on OpenAI Gym and Mujoco simulation platform. The robot diagram is as follows Figure 1 As shown. Build a neural network as the meta-action control strategy network π(A|S, Z), the detailed network structure is as follows Figure 3 The network input is the state S, which is the observation image of the environment and the robot itself taken by the monocular camera. The observation of the system from the robot's perspective can be exported through Mujoco's built-in function. The output is the robot's action A, which is a 56-dimensional vector composed of the joint angles and joint angular velocities of the robot, expressed as Joint j＝[1∶56] .Depend on Figure 3As shown in the figure, after the state S of the observed image is input into the control strategy network, it passes through 3 convolutional neural network layers (CNN) with a convolution size of 5, a step size of 2, and an activation function of ReLU, and then passes through 2 fully connected layers (FC) to obtain the intermediate latent variable Z. The input observation state S passes through another convolution layer and is concatenated with the latent variable z, and then passes through another two fully connected layers to finally obtain the robot's action A. In the network, the intermediate latent variable Z is constrained to a uniform distribution, expressed as Z~P(z). Set based on a latent variable z i The policy network is a meta-action, denoted as π i (A|S,z i ), i = 0, 1, ..., N, where N represents the number of meta-actions.

[0037] In addition, this simulation environment does not have a specific reward function, hoping that the robot can learn meta-actions that are irrelevant to the task. This paper designs an information theory optimization objective to constrain the deep reinforcement learning algorithm, which is specifically expressed as follows:

[0038] R=I(S;Z)+H(A|S)-I(A;Z|S)

[0039] =(H(Z)-H(Z|S))+H(A|S)-(H(A|S)-H(A|S,Z))

[0040] =H(Z)-H(Z|S)+H(A|S,Z)

[0041] By maximizing the above information theory optimization objectives, the robot deep reinforcement learning proximal policy optimization algorithm PPO training is carried out, and finally N meta-action strategies are obtained.

[0042] S2: Use the N obtained meta-action strategies to sample trajectories and obtain multiple trajectories for each joint The similarity of the trajectories of every two joints is calculated by the time warping algorithm. Considering that the trajectories of different joints may have hysteresis, it is assumed that the i-th point of one joint trajectory is similar to the j-th point of another joint trajectory. The sum of the distances of all similar points is taken as the regularized path distance, and finally the regularized path distance is used to measure the similarity of the two joint trajectories. Through dynamic programming, the shortest distance between two trajectories can be solved. For the regularized path distance matrix D(i, j), there is

[0043] D(i,j)=Dist(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}

[0044] Wherein, Dist(i, j) represents the distance between the i-th point in the first joint trajectory and the j-th point in the other joint trajectory, and the present invention uses the difference in the rotation angle of the joint to calculate. Using the recursive formula of the dynamic programming algorithm, the regular path distance matrix D(i, j) can be finally solved. The smaller the regular path distance matrix, the greater the similarity of the two joints.

[0045] By calculating the regular path distance matrix between different joints, the similarity relationship between joints is obtained, so as to perform hierarchical clustering on all joints of the robot. First, each joint is regarded as an independent cluster, and then in each clustering round, the two clusters with the largest similarity among all clusters, that is, the smallest regular path distance matrix, are merged into a large cluster until all are merged into one cluster, and a cluster tree structure is established. Based on the experience after multiple experiments, the present invention chooses to retain 5 joint clusters as the classification result of hierarchical clustering.

[0046] S3: Based on the classification results of the above joint groups, five corresponding joint group masks Mas are generated. n k = {M1, M2, M3, M4, M5}. The joint group mask is a 56-dimensional vector containing only 0 and 1. The joint positions that do not belong to the joint group are 0, and the joint positions that belong to the joint group are 1. The joint group mask is bitwise multiplied by the meta-action strategy output A obtained in step S1 to obtain the meta-action strategies for different joint groups. like Figure 3 shown.

[0047] S4: Build a hierarchical reinforcement learning control network, the network structure is a linear combination of the joint component action strategies obtained in step S3. The specific expression of the overall control network is as follows:

[0048]

[0049] The present invention uses the joint component action strategy as the weight initialization of the overall control network. Under the constraint of the reward function of the new task, the overall control network is trained to output the appropriate element action coefficient α through the reinforcement learning proximal policy optimization algorithm PPO (Proximal Policy Optimization). i , and finally learn new tasks based on the combination of multiple joint component action control strategies at the spatiotemporal level. This method avoids training the reinforcement learning algorithm from scratch, saving a lot of time costs, and learns new tasks faster and better by combining meta-actions at the spatiotemporal level and using prior knowledge of actions.

[0050] Those skilled in the art should understand that each module or each step of the above-mentioned application can be implemented with a general network structure. The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An intelligent voxel motion learning method based on joint grouping strategy, characterized in that: Propose an information-theoretic optimization objective to optimize the reinforcement learning algorithm to train the robot to learn a variety of meta-action strategies; A time-based dynamic regularization algorithm is proposed to calculate the trajectory similarity between joints, and then a hierarchical clustering algorithm is used to group robot joints. The meta-action control strategies of different joint groups are obtained according to the corresponding masks of the joint groups; Construct a hierarchical robot overall control strategy network, and achieve meta-action fusion at the temporal and spatial levels by linearly combining the weights of each joint component action strategy, so that new tasks can be learned quickly; at the same time, the coordination and coherence of the robot's actions are enhanced, so that better performance can be achieved on new tasks; The intelligent voxel action learning method based on joint grouping strategy is implemented by the following steps: S1: First, we build a meta-action strategy network, and then propose an information-theoretic optimization objective to replace the reward function that is highly relevant to the task. We conduct deep reinforcement learning training on the robot in a simulation environment, and use the distinguishability of meta-actions to obtain a rich and diverse series of meta-actions. S2: Use the existing meta-action strategy to sample a large number of trajectories, and use the temporal dynamic regularization algorithm to calculate the similarity based on the trajectory of each joint, and then group the joints according to hierarchical clustering to obtain joint groups; S3: Generate the corresponding joint group mask based on the joint group in S2, and combine the meta-action strategy obtained in step S1 to obtain the meta-action strategy network of different joint groups; S4: In the new robot application scenario, a hierarchical reinforcement learning control network is built to learn new tasks faster by combining the joint component actions obtained in step S3, and a reward function can be obtained at the same time; In step S1, the meta-action strategy network is a neural network, represented as ,in The input is the state S, which is specifically an observation image of the environment and the robot itself taken by a monocular camera; The output is the robot's action A, which is a 56-dimensional vector composed of the joint angles and joint angular velocities of the robot, expressed as ; In the network, the intermediate latent variable Z is constrained to a uniform distribution, expressed as ; Setting based on a latent variable The policy network is a meta-action, expressed as ; In step S1, for deep reinforcement learning, an information-theoretic optimization objective is designed to constrain the deep reinforcement learning algorithm; the information-theoretic optimization objective includes three design principles, namely (1) all states must be visited, different states visited by different meta-actions can be distinguished, and the mutual information between meta-actions and states is maximized. ; (2) Use states to distinguish meta-actions and minimize the mutual information between meta-actions and actions under the condition of states ; (3) Maximize the entropy of the action under the condition of the meta-action ; In summary, the proposed information theory optimization objective is expressed as follows: The robot performs deep reinforcement learning training by maximizing the above information theory optimization objectives and ultimately obtains N meta-action strategies.

2. The intelligent voxel motion learning method based on joint grouping strategy as claimed in claim 1, characterized in that: In step S2, the time dynamic warping algorithm uses a time warping function that satisfies certain conditions to describe the time correspondence between the test template and the reference template, and solves the warping function corresponding to the minimum cumulative distance when the two templates are matched; First, randomly initialize the state in the Mujoco simulation environment and use the meta-action strategy obtained in step S1 to generate K trajectories , thus obtaining the trajectory of each joint ; The similarity of the trajectories of every two joints is calculated through the time warping algorithm, and hierarchical clustering is performed based on the similarity; The hierarchical clustering algorithm described above: small clusters are merged and clustered from bottom to top; specifically, each joint is first regarded as an independent cluster, and then the two clusters with the shortest similarity are found at each step, and then merged into a large cluster, until all samples are merged into an overall cluster, and finally a tree structure is established; the clustering result with better overall effect is selected as the final joint grouping, which is expressed as .

3. The intelligent voxel motion learning method based on joint grouping strategy as claimed in claim 1, characterized in that: Furthermore, in step S3, the joint group mask is a 56-dimensional vector containing only 0 and 1; according to the joint clustering result obtained in step S2, each joint group generates a corresponding joint mask , where the joint position that does not belong to the joint group is 0, and the joint position that enters the joint group is 1; The meta-action strategies obtained in step S1 are output by bitwise multiplying the joint group masks, thereby obtaining the meta-action strategies for different joint groups, which are expressed as .

4. The intelligent voxel motion learning method based on joint grouping strategy as claimed in claim 1, characterized in that: Furthermore, in step S4, the hierarchical reinforcement learning control network is a linear combination of the joint component action strategies obtained in step S3; the specific expression of the overall control network is as follows: The joint component action strategy is used as the weight initialization of the overall control network; under the constraint of the reward function of the new task, the overall control network is trained to output appropriate meta-action coefficients through reinforcement learning proximal policy optimization algorithm (PPO) , so that new tasks can be completed faster and better.

Images