An offline teaching and learning system for long-range continuous control tasks of intelligent agents

Abstract

The present invention discloses an offline teaching learning system for a long-range continuous control task of an intelligent body, comprising: a long-range candidate path planner module, which is responsible for generating multiple candidate paths containing multiple sub-goal states according to the initial states of the manipulated object and the intelligent body in the task environment; an optimal path selection module, which is responsible for evaluating the value of the generated multiple candidate paths and selecting the path with the highest value as the optimal path; an optimal action selection module under sub-goal constraints, which is responsible for using the optimal path as a guidance path, using a single sub-goal state on the guidance path as a constraint, generating the action of the intelligent body according to the current manipulated object and intelligent body state to reach a new state, and after multiple state transitions, replacing the sub-goal until all sub-goal states on the guidance path are tracked, and completing the long-range continuous control task of the manipulated object. The present invention is conducive to the intelligent body to use human teaching data for offline learning in a real environment.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent body control, and in particular to an offline teaching and learning system for long-range continuous control tasks of intelligent bodies. Background Art

[0002] Reinforcement learning is already a widely available solution in the field of continuous control of robots. An agent learns by exploring and exploiting the environment. It continuously collects samples and learns how to get rewards based on these samples. Although the agent is able to learn skills through this mechanism, this online method of frequent interaction with the environment is not applicable in many scenarios. First, for some scenarios such as medical skill learning and high-altitude work learning, it is dangerous to let the agent interact with the environment in large quantities and frequently. Secondly, in real scenarios, agents such as robots may collide and wear out during the learning process, which greatly increases the learning cost. In order to solve the dilemma of online reinforcement learning, learning from demonstration data provides a feasible solution.

[0003] Agents can learn offline from pre-collected expert demonstrations, reducing the cost of interacting with the environment. The cost of learning in data is much smaller than the cost of learning in the environment. Some studies have shown that offline learning solutions represented by imitation learning and offline reinforcement learning can be used as prior skill learning methods to accelerate online learning and multi-task learning. Offline learning itself has also shown good performance in many scenarios. However, due to the lack of exploration and utilization in the environment, offline learning relies too much on expert teaching datasets, resulting in a distribution shift between expert datasets and the real environment, which reduces the performance of offline learning methods.

[0004] The distribution offset between the real environment and the teaching data is the main factor affecting the performance of offline reinforcement learning algorithms. In online reinforcement learning, the off-policy reinforcement learning algorithm collects data into the replay buffer by letting the agent explore in the environment, and then updates the strategy based on the data in the replay buffer. As the agent's strategy is updated and the exploration increases, the data in the replay buffer becomes more diverse, and its data distribution gradually becomes similar to the real environment. Generally speaking, off-policy reinforcement learning algorithms can be used for offline learning, as long as the replay buffer data is replaced with teaching data. The teaching data set can come from the data collected in the environment during the training process or testing of the behavior strategy, or from human experts. However, studies have shown that the effect of off-policy algorithms for offline learning is not good. This is due to the lack of error correction opportunities brought by online exploration in the environment. During testing, when the agent faces unexplored states in the environment, it is more likely to produce extrapolation errors and make wrong behaviors.

[0005] For some data sets from human experts, their data often consider the optimal path and come from deterministic strategies, which makes it easy for the agent to fall into the local optimal situation when learning, unlike online learning, which can escape the local optimal situation by exploring in the environment. During testing, when the agent faces an unexplored state in the environment, it will make wrong behaviors and may continue to accumulate errors, leading to the failure of the task. Secondly, in order to efficiently utilize and expand the coverage of the data set, some data does not come from the best experts. Offline reinforcement learning can use this data through Q-learning, but for imitation learning, this will cause the agent to make wrong decisions. Furthermore, the teaching data from human experts is a non-Markov process, containing human subjective behavior, and the assumption based on offline reinforcement learning is that the decision-making process of the agent must conform to the Markov characteristics. Reinforcement learning in such data will produce additional deviations. Summary of the invention

[0006] The purpose of the present invention is to learn a goal-oriented continuous control task of a robotic arm from offline teaching data, alleviate the extrapolation error problem caused by the decline in the exploration ability of offline reinforcement learning, and propose an offline teaching learning system for long-range continuous control tasks of an intelligent agent. Path selection and sub-goal tracking are used to constrain the state space of the environment and the action space of the intelligent agent. The timing information in the teaching data is used to fit the characteristics of human behavior on non-Markov data, thereby improving the success rate of the intelligent agent in performing continuous control tasks in a real environment.

[0007] To achieve the above purpose, the technical solution provided by the present invention is: an offline teaching learning system for long-range continuous control tasks of an intelligent body, comprising:

[0008] The long-range candidate path planner module is responsible for generating multiple candidate paths containing multiple sub-goal states according to the initial states of the manipulated object and the agent in the task environment; wherein the agent manipulates the manipulated object through its robotic arm;

[0009] The optimal path selection module is responsible for evaluating the value of multiple candidate paths generated by the long-range candidate path planner module and selecting the path with the highest value as the optimal path;

[0010] The optimal action selection module under sub-goal constraints is responsible for using the optimal path selected by the optimal path selection module as the guidance path, and the single sub-goal state on the guidance path as a constraint. It generates the action of the agent to reach a new state according to the current manipulated object and agent state. After multiple state transitions, the sub-goal is replaced until all sub-goal states on the guidance path are tracked, thereby completing the long-range continuous control task of the manipulated object.

[0011] Furthermore, the long-range candidate path planner module is a conditional variational autoencoder structure, including an encoder and a decoder, and the details are as follows:

[0012] The encoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network with two hidden layers and one fully connected layer, and two fully connected layer branches, and is only used in the training phase of the long-range candidate path planner module; the decoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network with two hidden layers and one fully connected layer, and one fully connected layer;

[0013] In the training phase of the long-range candidate path planner module, the agent is pre-manipulated to collect multiple teaching trajectories from the environment as training data sets, each teaching trajectory is represented by P i ={s0,a0,r0,s1,a1,r1,s2,a2,r2,...,s n ,a n ,r n}, n = 1, 2, 3, ..., where s n ,a n ,r n Represents the i-th teaching trajectory P i The parameter at the nth moment, s n is the state of the manipulated object and the agent, which is composed of the position and orientation parameters of the manipulated object and the joint angle of the robot arm of the agent. s0 is the initial state of the manipulated object and the agent, s1 is the new state reached by the agent after the initial state s0, s2 is the new state reached by the agent after the state s1, and a n For the agent in state s n The action to be performed next is the next position that each joint of the robot arm of the agent expects to reach, r n The reward obtained by the agent in the environment is measured by the distance between the new state and the expected target state;

[0014] During the training phase of the long-range candidate path planner module, the encoder is initialized with sparsely sampled initial states from the teaching trajectory. and the original value of the subgoal state sequence As input, Represents the tth in the original value of the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, after passing through the multi-layer perceptron and LSTM network, they are output through two fully connected layers respectively. s and σ s ; From μ s is the expectation, σ sA hidden variable z is sampled from a Gaussian distribution with standard deviation, and the decoder uses the hidden variable z and the initial state As input, output the predicted value of the sub-goal state sequence Represents the tth in the sub-goal state sequence prediction value i sub-goal states, i = 1, 2, 3, ..., H, with the mean square error between the original value and the predicted value of the sub-goal state sequence and the Gaussian distribution N (μ s ,σ s ) and the sum of the KL divergence between the standard Gaussian distribution N(0,1) is used as the loss function to train the long-range candidate path planner module, and the loss function is as follows:

[0015]

[0016] Furthermore, the trained long-range candidate path planner module closes its decoder, samples T latent variables z from the standard Gaussian distribution, takes each latent variable z and the initial state s0 as input, and predicts the sub-goal state sequence through its encoder. One latent variable z corresponds to one sub-goal state sequence, and T sub-goal state sequences are used as candidate paths.

[0017] Furthermore, the optimal path selection module includes an action generator and a state-action evaluator, and the specific situation is as follows:

[0018] The action generator is a conditional variational encoder structure, consisting of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers;

[0019] In the training phase of the optimal path selection module, the encoder of the action generator takes the tuple (s, a, r, s') as input, where s is the current state, a is the original value of the action executed in the current state, s' is the new state reached by the agent after executing action a in state s, r is the reward obtained by the agent after executing action a, and the encoder outputs μ h and σ h , from μ h is the expectation, σ h The decoder of the action generator takes the state s and the hidden variable z as input to generate an action prediction value The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ h ,σ h) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function:

[0020]

[0021] In the training phase of the optimal path selection module, m latent variables z are sampled from a standard Gaussian distribution, and the m latent variables z and the state s' are used to generate m actions a through the decoder of the action generator. i , i = 1, 2, ..., m, the state-action evaluator is based on the state-action pair (s', a i ) is used as input to obtain the evaluation score of the state-action pair. The state-action evaluator uses the following formula as the loss function, where Q(s, a) represents the evaluation score of the state-action pair (s, a), represents m pairs (s',a i ), minimize the loss function and update the state-action evaluator weights:

[0022]

[0023] Furthermore, the trained optimal path selection module is used to select each sub-goal state sequence obtained from the long-range candidate path planner module. represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, and the decoder of the action generator of the optimal path selection module selects each Generate an action, then pass the state-action evaluator to calculate the score of the state-action pair, and the score of the state-action pair is regarded as the sub-goal state The score of the sub-goal state sequence S_predict is calculated t The sum of all sub-goal scores is S_predict t The score of the prediction function is as follows: t As the best guidance path.

[0024] Furthermore, the optimal action selection module under the sub-goal constraint is composed of an action generator with sub-goal constraints and a state-action evaluator, and the specific situation is as follows:

[0025] The action generator is a conditional variational encoder structure, consisting of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers;

[0026] In the training phase of the optimal action selection module under the sub-goal constraints, the encoder of the action generator takes a tuple (s, s g ,a,r,s') is the input, where s g is the next sub-goal state constraint expected to be reached in the current state s, a is the original value of the action performed in the current state, s' is the new state reached by the agent after performing action a in state s, r is the reward obtained by the agent after performing action a, and the encoder reads the tuple (s, s g ,a,r,s') and output μ a and σ a , from μ a is the expectation, σ a The hidden variable z is sampled from a Gaussian distribution with standard deviation, and the decoder of the action generator is based on the state s, the sub-goal state constraint s g And hidden variable z as input, generate action prediction value The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ a ,σ a ) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function:

[0027]

[0028] In the training phase of the optimal action selection module under the sub-goal constraint, m latent variables z are sampled from a standard Gaussian distribution, m latent variables z, state s' and sub-goal constraint state s g Generate m actions a through the decoder of the action generator i , i = 1, 2, ..., m, the state-action evaluator is based on the state-action pair (s', a i ) and s g The state-action evaluator uses the following formula as the loss function to obtain the evaluation score of the state-action pair, where Q(s,s g ,a) indicates that in the sub-goal constraint state s g Next, the evaluation score of the state-action pair (s, a) is Indicates that in the sub-goal constraint state s g Next, m pairs (s', a i ), minimize the loss function and update the state-action evaluator weights:

[0029]

[0030] Furthermore, the trained optimal action selection module under sub-goal constraints tracks the sub-goal state sequence on the optimal guidance path. Each sub-goal state in represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, with As the sub-goal constraint, the decoder of the action generator of the optimal action selection module generates an action according to the current state s, and the optimal action a is selected by the state-action evaluator. The agent performs action a in the environment to reach the new state s'. After reaching the new state, the action is generated and selected for the new state. After the state transition reaches k times, the next sub-goal is changed. Continue to generate actions, execute, and transition states for sub-goal constraints until all sub-goal states are tracked;

[0031] After tracking all sub-goal states on the optimal guidance path, if the manipulated object and the agent have not yet reached the desired target state and the total number of state transitions is less than the threshold Then, according to the current state, the optimal guidance path is generated again through the long-range candidate path planner module and the optimal path selection module, and the optimal guidance path is re-tracked until the manipulated object and the intelligent agent reach the desired state or the total number of state transitions reaches the threshold.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] 1. The present invention implements dual constraints on the action space and future state of the intelligent agent by predicting the guidance path of the manipulated object and the current state of the intelligent agent, thereby preventing the intelligent agent from encountering too many states that have not been encountered during training in the real environment and reducing the extrapolation error caused by the distribution offset between the real environment and the teaching data.

[0034] 2. The present invention captures the behavioral characteristics in human teaching data through the LSTM network. Although the intelligent agent generates redundant states in task execution due to learning from human teaching data, the constantly changing sub-goal constraints enable the intelligent agent to recover from the redundant state, reduce the error accumulation in the execution of long-range continuous control tasks, and alleviate the bottleneck of offline reinforcement learning on non-Markov process data.

[0035] 3. Each module of the present invention can be replaced with other networks according to different input data types, which is convenient for combination with other fields such as image processing and lidar detection, and is conducive to migration to scenarios using image information or depth information, and adapting to the needs of actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the system of the present invention.

[0037] Figure 2 This is the structural diagram of the long-range candidate path planner module.

[0038] Figure 3 This is the structural diagram of the optimal path selection module.

[0039] Figure 4 Structural diagram of the optimal action selection module under sub-goal constraints. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0041] This embodiment discloses an offline teaching learning system for long-range continuous control tasks of intelligent agents. Figure 1 As shown, it includes a long-range candidate path planner module, an optimal path selection module and an optimal action selection module under sub-goal constraints. This embodiment adopts the Gym simulation environment and the Mujoco engine, takes the moving ball task in the Adroit task as the task scenario, and the intelligent agent has a 24-degree-of-freedom robotic arm.

[0042] The long-range candidate path planner module is responsible for generating multiple candidate paths containing multiple sub-goal states according to the initial states of the manipulated objects and the agent in the task environment, such as Figure 2 As shown in the figure, the long-range candidate path planner module is a conditional variational autoencoder structure, which includes an encoder and a decoder. The details are as follows:

[0043] The encoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network (long short-term memory network) with two hidden layers and one fully connected layer, and two fully connected layer branches, and is only used in the training phase of the long-range candidate path planner module; the decoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network with two hidden layers and one fully connected layer, and one fully connected layer;

[0044] In the training phase of the long-range candidate path planner module, the agent is pre-manipulated to collect multiple teaching trajectories from the environment as a training data set. In this embodiment, there are 5000 teaching trajectories, each of which is represented by P i ={s0,a0,r0,s1,a1,r1,s2,a2,r2,...,s n ,a n ,r n}, n = 1, 2, 3, ..., where s n ,a n ,rn Represents the i-th teaching trajectory P i The parameter at the nth moment, s n is the state of the manipulated object and the agent, which is composed of the position and orientation parameters of the manipulated object and the joint angle of the robot arm of the agent. s0 is the initial state of the manipulated object and the agent, s1 is the new state reached by the agent after the initial state s0, s2 is the new state reached by the agent after the state s1, and a n For the agent in state s n The action to be performed next is the next position that each joint of the robot arm of the agent expects to reach, r n The reward obtained by the agent in the environment is measured by the distance between the new state and the expected target state;

[0045] During the training phase of the long-range candidate path planner module, the encoder is initialized with sparsely sampled initial states from the teaching trajectory. and the original value of the subgoal state sequence As input, Represents the tth in the original value of the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, after passing through the multi-layer perceptron and LSTM network, they are output through two fully connected layers respectively. s and σ s . From μ s is the expectation, σ s A hidden variable z is sampled from a Gaussian distribution with standard deviation, and the decoder uses the hidden variable z and the initial state As input, output the predicted value of the sub-goal state sequence Represents the tth in the sub-goal state sequence prediction value i sub-goal states, i = 1, 2, 3, ..., H, with the mean square error between the original value and the predicted value of the sub-goal state sequence and the Gaussian distribution N (μ s ,σ s ) and the sum of the KL divergence between the standard Gaussian distribution N(0,1) is used as the loss function to train the long-range candidate path planner module, and the loss function is as follows:

[0046]

[0047] The optimal path selection module is responsible for evaluating the value of multiple candidate paths generated by the long-range candidate path planner module in the long-range candidate path planner module, and selecting a path with the highest value as the optimal path. Figure 3 As shown, the optimal path selection module includes an action generator and a state-action evaluator, and the specific situation is as follows:

[0048] The action generator is a conditional variational encoder structure, which consists of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers;

[0049] In the training phase of the optimal path selection module, the encoder of the action generator takes the tuple (s, a, r, s') as input, where s is the current state, a is the original value of the action executed in the current state, s' is the new state reached by the agent after executing action a in state s, r is the reward obtained by the agent after executing action a, and the encoder outputs μ h and σ h . From μ h is the expectation, σ h The decoder of the action generator takes the state s and the hidden variable z as input to generate an action prediction value The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ h ,σ h ) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function:

[0050]

[0051] In the training phase of the optimal path selection module, m=10 latent variables z are sampled from a standard Gaussian distribution, and the m latent variables z and the state s' are used to generate m actions a through the decoder of the action generator. i , i = 1, 2, ..., m, the state-action evaluator is based on the state-action pair (s', a i ) is used as input to obtain the evaluation score of the state-action pair. The state-action evaluator uses the following formula as the loss function, where Q(s, a) represents the evaluation score of the state-action pair (s, a), represents m pairs (s',a i ), minimize the loss function and update the state-action evaluator weights:

[0052]

[0053] The optimal action selection module under the sub-goal constraint is responsible for taking the optimal path selected by the optimal path selection module as the guiding path, taking the single sub-goal state on the guiding path as the constraint, and generating the action of the agent according to the current manipulated object and agent state to reach a new state. After multiple state transitions, the optimal action selection module changes the sub-goal until all sub-goal states on the guiding path are tracked, such as Figure 4 As shown, the optimal action selection module under the sub-goal constraint is composed of an action generator with sub-goal constraints and a state-action evaluator, and the specific situation is as follows:

[0054] The action generator is a conditional variational encoder structure, which consists of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers;

[0055] In the training phase of the optimal action selection module under the sub-goal constraints, the encoder of the action generator takes a tuple (s, s g ,a,r,s') is the input, where s g is the next sub-goal state constraint expected to be reached in the current state s, a is the original value of the action performed in the current state, s' is the new state reached by the agent after performing action a in state s, r is the reward obtained by the agent after performing action a, and the encoder reads the tuple (s, s g ,a,r,s') and output μ a and σ a . From μ a is the expectation, σ a The hidden variable z is sampled from a Gaussian distribution with standard deviation, and the decoder of the action generator is based on the state s, the sub-goal state constraint s g And hidden variable z as input, generate action prediction value The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ a ,σ a ) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function:

[0056]

[0057] In the training phase of the optimal action selection module under the sub-goal constraint, m=10 latent variables z, m latent variables z, state s' and sub-goal constraint state s are sampled from a standard Gaussian distribution. g Generate m actions a through the decoder of the action generatori , i = 1, 2, ..., m, the state-action evaluator is based on the state-action pair (s', a i ) and s g The state-action evaluator uses the following formula as the loss function to obtain the evaluation score of the state-action pair, where Q(s,s g ,a) indicates that in the sub-goal constraint state s g Next, the evaluation score of the state-action pair (s, a) is Indicates that in the sub-goal constraint state s g Next, m pairs (s', a i ), minimize the loss function and update the state-action evaluator weights:

[0058]

[0059] In the test application, the trained long-range candidate path planner module, optimal path selector and optimal action selection module are integrated to enable the intelligent agent to complete the long-range continuous control task of the manipulated object. The specific situation is as follows:

[0060] For the long-range candidate path planner module, close its decoder, sample T = 10 hidden variables z from the standard Gaussian distribution, use each hidden variable z and the initial state s0 as input to predict the sub-goal state sequence through its encoder, 1 hidden variable z corresponds to 1 sub-goal state sequence, and T sub-goal state sequences are used as candidate paths;

[0061] For the optimal path selection module, for each sub-goal state sequence obtained from the long-range candidate path planner module represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, and the decoder of the action generator of the optimal path selection module selects each Generate an action, then pass the state-action evaluator to calculate the score of the state-action pair, and the score of the state-action pair is regarded as the sub-goal state The score of the sub-goal state sequence S_predict is calculated t The sum of all sub-goal scores is S_predict t The score of the prediction function is as follows: t As the optimal guidance path;

[0062] For the optimal action selection module under the sub-goal constraints, track the sub-goal state sequence on the optimal guidance path Each sub-goal state in represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, with The action is generated by the decoder of the action generator of the optimal action selection module according to the current state s, and the optimal action a is selected by the state-action evaluator. The agent performs action a in the environment to reach the new state s'. After reaching the new state, the action is generated and selected for the new state. After the state transition reaches k = 10 times, the next sub-goal is changed. Continue to generate actions, execute, and transition states for sub-goal constraints until all sub-goal states are tracked;

[0063] After tracking all sub-goal states on the optimal guidance path, if the manipulated object and the agent have not yet reached the desired target state and the total number of state transitions is less than the threshold Then, according to the current state, the optimal guidance path is generated again through the long-range candidate path planner module and the optimal path selection module, and the optimal guidance path is re-tracked until the manipulated object and the intelligent agent reach the desired state or the total number of state transitions reaches the threshold.

[0064] In summary, the present invention proposes an offline teaching and learning system for long-range continuous control tasks of intelligent agents, which can make up for the bottleneck of current offline reinforcement learning methods in the field of robot teaching and learning, reduce the extrapolation error caused by the distribution offset between the real environment and the teaching data, and enable the intelligent agent to use human teaching data of non-Markov processes for offline learning, thereby reducing the cost and loss of online learning. It has broad research and practical application value and is worthy of promotion.

[0065] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An offline teaching learning system for long-range continuous control tasks of intelligent agents, characterized in that: include: The long-range candidate path planner module is responsible for generating multiple candidate paths containing multiple sub-goal states according to the initial states of the manipulated object and the agent in the task environment; wherein the agent manipulates the manipulated object through its robotic arm; The optimal path selection module is responsible for evaluating the value of multiple candidate paths generated by the long-range candidate path planner module and selecting the path with the highest value as the optimal path; The optimal action selection module under sub-goal constraints is responsible for taking the optimal path selected by the optimal path selection module as the guidance path, taking the single sub-goal state on the guidance path as the constraint, generating the action of the agent to reach the new state according to the current manipulated object and agent state, and changing the sub-goal after multiple state transitions until all sub-goal states on the guidance path are tracked, thereby completing the long-range continuous control task of the manipulated object; The long-range candidate path planner module is a conditional variational autoencoder structure, including an encoder and a decoder, as follows: The encoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network with two hidden layers and one fully connected layer, and two fully connected layer branches, and is only used in the training phase of the long-range candidate path planner module; the decoder is composed of a multi-layer perceptron MLP with two fully connected layers, an LSTM network with two hidden layers and one fully connected layer, and one fully connected layer; In the training phase of the long-range candidate path planner module, the agent is pre-manipulated to collect multiple teaching trajectories from the environment as training data sets, each teaching trajectory is represented by P I ={s0,a0,r0,s1,a1,r1,s2,a2,r2,...,s n ,a n ,r n }, where s n ,a n ,r n Indicates the Ith teaching trajectory P I The parameter at the nth moment, s n is the state of the manipulated object and the agent, which is composed of the position and orientation parameters of the manipulated object and the joint angle of the robot arm of the agent. s0 is the initial state of the manipulated object and the agent, s1 is the new state reached by the agent after the initial state s0, s2 is the new state reached by the agent after the state s1, and a n For the agent in state s n The action to be performed next is the next position that each joint of the robot arm of the agent expects to reach, r n The reward obtained by the agent in the environment is measured by the distance between the new state and the expected target state; During the training phase of the long-range candidate path planner module, the encoder is initialized with sparsely sampled initial states from the teaching trajectory. and the original value of the subgoal state sequence As input, Represents the tth in the original value of the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, after passing through the multi-layer perceptron and LSTM network, they are output through two fully connected layers respectively. s and σ s ; From μ s is the expectation, σ s A hidden variable z′ is sampled from a Gaussian distribution with standard deviation, and the decoder uses the hidden variable z′ and the initial state As input, output the predicted value of the sub-goal state sequence Represents the tth in the sub-goal state sequence prediction value i sub-goal states, i = 1, 2, 3, ..., H, with the mean square error between the original value and the predicted value of the sub-goal state sequence and the Gaussian distribution N (μ s ,σ s ) and the sum of the KL divergence between the standard Gaussian distribution N(0,1) is used as the loss function to train the long-range candidate path planner module, and the loss function is as follows: The trained long-range candidate path planner module closes its decoder, samples T latent variables z from the standard Gaussian distribution, takes each latent variable z and the initial state s0 as input, and predicts the sub-goal state sequence through its encoder. One latent variable z corresponds to one sub-goal state sequence, and T sub-goal state sequences are used as candidate paths.

2. According to claim 1, an offline teaching and learning system for long-range continuous control tasks of an intelligent agent is characterized in that: The optimal path selection module includes an action generator and a state-action evaluator, and the specific situation is as follows: The action generator is a conditional variational encoder structure, consisting of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers; In the training phase of the optimal path selection module, the encoder of the action generator takes the tuple (s, a, r, s') as input, where s is the current state, a is the original value of the action executed in the current state, s' is the new state reached by the agent after executing action a in state s, r is the reward obtained by the agent after executing action a, and the encoder outputs μ h and σ h , from μ h is the expectation, σ h The decoder of the action generator takes the state s and the hidden variable z' as input to generate the action prediction value The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ h ,σ h ) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function: In the training phase of the optimal path selection module, m latent variables z are sampled from a standard Gaussian distribution, and the m latent variables z and the state s' are used to generate m actions a through the decoder of the action generator. i′ , i′=1,2,...,m, the state-action evaluator is based on the state-action pair (s',a i′ ) is used as input to obtain the evaluation score of the state-action pair. The state-action evaluator uses the following formula as the loss function, where Q(s, a) represents the evaluation score of the state-action pair (s, a), represents m pairs (s',a i′ ), minimize the loss function and update the state-action evaluator weights:

3. According to claim 2, an offline teaching and learning system for long-range continuous control tasks of an intelligent agent is characterized in that: The trained optimal path selection module, for each sub-goal state sequence obtained from the long-range candidate path planner module represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, and the decoder of the action generator of the optimal path selection module selects each Generate an action, then pass the state-action evaluator to calculate the score of the state-action pair, and the score of the state-action pair is regarded as the sub-goal state The score of the sub-goal state sequence S_predict is calculated t The sum of all sub-goal scores is S_predict t The score of S_predict is selected, and the S_predict with the highest score is selected. t As the best guidance path.

4. According to claim 1, an offline teaching and learning system for long-range continuous control tasks of an intelligent agent is characterized in that: The optimal action selection module under sub-goal constraints is composed of an action generator with sub-goal constraints and a state-action evaluator, as follows: The action generator is a conditional variational encoder structure, consisting of an encoder and a decoder, wherein the encoder consists of an MLP with 4 fully connected layers and two fully connected layer branches, and the decoder is an MLP with 3 fully connected layers; the state-action evaluator consists of an MLP with 3 fully connected layers; In the training phase of the optimal action selection module under the sub-goal constraints, the encoder of the action generator takes a tuple (s, s g ,a,r,s') is the input, where s g is the next sub-goal state constraint expected to be reached in the current state s, a is the original value of the action performed in the current state, s' is the new state reached by the agent after performing action a in state s, r is the reward obtained by the agent after performing action a, and the encoder reads the tuple (s, s g ,a,r,s') and output μ a and σ a , from μ a is the expectation, σ a The hidden variable z′ is sampled from a Gaussian distribution with standard deviation, and the decoder of the action generator is based on the state s, the sub-goal state constraint s g And hidden variables z′ as input to generate action prediction values The action generator uses the following formula as the loss function: the original action value a and the predicted action value The mean square error between the Gaussian distribution N(μ a ,σ a ) and the sum of the KL divergences between the standard Gaussian distribution N(0,1), and the action generator weights are updated by minimizing the loss function: In the training phase of the optimal action selection module under the sub-goal constraint, m latent variables z are sampled from a standard Gaussian distribution, m latent variables z, state s' and sub-goal constraint state s g Generate m actions a through the decoder of the action generator i′ , i′=1,2,...,m, the state-action evaluator is based on the state-action pair (s',a i' ) and s g To obtain the evaluation score for the state-action pair, the state-action evaluator uses the following formula as the loss function, where Q(s,s g ,a) indicates that in the sub-goal constraint state s g Next, the evaluation score of the state-action pair (s, a) is Indicates that in the sub-goal constraint state s g Next, m pairs (s', a i′ ), minimize the loss function and update the state-action evaluator weights:

5. An offline teaching and learning system for long-range continuous control tasks of an intelligent agent according to claim 4, characterized in that: The optimal action selection module under the trained sub-goal constraints tracks the sub-goal state sequence on the optimal guidance path Each sub-goal state in represents the tth in the sub-goal state sequence i sub-goal states, i = 1, 2, 3, ..., H, with The action is generated by the decoder of the action generator of the optimal action selection module according to the current state s, and the optimal action is selected by the state-action evaluator. The agent performs actions in the environment to reach a new state s'. After reaching the new state, the action is generated and selected for the new state. After the state transition reaches k times, the next sub-goal is changed. Continue to generate actions, execute, and transition states for sub-goal constraints until all sub-goal states are tracked; After tracking all sub-goal states on the optimal guidance path, if the manipulated object and the agent have not yet reached the desired target state and the total number of state transitions is less than the threshold Then, according to the current state, the optimal guidance path is generated again through the long-range candidate path planner module and the optimal path selection module, and the optimal guidance path is re-tracked until the manipulated object and the intelligent agent reach the desired state or the total number of state transitions reaches the threshold.

Images