Robot task planning method and control system based on sub-target reasoning

Through the variational autoencoder generation and optimization of sub-target sequences, combined with the data update of the post-hoc sampler, the problem that a large amount of manual intervention is solved by the robot performing long-term tasks, and the autonomous decomposition and efficient completion of robot tasks are achieved.

CN118952204APending Publication Date: 2024-11-15NINGBO DIGITAL TWIN (EASTERN UNIV OF TECH) RES INST +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411197111.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Robots require a lot of manual intervention or pre-planning of all sub-objectives when performing long-term tasks, resulting in inefficiency and difficulty in dealing with a wide range of combination decision-making problems.

Method used

Using a robot task planning method based on a variational autoencoder, multiple rounds of candidate sub-target sequences are generated through the current state of the real environment, and sub-target sequences are selected using the state value score, and data is collected from the real environment and updated by combining the post-sampler. Autoencoder parameters, automatically generate sub-objectives in reinforcement learning.

Benefits of technology

It realizes the autonomous decomposition and execution of robot tasks, improves the efficiency and quality of task completion, reduces manual intervention, and can independently deal with a wide range of combined decision-making problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118952204A_ABST
    Figure CN118952204A_ABST
Patent Text Reader

Abstract

The invention relates to a robot task planning method based on sub-target reasoning and a control system. The method comprises the following steps: by using a variational auto-encoder, taking the current state of a real environment as input information, and taking the distance between a minimized reconstructed final target and an original final target as a target, generating a plurality of rounds of candidate sub-target sequences to obtain a plurality of groups of candidate sub-target sequences; obtaining a state value score of each group of candidate sub-target sequences, selecting the sub-target sequences based on the state value scores, and providing the selected sub-target sequences to the reinforcement learning module to guide the reinforcement learning module to realize a low-level strategy in a real environment; using a post sampler to collect conversion information from a real environment as an offline data set; and sampling a training sample from the offline data set, and updating the parameters of the variational auto-encoder by using the training sample. According to the invention, reliable and automatic generation of the sub-target of reinforcement learning is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of robot control technology, and in particular to a robot task planning method and control system based on sub-goal reasoning. Background Art

[0002] Long-term tasks are an important research topic in the field of robotics. Long-term tasks usually consist of many simpler but sequential subtasks, and their successful completion requires long-term planning and reasoning skills. In daily human activities, we often perform long-term tasks, which are characterized by a hierarchical structure and contain multiple goals. Humans can perform these tasks relatively easily, mainly because we understand the sub-goal dependency structure inherent in the overall task. However, for robots, performing long-term tasks faces many challenges.

[0003] In traditional reinforcement learning methods, robots face challenges in performing long-term tasks due to inefficient exploration and sparse rewards. Current efforts to solve long-term tasks mainly include imitating expert trajectories, improving goal-oriented exploration, and adopting divide-and-conquer strategies. For example, imitation learning can provide practical guidance for robots by decomposing complex long-term tasks into smaller ones. However, it is usually plagued by time-consuming demonstration collection and non-optimal expert strategies.

[0004] Although improving the exploration efficiency of robots is beneficial for discovering effective and feasible sub-goals, the exploration of goals is still limited to neighboring states close to the explored area. Due to the lack of experience in the rarely explored state space, there is still a huge efficiency gap for broader reinforcement learning applications. In addition, some studies focus on decomposing complex tasks into simple tasks to solve long-term tasks. However, these methods require explicitly specifying the number of sub-goals or decision layers, which means that the sub-goals of the entire task must be planned before executing the entire task. Therefore, how to develop robots that can solve a wide range of combinatorial decision-making problems without requiring a lot of human intervention or pre-planning of all sub-goals remains a long-term challenge. Summary of the invention

[0005] The robot task planning method and control system based on sub-goal reasoning provided by the embodiments of the present invention at least solve the problem faced by robots in performing long-term tasks that require a lot of manual intervention or pre-planning of all sub-goals.

[0006] A robot task planning method based on sub-goal reasoning, comprising:

[0007] Using a variational autoencoder, the current state of the real environment is used as input information, and the distance between the reconstructed final goal and the original final goal is minimized. Multiple rounds of candidate sub-goal sequences are generated to obtain multiple groups of candidate sub-goal sequences.

[0008] Obtaining a state value score for each group of the candidate sub-goal sequences, selecting a sub-goal sequence based on the state value score, and providing the selected sub-goal sequence to a reinforcement learning module to guide the reinforcement learning module to implement a low-level strategy in a real environment;

[0009] A post-hoc sampler is used to collect transition information from a real environment as an offline dataset, wherein the transition information includes information of the initial state, action, reward, new state, and goal;

[0010] Training samples are sampled from the offline data set, and the parameters of the variational autoencoder are updated using the training samples.

[0011] In some of the embodiments, the current state includes an initial state and a new state, wherein the new state is the new state of the real environment after the reinforcement learning module implements a low-level action strategy in the real environment under the guidance of a sub-goal.

[0012] In some embodiments, the variational autoencoder includes an encoder based on an explicit strategy and a decoder based on an implicit strategy; using the variational autoencoder to take the current state of the real environment as input information and to minimize the distance between the reconstructed final goal and the original final goal as the goal, generating each round of candidate sub-goal sequence includes:

[0013] Encoder processing status and the final goal g, obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution; randomly sample sub-goals from the probability distribution determined based on the parameters And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal

[0014] The optimized sub-goal Provided to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in the real environment and obtain the new state of the real environment from the reinforcement learning module Where 1≤i≤n, is the initial state.

[0015] In some embodiments, sampling training samples from the offline data set, and updating the parameters of the variational autoencoder using the training samples includes:

[0016] The parameters of the variational autoencoder are updated by using the stochastic gradient descent method to optimize the loss of the sub-target generated by the variational autoencoder and the loss of the ex post sampler, wherein the loss of the sub-target generated by the variational autoencoder is the distance between the reconstructed final target and the original final target, and the loss of the ex post sampler represents the probability of downsampling the sub-target corresponding to the several state actions based on the probability distribution of the sub-target determined by the parameters output by the encoder.

[0017] A robot control system, comprising: a reinforcement learning module and a sub-goal generation module, wherein the sub-goal generation module comprises:

[0018] A variational autoencoder is used to generate multiple rounds of candidate sub-goal sequences with the current state of the real environment as input information and the goal of minimizing the distance between the reconstructed final goal and the original final goal, thereby obtaining multiple groups of candidate sub-goal sequences;

[0019] A value selector, configured to obtain a state value score of each group of the candidate sub-goal sequences, select a sub-goal sequence based on the state value score, and provide the selected sub-goal sequence to a reinforcement learning module to guide the reinforcement learning module to implement a low-level strategy in a real environment;

[0020] A post-event sampler for collecting transition information from a real environment as an offline dataset, wherein the transition information includes information of the initial state, action, reward, new state, and goal;

[0021] The variational autoencoder is also used to sample training samples from the offline data set and use the training samples to update the parameters of the variational autoencoder.

[0022] In some of the embodiments, the variational autoencoder includes an encoder based on an explicit strategy and a decoder based on an implicit strategy;

[0023] The encoder, used to process the state and the final goal g, obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution;

[0024] The decoder is used to randomly sample subgoals from a probability distribution determined based on the parameter And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal

[0025] An electronic device includes: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0026] A non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0027] The robot task planning method and control system based on sub-goal reasoning provided by the embodiments of the present invention use a variational autoencoder to take the current state of the real environment as input information, and to minimize the distance between the reconstructed final goal and the original final goal as the goal, to perform multiple rounds of candidate sub-goal sequence generation to obtain multiple groups of candidate sub-goal sequences; obtain the state value score of each group of candidate sub-goal sequences, select the sub-goal sequence based on the state value score, and provide the selected sub-goal sequence to a reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in the real environment; use a post-sampler to collect conversion information from the real environment as an offline data set, the conversion information includes information about the initial state, action, reward, new state and goal; sample training samples from the offline data set, and use the training samples to update the parameters of the variational autoencoder, thereby solving the problem of requiring a large amount of manual intervention or pre-planning of all sub-goals faced by robots in performing long-term tasks, and realizing reliable and automatic generation of sub-goals of reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of a robot task planning method based on sub-goal reasoning according to an embodiment of the present invention.

[0030] Figure 2 It is a schematic diagram of a robot task planning method based on sub-goal reasoning according to an embodiment of the present invention.

[0031] Figure 3 It is a schematic diagram of the process of updating the variational autoencoder according to an embodiment of the present invention.

[0032] Figure 4 It is a structural block diagram of a robot control system according to an embodiment of the present invention.

[0033] Figure 5 It is a structural schematic diagram of the electronic device created by the present invention. DETAILED DESCRIPTION

[0034] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0035] Among the related technologies, the main technologies for robots to perform long-term tasks include imitation learning, goal-oriented exploration, and divide-and-conquer strategies.

[0036] 1. Imitation learning: Imitation learning is a method of learning tasks by imitating the behavior trajectories of experts. It can decompose complex long-term tasks into smaller subtasks and provide practical guidance for robots. However, imitation learning usually takes a lot of time to collect demonstrations and is often limited by the degree of optimization of expert strategies, and cannot guarantee that the generated strategy is optimal.

[0037] 2. Goal-oriented exploration: Goal-oriented exploration strategies focus on how to improve the robot's exploration efficiency in order to discover effective and feasible sub-goals. However, the exploration scope of this strategy is usually limited to neighboring states close to the explored area, and it lacks experience in the state space that is rarely explored, and cannot cope with a wider range of task scenarios.

[0038] 3. Divide and conquer strategy: This strategy attempts to solve long-term tasks by decomposing complex tasks into simple tasks. However, these methods usually require explicit specification of the number of sub-goals or decision layers, that is, the sub-goals of the entire task must be planned before executing the entire task, which is not ideal in practice.

[0039] The invention aims to propose a method that can autonomously decompose long-term tasks and generate effective sub-goals, thereby improving the efficiency and quality of the robot's task completion. To this end, the invention provides a robot task planning method based on sub-goal reasoning, also known as a variational autoencoder-based sub-goal inference algorithm (VAESI), to generate sub-goals for guiding reinforcement learning to achieve low-level strategies. Figure 1 Flowchart of a robot task planning method based on sub-goal reasoning according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0040] Step S101, using a variational autoencoder with the current state of the real environment as input information, with the goal of minimizing the distance between the reconstructed final goal and the original final goal, to generate multiple rounds of candidate sub-goal sequences to obtain multiple groups of candidate sub-goal sequences.

[0041] Step S102, obtaining the state value score of each group of candidate sub-goal sequences, selecting a sub-goal sequence based on the state value score, and providing the selected sub-goal sequence to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in a real environment.

[0042] Step S103, using a post-sampler to collect conversion information from the real environment as an offline data set, the conversion information includes information of the initial state, action, reward, new state and target.

[0043] Step S104: sampling training samples from the offline data set, and using the training samples to update the parameters of the variational autoencoder.

[0044] In step S101, a sub-goal generator based on a variational autoencoder is built, where the encoder is an explicit strategy and the decoder is an implicit strategy. By giving an initial state and a final goal, a series of sub-goals are generated to guide the robot to reach the final goal.

[0045] The variational autoencoder consists of an encoder and a decoder. The role of the encoder is to compress the current state of a complex environment or into a low-dimensional vector representation (also called latent representation or embedding). This vector should contain important information about the original state so that the decoder can generate meaningful sub-goals based on it. The encoder can be a neural network model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). In the embodiment of the invention, a fully connected neural network is used as an encoder.

[0046] The function of the decoder of the variational autoencoder in the embodiment of the invention is to use the probability distribution obtained after reparameterizing the parameters of the encoder, randomly sample the sub-goals, and optimize the generated sub-goals based on the sub-goals obtained by random sampling and the current state, with the goal of minimizing the difference (distance) between the reconstructed final goal and the original minimum goal. In the embodiment of the invention, the decoder can also be a neural network model, such as a recurrent neural network (RNN) or a Transformer decoder.

[0047] Figure 2 Schematic diagram of a robot task planning method based on sub-goal reasoning according to an embodiment of the present invention. Figure 2The variational autoencoder includes an encoder based on an explicit strategy and a decoder based on an implicit strategy; the variational autoencoder uses the current state of the real environment as input information, with the goal of minimizing the distance between the reconstructed final goal and the original final goal, to generate each round of candidate sub-goal sequences, including: the encoder processes the state and the final goal g, obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution; randomly sample sub-goals from the probability distribution determined based on the parameters And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal

[0048] New state for next subgoal generation It is the new state of the real environment after the reinforcement learning module implements the low-level action strategy in the real environment under the guidance of the sub-goal, where 1≤i≤n, is the initial state.

[0049] The optimized sub-goal Provided to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in the real environment and obtain the new state of the real environment from the reinforcement learning module Specifically, after the variational autoencoder generates a sub-goal, the generated sub-goal is provided to the reinforcement learning module, which will implement a low-level action strategy in the real environment based on the guidance of the sub-goal within a predetermined time step T. If the sub-goal has been reached within time step T, the reinforcement learning module obtains the state when the sub-goal is reached from the real environment The state is fed back to the variational autoencoder to generate the next sub-goal. If the sub-goal is not reached within time step T, the reinforcement learning module obtains the state at time step T from the real environment. And feed the state back to the variational autoencoder to generate the next sub-goal.

[0050] In some embodiments, the loss of the sub-goal generated by the variational autoencoder is based on the final goal g reconstructed by the decoder ′ The expected distance from the final target g is determined. For example, the loss of the variational autoencoder to generate the sub-target adopts the loss function express.

[0051] Step S101 specifically includes the following steps:

[0052] Step 1: First, we take the initial state s0 and the final goal g as input and use the encoder ψ to convert this information into mean and standard deviation. This process involves two fully connected networks, which preprocess the state and final goal and then generate the mean and standard deviation. The purpose of this conversion is to convert the state and goal information into a probability distribution from which sub-goals can be sampled.

[0053] Step 2: Next, use these means and standard deviations to sample subgoals from a normal distribution In this process, we use a trick called reparameterization. This allows us to sample subgoals from a normal distribution defined by mean and standard deviation without sacrificing the differentiability of the model.

[0054] Encoder reparameterization usually refers to the process of transforming or optimizing the parameter representation of the encoder model to improve the model's learning effect, stability, or adapt to new task requirements. In variational autoencoders (VAEs), reparameterization techniques are used to deal with the uncertainty of latent variables. In VAEs, the encoder outputs the parameters of a probability distribution instead of directly outputting the value of the latent variable. Through the reparameterization technique, we can separate the random sampling process from the encoder parameters, allowing the model to effectively propagate gradients and learn. This process usually involves re-representing the encoder output using a specific probability distribution (such as a normal distribution), and then randomly sampling from this distribution to obtain the value of the latent variable.

[0055] Step 3: Finally, we use the decoder ψ ′ Accept sub-goals As input, and conditional on the current state s0 to generate the reconstructed final target g ′ The goal of the decoder is to reconstruct the final goal as accurately as possible using the subgoals and the current state information. In this way, we can ensure that the generated subgoals can help achieve the final goal. The loss function in the subgoal generation process is: We hope that the expected distance (E) between the reconstructed final target and the true final target is as small as possible.

[0056] Continue to refer Figure 2 In order to obtain a better sub-goal sequence result, in step S102, the variational autoencoder performs K rounds of candidate sub-goal sequence generation process to generate K candidate sub-goal sequences in total. In step S102, the candidate sub-goal sequence with the highest state value score is used as the selected sub-goal sequence.

[0057] Through the above steps, these sub-goal candidates are evaluated and ranked using the state value function. The state value function can estimate the expected return of reaching a given goal in a given state. In this way, the value evaluation of each candidate sub-goal sequence can be obtained. In the above steps, the sub-goal with the highest state value is selected as the optimal sub-goal. This means that this sub-goal is most likely to help us reach the final goal. After selecting the optimal sub-goal, it can be used to guide the low-level action strategy of the reinforcement learning model.

[0058] After obtaining the selected sub-goal sequence in step S102, the sub-goal sequence is provided to the reinforcement learning module to guide the reinforcement learning module to implement low-level action strategies in the real environment, control the robot's actions, and thus achieve changes in the real environment state. In the embodiments of the present invention, the reinforcement learning model is not limited to a specific algorithm model. For example, a soft actor-critic (SAC) algorithm can be used, and other reinforcement learning algorithms, such as a deep Q network (DQN), etc., can also be used to implement low-level strategies.

[0059] In order to ensure that the generated sub-goals are feasible, the sub-goal sequence obtained through the above steps S101 to S102 has further optimization space. To this end, in the embodiment created by the present invention, a technique called post-experience replay is introduced to optimize the parameters of the above variational autoencoder by sampling from valid sub-goal data. These valid sub-goal data They are subgoals extracted from valid trajectories that have actually been executed, so they are highly feasible.

[0060] When the robot is performing a task, the real environment feeds back the current state and the feedback (reward) of performing a certain action to the reinforcement learning model, and the reinforcement learning model generates actions based on this information for the robot to interact with the real environment. By repeating this process, we can collect state-action conversion data into the experience replay buffer. These conversion data include information about the initial state, action, reward, new state, and goal.

[0061] Among them, Experience Replay Buffer is an important technology in deep reinforcement learning, which is used to improve the efficiency and stability of the learning process. In the traditional online learning process, the agent usually immediately uses the experience just obtained (including state, action, reward and next state) to update its strategy or value function. However, this strategy of continuously relying on the latest experience is called "on-policy" and may cause the data to be too correlated, thereby reducing the efficiency and stability of learning. The way the experience replay buffer solves this problem is to store all the experience (or part of the experience) collected by the agent in the past period of time, instead of relying solely on the latest experience. This buffer is like a data structure (such as a queue or a heap) that can store a large number of experience samples. During the learning process, the agent will randomly extract some past experience samples from the replay buffer (this is called "off-policy" learning) instead of using only the latest experience.

[0062] Figure 3 Schematic diagram of the process of updating the variational self-encoder according to the embodiment of the present invention. Figure 3 In the process of updating the parameters of the variational autoencoder in step S104, the invention creates an embodiment using the maximum likelihood estimation method (Maximum Likelihood Estimation, referred to as MLE) to maximize the log-likelihood of the effective sub-goals in the offline data set under the unknown sub-goal generation strategy. The goal of this process is to make the sub-goals generated by the model as close as possible to the effective sub-goal distribution, and the degree of difference between the two can be measured by the divergence (KL). The loss function of the post-sampler is: Right now The probability of sampling a sub-goal under the probability distribution determined by the encoder parameters of the variational autoencoder, we expect the probability to be as large as possible.

[0063] In step S104, the stochastic gradient descent method is used to update the parameters of the variational autoencoder to optimize the loss of the variational autoencoder generation sub-goal and the loss of the post-sampler Among them, the loss of the sub-goal generated by the variational autoencoder is the distance between the reconstructed final goal and the original final goal, and the loss of the post-sampler represents the probability of downsampling the sub-goal corresponding to several state actions based on the probability distribution of the sub-goal determined by the parameters output by the encoder.

[0064] The above-mentioned sub-goal generation method (VAESI) adopts a divide-and-conquer strategy to decompose long-term tasks by generating sub-goals, and improves the quality and effectiveness of generated sub-goals through post-samplers and value selectors. The design inspiration of VAESI comes from humans: humans can not only infer sub-goals to decompose long-term tasks, but also infer the final results given sub-goals. VAESI includes an explicit model for generating sub-goals and an implicit model for improving the quality of generated sub-goals. This method of combining explicit models and implicit models enables VAESI to autonomously decompose and execute long-term tasks without a lot of manual intervention. The invention combines explicit models and implicit models to autonomously generate and optimize sub-goals, effectively complete long-term tasks, especially the ability to autonomously decompose and execute long-term tasks, improve the efficiency and performance of task completion, and provide a new framework for solving a wide range of combinatorial decision-making problems, which helps to promote the development of robotics technology.

[0065] The present invention also provides a robot control system. Figure 4 : is a block diagram of a robot control system according to an embodiment of the present invention, such as Figure 4 As shown, the system includes: a reinforcement learning module 41 and a sub-goal generation module 42, wherein the sub-goal generation module 42 includes:

[0066] The variational autoencoder 421 is used to take the current state of the real environment as input information, minimize the distance between the reconstructed final goal and the original final goal, perform multiple rounds of candidate sub-goal sequence generation, and obtain multiple groups of candidate sub-goal sequences.

[0067] The value selector 422 is used to obtain the state value score of each group of candidate sub-goal sequences, select the sub-goal sequence based on the state value score, and provide the selected sub-goal sequence to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in a real environment.

[0068] The post-event sampler 423 is used to collect transition information from the real environment as an offline data set, where the transition information includes information of the initial state, action, reward, new state, and target.

[0069] The variational autoencoder 421 is also used to sample training samples from the offline data set and use the training samples to update the parameters of the variational autoencoder.

[0070] In some of the embodiments, the variational autoencoder 421 includes an encoder based on an explicit strategy and a decoder based on an implicit strategy.

[0071] Encoder, used to process state and the final goal g, and obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution.

[0072] A decoder that randomly samples subgoals from a probability distribution based on this parameter And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal

[0073] The variational autoencoder 421 is also used to transform the optimized sub-goal Provided to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in the real environment and obtain the new state of the real environment from the reinforcement learning module Where 1≤i≤n, is the initial state.

[0074] In some of the embodiments, the current state includes an initial state and a new state, and the new state is the new state of the real environment after the reinforcement learning module implements the low-level action strategy in the real environment under the guidance of the sub-goal.

[0075] In some embodiments, the loss of the sub-goal generated by the variational autoencoder is based on the final goal g reconstructed by the decoder ′ The desired determination of the distance to the final target g.

[0076] In some of the embodiments, the variational autoencoder performs K rounds of candidate sub-goal sequence generation process; the value selector 422 is used to: obtain a set of candidate sub-goal sequences, the set of candidate sub-goal sequences includes K candidate sub-goal sequences; obtain a state value score for each candidate sub-goal sequence in the K candidate sub-goal sequences, and select the candidate sub-goal sequence with the highest state value score as the selected sub-goal sequence.

[0077] In some of the embodiments, the variational autoencoder 421 is used to update the parameters of the variational autoencoder using the stochastic gradient descent method to optimize the loss of the variational autoencoder generating the sub-target and the loss of the post-sampler, wherein the loss of the variational autoencoder generating the sub-target is the distance between the reconstructed final target and the original final target, and the loss of the post-sampler represents the probability of downsampling the sub-target corresponding to a number of state actions based on the probability distribution of the sub-target determined by the parameters output by the encoder.

[0078] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0079] The embodiment of the invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the invention when executed by the at least one processor.

[0080] refer to Figure 5 , the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the invention will now be described, which is an example of the hardware device that can be applied to various aspects of the invention. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the invention described and / or required herein.

[0081] like Figure 5 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0082] Multiple components in the electronic device are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device, and the input unit 506 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0083] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM 502 and / or a communication unit 509. In some embodiments, the computing unit 501 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0084] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of the present invention, the machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable signal medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0087] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0088] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0089] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A robot task planning method based on sub-goal reasoning, Features include: Using a variational autoencoder, the current state of the real environment is used as input information, and the distance between the reconstructed final goal and the original final goal is minimized. Multiple rounds of candidate sub-goal sequences are generated to obtain multiple groups of candidate sub-goal sequences. Obtaining a state value score for each group of the candidate sub-goal sequences, selecting a sub-goal sequence based on the state value score, and providing the selected sub-goal sequence to a reinforcement learning module to guide the reinforcement learning module to implement a low-level strategy in a real environment; A post-hoc sampler is used to collect transition information from a real environment as an offline dataset, wherein the transition information includes information of the initial state, action, reward, new state, and goal; Training samples are sampled from the offline data set, and the parameters of the variational autoencoder are updated using the training samples.

2. The method according to claim 1, It is characterized in that The current state includes an initial state and a new state, wherein the new state is a new state of the real environment after the reinforcement learning module implements a low-level action strategy in the real environment under the guidance of a sub-goal.

3. The method according to claim 1, It is characterized in that The variational autoencoder includes an encoder based on an explicit strategy and a decoder based on an implicit strategy; using the variational autoencoder to take the current state of the real environment as input information and to minimize the distance between the reconstructed final goal and the original final goal as the goal, generating each round of candidate sub-goal sequence includes: Encoder processing status and the final goal g, obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution; randomly sample sub-goals from the probability distribution determined based on the parameters And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal The optimized sub-goal Provided to the reinforcement learning module to guide the reinforcement learning module to implement low-level strategies in the real environment and obtain the new state of the real environment from the reinforcement learning module Where 1≤i≤n, is the initial state.

4. The method according to claim 2, It is characterized in that Sampling training samples from the offline data set, and using the training samples to update the parameters of the variational autoencoder includes: The parameters of the variational autoencoder are updated by using the stochastic gradient descent method to optimize the loss of the sub-target generated by the variational autoencoder and the loss of the ex post sampler, wherein the loss of the sub-target generated by the variational autoencoder is the distance between the reconstructed final target and the original final target, and the loss of the ex post sampler represents the probability of downsampling the sub-target corresponding to the several state actions based on the probability distribution of the sub-target determined by the parameters output by the encoder.

5. A robot control system, Features include: A reinforcement learning module and a sub-goal generation module, wherein the sub-goal generation module includes: A variational autoencoder is used to generate multiple rounds of candidate sub-goal sequences with the current state of the real environment as input information and the goal of minimizing the distance between the reconstructed final goal and the original final goal, thereby obtaining multiple groups of candidate sub-goal sequences; A value selector, configured to obtain a state value score of each group of the candidate sub-goal sequences, select a sub-goal sequence based on the state value score, and provide the selected sub-goal sequence to a reinforcement learning module to guide the reinforcement learning module to implement a low-level strategy in a real environment; A post-event sampler for collecting transition information from a real environment as an offline dataset, wherein the transition information includes information of the initial state, action, reward, new state, and goal; The variational autoencoder is also used to sample training samples from the offline data set and use the training samples to update the parameters of the variational autoencoder.

6. The system according to claim 5, It is characterized in that The variational autoencoder includes an encoder based on an explicit strategy and a decoder based on an implicit strategy; The encoder, used to process the state and the final goal g, obtain the parameters used to represent the probability distribution of the sub-goals, which include the mean and variance of the probability distribution; The decoder is used to randomly sample subgoals from a probability distribution determined based on the parameter And the randomly sampled sub-goals As the input information of the decoder, the state As a condition, minimize the final goal of reconstruction g ′ The distance between the original final goal g is the goal to optimize the sub-goal

Citation Information

Cited By

  • Data generation method for robot control model and related equipment

    CN121132627A