A reinforcement learning system for multi-task long-range decision making of a robotic arm

Abstract

The present disclosure provides a reinforcement learning system for multi-task long-range decision of a mechanical arm, comprising: a hybrid world model module, taking a visual observation at a current time, an executed action and a first classification task variable of a current task as input, obtaining a multi-modal distribution of the spatial appearance of the input in a hidden space by using Gaussian mixture variables, and outputting a reconstructed image; a predictive experience replay module, taking a second classification task variable as input, performing trajectory replay by using a generated model copy, a world model copy and an action model copy of the last task, training the generator of the current task and the mechanical arm by using the replayed trajectory and the current trajectory, and determining the trained mechanical arm and agent. Through the present disclosure, the mechanical arm is controlled to realize efficient memory data replay, overcome the catastrophic forgetting of the world model, and make flexible decisions when facing multiple tasks in the multi-task long-range decision scene of the mechanical arm, thereby improving the autonomy and adaptability of the mechanical arm.

Description

Technical Field

[0001] This disclosure relates to the fields of computer image processing and machine learning technology, and more specifically, to a reinforcement learning system for long-range decision-making in multi-task robotic arms. Background Technology

[0002] Robotic arm control can be used to accomplish various complex tasks, such as object grasping, assembly, or manipulation. Traditional robotic arm control methods are mainly based on predefined rules and algorithms, which are difficult to handle the challenges of multi-task long-range decision-making.

[0003] In recent years, reinforcement learning has become an effective method for solving multi-task long-range decision-making problems and has been gradually applied to the field of robotic arm control. Multi-task long-range decision reinforcement learning aims to control robotic arms to learn to make a series of decisions in complex environments to complete a series of tasks, rather than just a single task.

[0004] In multi-task long-range decision reinforcement learning, robotic arms learn how to make decisions among different tasks through interaction with the environment. By designing appropriate reward functions and state representations, robotic arms can learn to select suitable actions based on the current environmental state and the target task to achieve longer-term goals.

[0005] Multi-task long-range decision reinforcement learning offers several advantages in robotic arm control. First, it enables the robotic arm to make flexible decisions when faced with multiple tasks, improving its autonomy and adaptability. Second, by learning long-term decisions, the robotic arm can make optimal trade-offs among multiple tasks, achieving more efficient operation. Furthermore, multi-task long-range decision reinforcement learning reduces reliance on domain expert knowledge, allowing the robotic arm to autonomously learn and improve its control strategies.

[0006] Multi-task long-range decision reinforcement learning brings new opportunities to robotic arm control. By teaching robotic arms to make long-term decisions in multi-task environments, their intelligence and adaptability can be improved, driving the development of robotic arm control technology. This approach offers greater potential for the application of robotic arms in various fields, enabling them to cope with more complex and diverse task requirements. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this disclosure is to provide a reinforcement learning system for long-range decision-making in multi-task robotic arms.

[0008] To achieve the above objectives, according to one aspect of this disclosure, a reinforcement learning system for long-range multi-task decision-making in robotic arms is provided, comprising:

[0009] The hybrid world model module takes the current visual observation, the action performed, and the first-class task variable of the current task as input, and uses Gaussian mixture variables to obtain the latent space visual dynamics and the multimodal distribution of the spatial appearance in the input / output observation space, and outputs the reconstructed image.

[0010] The predictive experience replay module takes the second-class task variables as input, uses copies of the previous task's generation model, world model, and action model to replay the trajectory, and uses the replayed trajectory and the current trajectory to train the generator and robotic arm for the current task, thus determining the trained generator and robotic arm.

[0011] Optionally, the hybrid world model module includes:

[0012] The representation module takes the current visual observation and the executed action as input and outputs the first hidden state;

[0013] The transition module takes the first classification task variable as input and outputs the predicted second hidden state.

[0014] The observation module takes the first classification task variable and the first hidden state as input and outputs the reconstructed image.

[0015] The reward module takes the first classification task variable and the first hidden state as input and outputs the predicted reward.

[0016] Optionally, the representation module and the transition module are jointly optimized using KL divergence to learn the prior and posterior distributions of the first hidden state.

[0017] Optionally, the hybrid world module also uses the first classification task variable as additional input to process covariate offsets in the input space.

[0018] Optionally, the first hidden state and the second hidden state are Gaussian mixture distributions conditioned on the first classification task variable.

[0019] Optionally, the predictive experience replay module is further configured to input the second classification task variable into the generator model copy of the previous task and output the initial frame of the trajectory of the previous task.

[0020] Optionally, the predictive experience playback module is further configured to determine the zero-initialization action of the trajectory of the previous task based on the initial frame of the trajectory of the previous task.

[0021] Optionally, the predictive experience replay module is further configured to input the initial frame of the trajectory of the previous task and the zero-initialization action of the trajectory of the previous task into the world model copy and action model copy of the previous task to generate the replay trajectory of the previous task.

[0022] Optionally, the predictive experience replay module is further configured to mix the replayed trajectory and the real trajectory to generate a hybrid trajectory, and use the hybrid trajectory to train the generative model of the current task and the robotic arm to determine the trained generative model and the robotic arm.

[0023] Optionally, the predictive experience replay module is further configured to update the current task's generation model copy, world model copy, and action model copy based on the previous task's generation model, world model, and action model.

[0024] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:

[0025] Through the above technical solution, based on the first and second category task variables, a Gaussian mixture prior is learned to obtain the latent dynamics of the current task. A copy of the generative model from the previous task is used as an additional generative model to reproduce the initial video frames of the historical task. Trajectory replay is performed using copies of the world model and the action model, mixing the replayed trajectory with the real trajectory to train the generative model and the robotic arm for the current task. The trained generative model and robotic arm are then determined, with the training process and trajectory replay process alternating. This disclosure enables the robotic arm to achieve efficient data replay, overcoming the catastrophic forgetting of the world model. In multi-task long-range decision-making scenarios, the robotic arm can make flexible decisions when facing multiple tasks, improving its autonomy and adaptability, and has broad application prospects and value. Attached Figure Description

[0026] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0027] Figure 1 This is a schematic diagram illustrating a reinforcement learning system for long-range decision-making in multi-task robotic arms, according to an exemplary embodiment.

[0028] Figure 2 This is a schematic diagram illustrating a reinforcement learning process for long-range multi-task decision-making of a robotic arm, according to an exemplary embodiment. Detailed Implementation

[0029] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.

[0030] Figure 1 This is a schematic diagram illustrating a reinforcement learning system for long-range decision-making in multi-task robotic arms, according to an exemplary embodiment.

[0031] like Figure 1 As shown, this disclosure provides a reinforcement learning system for multi-task long-range decision-making of robotic arms, including a hybrid world model module and a predictive experience replay module.

[0032] The Mixed World Model module takes the current visual observation, the action performed, and the first classification task variable of the current task as input, and uses Gaussian mixture variables to obtain the multimodal distribution of the spatial appearance of the input in the latent space, and outputs the reconstructed image.

[0033] The mixture world model learns a mixture Gaussian prior based on a set of categorical task variables to obtain the latent dynamics of the current task. The categorical task variables include a first categorical task variable k and a second categorical task variable k.

[0034] The predictive experience replay module takes the second-class task variables as input, uses copies of the generated model, world model, and action model from the previous task to replay the trajectory, and uses the replayed trajectory and the real trajectory to train the generated model and robotic arm for the current task, thus determining the trained generated model and robotic arm.

[0035] Among them, the predictive experience replay module performs data augmentation through trajectory replay to expand the training data for training the world model of the robotic arm, increase the number of observation-action pairs, and prevent the world model from overfitting the limited training data, which would result in poor performance of samples that are outside the distribution range.

[0036] Through the above technical solution, based on the first and second category task variables, a Gaussian mixture prior is learned to obtain the latent dynamics of the current task. A copy of the generative model from the previous task is used as an additional generative model to reproduce the initial video frames of the historical task. Trajectory replay is performed using copies of the world model and the action model. The replayed trajectory and the current trajectory are mixed to train the generative model and the robotic arm for the current task, determining the trained generative model and robotic arm. The training process and the trajectory replay process are performed alternately. This disclosure enables the robotic arm to achieve efficient data replay, overcoming the catastrophic forgetting of the world model. In multi-task long-range decision-making scenarios, the robotic arm can make flexible decisions when facing multiple tasks, improving its autonomy and adaptability, and has broad application prospects and value.

[0037] In one possible embodiment, the hybrid world model module may have different encoding and decoding architectures when applied to a specific task, but its encoding and decoding architectures always follow the unified learning paradigm of Gaussian mixture representation, using Gaussian mixture variables to obtain the multimodal distribution of the visual dynamics of the latent space and the spatial appearance in the observation space of the input / output.

[0038] In one possible embodiment, the hybrid world model module includes:

[0039] The representation module takes the current visual observation and the action performed as input and outputs the first hidden state.

[0040] The definition of the representation module is as follows:

[0041]

[0042] Among them, z t This represents the first hidden state at time t. Let z represent the visual observation at time t. t-1 This represents the hidden state at time t-1. Let represent the action of the k-th class sample at time t-1, where k represents the first classification task variable.

[0043] The representation module is used to encode the visual observations and actions performed at the current moment in order to predict the first hidden state.

[0044] like Figure 1 As shown, in one possible embodiment, the hybrid world module takes the first classification task variable as additional input and processes the covariate offset in the input space.

[0045] For example, the representation module takes the first classification task variable k∈{1,…,K} in the input space and processes the covariate shift in the input space.

[0046] The transition module takes the first classification task variable as input and outputs the predicted second hidden state.

[0047] The transition module is used to predict the second hidden state. With approximation of the first hidden state z t The posterior state.

[0048] The definition of the transfer module is:

[0049]

[0050] in, Represents the second hidden state, z t-1 Let represent the hidden state at time t-1, and let represent the action of the k-th sample at time t-1.

[0051] In one possible embodiment, the characterization module and the transition module are jointly optimized using KL divergence to learn the prior and posterior distributions of the first hidden state.

[0052] The observation module takes the first classification task variable and the first hidden state as input and outputs the reconstructed image.

[0053] The observation module is used to employ the first classification task variable k and the first hidden state z. t Reconstruct the image.

[0054] The definition of the observation module is:

[0055]

[0056] in, The image represents the reconstructed image, z t Let k represent the first hidden state and k represent the first category task variable.

[0057] The reward module takes the first category task variable and the first hidden state as input and outputs the predicted reward.

[0058] The reward module uses the first classification task variable k and the first hidden state z. t Predict the reward.

[0059] The definition of the reward module is:

[0060]

[0061] in, z represents the predicted reward. t Let k represent the first hidden state and k represent the first category task variable.

[0062] In the hybrid world model module, the first hidden state z t Second hidden state It is a Gaussian mixture distribution conditioned on the first category task variable.

[0063] In one possible embodiment, for the task The loss function of the world model includes the reconstruction loss of the input frame, the predicted reward loss, and the KL divergence loss. The loss function of the world model is:

[0064]

[0065] in, Represents the reconstruction loss of the input frame. D represents the predicted reward loss. KL (·||·) represents the KL divergence loss.

[0066] Where α = 1.0, This represents the training trajectory of the sampled data.

[0067] In one possible embodiment, It can be a training trajectory sampled from the data buffer of the current task or from the model sampled from the replayed trajectory.

[0068] In one possible implementation, the predictive experience replay module combines a generator copy of the previous task with a robotic arm to efficiently generate historical trajectories. To overcome the covariance of image appearance in time-varying environments, the introduced generator copy of the previous task also employs a Gaussian mixture distribution to form a latent prior e.

[0069] The predictive experience replay module retains a generator after training on the previous task. A world module A copy of the action model π, and two copies of the generator. World Model Copy Action model copy π'.

[0070] The generator is the generative model mentioned above.

[0071] In one possible embodiment, the predictive experience replay module is further configured to input the second classification task variables into a copy of the generative model of the previous task and output the initial frame of the trajectory of the previous task.

[0072] The predictive experience replay module is also used to determine the zero-initialization action of the trajectory of the previous task based on the initial frame of the trajectory of the previous task.

[0073] The predictive experience replay module is also used to input the initial frame of the trajectory of the previous task and the zero-initialized action of the trajectory of the previous task into the world model copy and action model copy of the previous task to generate the replay trajectory of the previous task.

[0074] For example, for each historical task Using generator copies Generate the initial frame of its historical trajectory, and generate a zero-initialization action a0 based on the initial frame, then use a copy of the world model. The action model copy π' is repeatedly executed to replay the trajectory, generating the replayed trajectory, i.e., the future sequence.

[0075]

[0076]

[0077]

[0078] In one possible embodiment, the predictive experience replay module is further configured to mix the replayed trajectory with the current trajectory to generate a hybrid trajectory, and use the hybrid trajectory to train the generator and robotic arm at the current moment, and determine the generator and robotic arm that have been trained.

[0079] Among them, the replay trajectory And the actual trajectory, i.e., the current task The real trajectory is mixed to generate a hybrid trajectory.

[0080] In one possible embodiment, during the current task In the process of predictive experience replay, the world model Minimize:

[0081]

[0082] in, Indicates a mixed trajectory. Indicates the replay trajectory. Represents the actual trajectory.

[0083] Furthermore, generator The objective function can be:

[0084]

[0085] Where q(·|·) represents the representation module, p(·|·) represents the encoding module, and D KL (·||·) represents the KL divergence loss, where k represents the first-class classification task variable. This represents the second category task variable. This represents the visual observation at time 1 of the k-th class. Indicates the first Visual observation at time 1, where e represents the true encoded vector. This represents the encoding vector for replay, where β is 10. -4 .

[0086] In one possible embodiment, the predictive experience replay module is further configured to update the generator copy, world model copy, and action model copy of the current task based on the generator model, world model, and action model of the previous task.

[0087] During predictive experience playback, i.e., during trajectory replay, the generator copy... World Model Copy The motion model copy π' will be frozen as the task is progressing and continuously updated to the next task until the task ends.

[0088] Among them, the world model They participated in the predictive experience replay process.

[0089] The above technical solution trains an additional generative model, i.e., a generator, to generate initial video frames from historical tasks and inputs them into a learned copy of the world model to generate subsequent image sequences for the robotic arm to learn. This process alternates with the training process of the robotic arm, eliminating the need to retain a large amount of historical data in the buffer and saving memory.

[0090] Figure 2 This is a schematic diagram illustrating a reinforcement learning process for long-range multi-task decision-making of a robotic arm, according to an exemplary embodiment.

[0091] like Figure 2 As shown, in one possible embodiment, the robotic arm interacts with environment A to learn task decisions in environment A and completes a series of tasks in environment A. After training the robotic arm in environment A, a generated model copy, a world model copy, and an action model copy of the robotic arm in environment A are retained.

[0092] In environment B, the robotic arm in environment B is updated based on the generated model copy, world model copy, and motion model copy preserved in environment A. The data from the robotic arm performing a series of tasks in environment A is then used to replay the trajectory, determining the replayed trajectory data. The robotic arm in environment B uses replayed trajectory data. Interact with environment B to learn task decisions in environment B and complete a series of tasks in environment B. After training the robotic arm in environment B, retain a generated model copy of the robotic arm, a world model copy, and an action model copy in environment B.

[0093] In environment C, the robotic arm in the current environment C is updated based on the generated model copy, world model copy, and motion model copy of the robotic arm preserved in environment B, and the data of the robotic arm in environment B performing a series of tasks in environment B is also transferred. The data from the robotic arm's actions in environment A, along with data from a series of individuals within environment A, are used to recreate the trajectory and determine the recreated trajectory data. The robotic arm in environment C uses replay trajectory data. and replay trajectory data Interact with environment C to learn how characters make decisions in environment C and complete a series of tasks in environment C.

[0094] The above technical solution trains an additional generative model copy to replay the initial video frames in the historical task, and inputs it into the learned world model copy and action model copy to generate subsequent image sequences for data training, i.e., replay data. This eliminates the need to retain a large amount of data in the buffer, saving memory.

[0095] The specific embodiments of this disclosure have been described above. It should be understood that this disclosure is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this disclosure. The above-described preferred features can be used in any combination without conflict.

Claims

1. A reinforcement learning system for long-range decision-making in multi-task robotic arms, characterized in that, include: The hybrid world model module takes the current visual observation, the action performed, and the first-class task variable of the current task as input, and uses Gaussian mixture variables to obtain the latent space visual dynamics and the multimodal distribution of the spatial appearance in the input / output observation space, and outputs the reconstructed image. The predictive experience replay module takes the second-class task variables as input, uses the generator model copy, world model copy, and action model copy of the previous task to replay the trajectory, and uses the replayed trajectory and the current trajectory to train the generator and robotic arm of the current task, and determines the trained generator and robotic arm. The predictive experience replay module is further used to input the initial frame of the trajectory of the previous task and the zero-initialization action of the trajectory of the previous task into the world model copy and action model copy of the previous task to generate the replay trajectory of the previous task. The predictive experience replay module is also used to mix the replayed trajectory and the real trajectory to generate a hybrid trajectory, and use the hybrid trajectory to train the generative model of the current task and the robotic arm to determine the trained generative model and the robotic arm.

2. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 1, characterized in that, The hybrid world model module includes: The representation module takes the current visual observation and the executed action as input and outputs the first hidden state; The transition module takes the first classification task variable as input and outputs the predicted second hidden state. The observation module takes the first classification task variable and the first hidden state as input and outputs the reconstructed image. The reward module takes the first classification task variable and the first hidden state as input and outputs the predicted reward.

3. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 2, characterized in that, The representation module and the transition module are jointly optimized using KL divergence to learn the prior and posterior distributions of the first hidden state.

4. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 2, characterized in that, The hybrid world model module also uses the first classification task variable as additional input to process covariate shifts in the input space.

5. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 2, characterized in that, The first hidden state and the second hidden state are Gaussian mixture distributions conditioned on the first classification task variable.

6. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 1, characterized in that, The predictive experience replay module is also used to input the second classification task variable into the generator model copy of the previous task and output the initial frame of the trajectory of the previous task.

7. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 6, characterized in that, The predictive experience playback module is also used to determine the zero-initialization action of the trajectory of the previous task based on the initial frame of the trajectory of the previous task.

8. The reinforcement learning system for long-range decision-making in multi-task robotic arms according to claim 1, characterized in that, The predictive experience replay module is also used to update the current task's generation model copy, world model copy, and action model copy based on the previous task's generation model, world model, and action model.

Images