Mechanical arm strategy learning method and system based on mixed target remarking

Abstract

The invention discloses a mechanical arm strategy learning method and system based on mixed target remarking, and provides an improved experience playback mechanism aiming at the problems of low sample efficiency and difficult strategy convergence caused by sparse rewards in multi-target reinforcement learning. The method comprises the following steps: constructing a Markov decision process containing a state, an action and a target space; controlling the mechanical arm to interact with the environment and storing track data; in the training stage, a Next-Future mixed strategy is adopted to carry out target remarking on sampled data, that is, a next state of current conversion is forcibly selected as a virtual target to ensure that a single-step non-negative reward is obtained, and meanwhile, a randomly selected future state is combined as the virtual target to promote value propagation; network parameters are updated using a truncated quantile commentator (TQC) algorithm to suppress over-estimation deviation. According to the method, learning signals are enhanced through deterministic single-step rewards, and the control precision and the convergence speed of the mechanical arm in complex tasks such as pushing, grabbing and taking and placing are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the fields of robot control, automation and artificial intelligence technology, specifically a method and system for learning robotic arm strategies based on hybrid target relabeling. Background Technology

[0002] In the field of robotics, deep reinforcement learning (DRL) has become a highly promising solution for tasks such as object grasping, flipping, and stacking. However, traditional reinforcement learning methods face the serious challenge of low sample efficiency when applied to robotic arm control, requiring a large amount of trial and error data to learn effective policies, which is often expensive and time-consuming in real-world physical environments.

[0003] To address the issue of sample efficiency, Hindsight Experience Replay (HER) technology has been widely adopted. HER allows agents to learn from failed attempts by relabeling trajectories that failed to achieve their goals as successful trajectories that achieved some virtual goals, thereby obtaining learning signals in a sparse binary reward environment. However, existing HER algorithms mainly rely on heuristic relabeling strategies (such as randomly selecting a future state as the goal). This approach lacks a fundamental framework and performs poorly in tasks requiring high precision. As the required precision threshold increases, randomly sampled targets are unlikely to fall precisely within the success region, resulting in sparse effective reward signals and a significant decrease in learning efficiency.

[0004] The differences between this application and the prior art are as follows:

[0005] Technical comparison with patent CN115169747A "Method, apparatus and related equipment for predicting non-stationary time series of power load";

[0006] The core idea of ​​the technical solution in patent CN115169747A is to decompose the power load sequence into low-frequency and high-frequency subsequences using discrete wavelet transform, and then predict them using both the ARIMAX model and the attention-based LSTM model. Finally, the final prediction result is obtained through wavelet reconstruction, denormalization, and superposition. In contrast, the core idea of ​​this patent is to predict the net load average and load fluctuation entropy spectrum in parallel, analyze the fluctuation risk type based on the entropy spectrum, match adjustment resources from a heterogeneous resource library, and generate comprehensive dispatch instructions. The application objectives of the two are significantly different.

[0007] The core objective of patent CN115169747A is to solve a purely predictive accuracy problem. Its focus is on how to more accurately predict future load values ​​from nonlinear, non-stationary load time series. In contrast, this patent addresses the safety, economy, and refined scheduling of power grid operation. Its focus is on how to utilize predictive information to prepare resources in advance to cope with the risks brought about by load fluctuations. The core objectives of the two are clearly different.

[0008] Patent CN115169747A treats "fluctuation characteristics" by considering the load sequence as composed of "components" of different frequencies. It uses wavelet decomposition to break down the original sequence into low-frequency (trend term) and high-frequency (detail term), but it neither preserves nor outputs the independent characteristics of the fluctuations. In contrast, while this patent also performs wavelet decomposition, its purpose is not to predict the components themselves, but rather to calculate the permutation entropy of each scale component, thus forming a vector called the "load fluctuation entropy spectrum." The entropy value quantifies the disorder and complexity of the fluctuations at that scale, and the entropy spectrum is used to identify and classify fluctuation risks (such as high-frequency fluctuation risk, mid-frequency ramp risk). The two patents clearly differ in their approaches to "fluctuation characteristics."

[0009] Comparison with the technology of patent CN118445600A "Power load prediction method, device, medium and equipment based on feature extraction";

[0010] Patent CN118445600A aims to improve the accuracy of final load forecast values. It focuses on extracting more useful features from raw data to improve model learning, using Discrete Wavelet Transform (DWT), Discrete Wavelet Flip (DFT), and statistical methods as feature engineering tools to expand the dimensionality and information content of input features. In contrast, this patent's purpose is not to better predict load values, but rather to determine the types of risks future load fluctuations will pose to grid stability. After using DWT decomposition, it does not predict the specific values ​​of each component, but rather calculates the permutation entropy of each component. The application goals of the two patents differ significantly.

[0011] The output of the prediction model in patent CN118445600A is a single (or set of) load forecast values. These forecasts can be provided to dispatchers as reports or used as input data for another independent dispatch system. They do not involve any dispatch decisions themselves. In contrast, the output of the prediction model in this patent is a set of structured, decision-oriented information packets. These outputs are directly fed into the decision and command generation module to generate dispatch commands for the power grid, serving as the starting point for control actions. The output properties of the two are completely different from their subsequent applications.

[0012] Patent CN118445600A does not involve resource libraries, resource matching, or any form of scheduling decision-making. All its innovations focus on data preprocessing (feature extraction and compression) and predictive models (Autoformer integration), belonging to the pure data analysis level. In contrast, this patent explicitly includes a "scheduling decision-making and resource pre-matching module," which has a built-in heterogeneous resource feature library storing performance parameters (such as response time and adjustment rate) of various resources (e.g., thermal power units, energy storage batteries). Based on the diagnosed risk type, the system automatically selects and matches the most suitable resources from the library to form a "prepared adjustment resource pool." The two patents are fundamentally different in whether they include "resource matching" and "scheduling decision-making" stages. Summary of the Invention

[0013] The purpose of this invention is to propose a strategy learning method and system for robotic arms based on hybrid target relabeling, which overcomes the shortcomings of low sample efficiency and difficulty in high-precision control convergence in existing technologies for robotic arms under multi-target reinforcement learning tasks.

[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0015] The robotic arm policy learning method based on hybrid target relabeling includes the following steps:

[0016] S1. Construct a multi-objective Markov decision process for robotic arm tasks, defining the state space, action space, target space, and binary reward function;

[0017] S2. Control the robotic arm to interact with the environment, collect interaction trajectory data containing the original target, and store the trajectory data in the experience playback buffer;

[0018] S3. Sample a batch of transformation data from the experience playback buffer, and generate virtual targets using a hybrid target relabeling strategy to relabel the transformation data; the hybrid target relabeling strategy includes: for each transformation of the sampled data, select the next moment state of the transformation in the trajectory as the first type of virtual target, i.e., the "Next" strategy; select the state of any subsequent moment in the trajectory where the transformation is located as the second type of virtual target, i.e., the "Future" strategy;

[0019] S4. Train a deep reinforcement learning network using the relabeled data. The network includes a policy network and a critic network, wherein the critic network uses a truncated quantile critic architecture for value estimation and parameter updates.

[0020] As a preferred technical solution of the present invention: in step S1, the robotic arm task includes at least one of push, pick-and-place, slide, and stack; the state space includes the position and speed of the robotic arm end effector, as well as the relative position and rotation angle of the manipulated object.

[0021] As a preferred technical solution of the present invention: In step S3, the specific execution method of the hybrid target relabeling strategy is as follows: setting the relabeling ratio parameter. For each sample transformation ,generate One relabeled sample; where the first sample is forced to target Replace with the state of the next time step Ensure that the reward calculated according to the binary reward function is non-negative; the rest... One sample from this trajectory Samples are taken from random states after time step 1 as virtual targets.

[0022] As a preferred embodiment of the present invention: the binary reward function is defined as follows: when the distance between the position of the robotic arm end or the manipulated object and the target position is less than a preset threshold... When the reward is 0 or 1, the reward is -1; otherwise, the reward is -1. After remarking through the "Next" strategy, the calculated reward value is always non-negative.

[0023] As a preferred technical solution of the present invention: In step S4, the truncated quantile critic architecture suppresses overestimation bias by: constructing multiple critic networks, each critic network outputting multiple quantiles of the return distribution; when calculating the target Q value, collecting the quantiles output by all target critic networks and sorting them by numerical value; truncating the quantiles with the largest values ​​after sorting. For partial quantiles, only the mean of the remaining quantiles is used as the target Q value for network updates.

[0024] This invention relates to a robotic arm policy learning system based on hybrid target relabeling, the system comprising:

[0025] The interactive sampling module is used to control the robotic arm to perform actions and obtain environmental feedback status information to generate trajectory data;

[0026] A data storage module is used to store the trajectory data and the corresponding original target;

[0027] The relabeling module is used to read trajectory data during training and apply the "Next-Future" strategy to replace the original target with the next state target or a future random state target, generating enhanced training samples;

[0028] The policy update module, comprising a truncated quantile critic network and a policy network, is used to calculate the loss function and update the network weights based on the augmented training samples, and output the optimal control policy.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The "Next" strategy proposed in this invention utilizes the characteristic that a single-step transition is bound to succeed, and can deterministically provide a non-negative reward signal, which significantly accelerates the learning process of the value function and greatly reduces the number of training steps required on most tasks compared with the traditional HER method.

[0031] 2. By combining the "Next" and "Future" strategies, this invention not only ensures the generation of dense reward signals, but also takes into account the value propagation of long-range targets, enabling the robotic arm to complete delicate operation tasks such as pushing and stacking with extremely small error tolerance, and the success rate is significantly better than that of the prior art.

[0032] 3. The introduction of the truncated quantile commentator (TQC) mechanism effectively suppresses the overestimation of Q-values ​​and ensures the stability of policy updates in complex continuous action spaces. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0034] Figure 2 This is a schematic diagram comparing the principles of the hybrid target relabeling strategy (Next-Future) in this invention with traditional strategies;

[0035] Figure 3 This is a comparison chart of the success rate convergence curves of the method of the present invention in the task of pushing objects with a robotic arm;

[0036] Figure 4 This is a schematic diagram of the module structure of the system of the present invention. Detailed Implementation

[0037] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0038] The diagram illustrating the principle comparison between the hybrid target relabeling strategy (Next-Future) and the traditional strategy is shown below. Figure 2 As shown in the diagram, the module structure is as follows: Figure 4 As shown.

[0039] Example 1:

[0040] like Figure 1 As shown in the figure, this embodiment discloses a robotic arm policy learning method based on hybrid target relabeling, and the specific steps are as follows:

[0041] Step S1: Environment setup.

[0042] A simulation environment based on the PyBullet physics engine (such as panda-gym) is used. The state space of the robotic arm is defined, including the end-effector position, velocity, object position, and object orientation. The motion space includes the displacement increments of the end-effector and the opening and closing states of the grippers. A reward function is defined. Defined as: when the Euclidean distance between the object's position and the target position is less than a threshold. hour, (Success), otherwise (fail) .

[0043] Step S2: Interaction and storage.

[0044] Initialize policy network and the network of critics The robotic arm interacts with the environment based on the current strategy to generate a trajectory. Transform each tuple in the trajectory Store in the experience replay buffer .

[0045] Step S3: Mixed target relabeling (Next-Future).

[0046] From the buffer A batch of transformed data is sampled. For each sampled transformation, the following remarking operation is performed:

[0047] 1. The goal of this transformation Replace with .because This is the transformed state. According to the definition of the reward function, the distance is 0 at this point, which is necessarily less than the threshold. Therefore, the reward obtained from this transformation is adjusted to a non-negative value (such as 0 or 1). This provides the agent with a definite success signal.

[0048] 2. In addition to the "Next" sample, generate another sample. Each relabeled sample (e.g.) These samples' targets are from this trajectory. Choose from random states after a given time step. This helps propagate value from this step to a more distant target state.

[0049] Step S4: Network update (based on TQC).

[0050] The network parameters are updated using the Truncation Quantile Critics (TQC) method, as follows:

[0051] 1. Create multiple commentator network heads (e.g., 5), each head outputting... Quantities (e.g., 25).

[0052] 2. When calculating the target value, summarize the quantiles of all target network outputs, sort them, and remove the largest quantile. The target Q value is the average of the remaining portion, calculated as the percentage (e.g., 8%).

[0053] 3. Calculate the Huber quantile loss and update the commentator network parameters.

[0054] 4. Maximize the truncated Q value to update the policy network parameters.

[0055] Example 2

[0056] To verify the effectiveness of the invention, it was tested in a standard robotic arm "push" task.

[0057] Experimental parameter settings: learning rate Batch size: 2048, Buffer size: Discount factor 23. The network structure uses three fully connected layers, each with 512 neurons, and the activation function is ReLU.

[0058] like Figure 3 As shown, the horizontal axis represents the number of simulation steps, and the vertical axis represents the success rate. The method of this invention (Next-Future) (green curve) achieves a success rate close to 1.0 after approximately 200k steps, while the traditional HER (Future) strategy (red curve) only achieves a success rate of around 0.2 at the same number of steps, and its convergence speed is significantly slower. Experimental results demonstrate that this invention outperforms existing technologies in both sample efficiency and final success rate.

[0059] Example 3

[0060] This embodiment addresses the more challenging "Stacking" task. In this task, the robotic arm needs to stack two cubes together. Utilizing the hybrid relabeling strategy of this invention, the "Next" strategy enhances the robotic arm's immediate feedback learning for minute action steps such as "grasping" and "moving," while the "Future" strategy learns long-term planning. Experimental data show that the method of this invention achieves a maximum success rate of 0.93 in the stacking task, which is superior to the 0.9027 of the traditional method, demonstrating the effectiveness of this system in complex, multi-stage tasks.

[0061] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. The multimodal fusion described in the present invention can be extended to all modalities that can be obtained in any scenario, and is not limited to the environment and device modalities in the examples of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A robotic arm policy learning method based on hybrid target relabeling, characterized in that, Includes the following steps: S1. Construct a multi-objective Markov decision process for robotic arm tasks, defining the state space, action space, target space, and binary reward function; S2. Control the robotic arm to interact with the environment, collect interaction trajectory data containing the original target, and store the trajectory data in the experience playback buffer; S3. Sample a batch of transformed data from the experience playback buffer, generate virtual targets using a hybrid target relabeling strategy, and relabel the transformed data; The hybrid target relabeling strategy includes: for each transformation of the sampled data, selecting the next time-instance state of the transformation in the trajectory as the first type of virtual target, i.e., the "Next" strategy; The state at any subsequent moment in the trajectory where the transformation occurs is selected as the second type of virtual target, namely the "Future" strategy; S4. Train a deep reinforcement learning network using the relabeled data. The network includes a policy network and a critic network, wherein the critic network uses a truncated quantile critic architecture for value estimation and parameter updates.

2. The robotic arm strategy learning method based on hybrid target relabeling according to claim 1, characterized in that, In step S1, the robotic arm task includes at least one of push, pick-and-place, slide, and stack; the state space includes the position and speed of the robotic arm end effector, as well as the relative position and rotation angle of the manipulated object.

3. The robotic arm strategy learning method based on hybrid target relabeling according to claim 1 or 2, characterized in that, In step S3, the specific execution method of the hybrid target remarking strategy is as follows: setting the remarking ratio parameter. For each sample transformation ,generate One relabeled sample; where the first sample is forced to target Replace with the state of the next time step Ensure that the reward calculated according to the binary reward function is non-negative; the rest... One sample from this trajectory Samples are taken from random states after time step 1 as virtual targets.

4. The robotic arm strategy learning method based on hybrid target relabeling according to claim 3, characterized in that, The binary reward function is defined as follows: when the distance between the position of the robotic arm's end effector or the manipulated object and the target position is less than a preset threshold... When the reward is 0 or 1, the reward is -1; otherwise, the reward is -1. After remarking through the "Next" strategy, the calculated reward value is always non-negative.

5. The robotic arm strategy learning method based on hybrid target relabeling according to claim 1 or 2, characterized in that, In step S4, the truncated quantile critic architecture suppresses overestimation bias by: constructing multiple critic networks, each outputting multiple quantiles of the return distribution; when calculating the target Q-value, collecting the quantiles output by all target critic networks and sorting them by value; truncating the quantiles with the largest values ​​after sorting. For partial quantiles, only the mean of the remaining quantiles is used as the target Q value for network updates.

6. A robotic arm policy learning system based on hybrid target relabeling, used to implement the method according to any one of claims 1-5, characterized in that, The system includes: The interactive sampling module is used to control the robotic arm to perform actions and obtain environmental feedback status information to generate trajectory data; A data storage module is used to store the trajectory data and the corresponding original target; The relabeling module is used to read trajectory data during training and apply the "Next-Future" strategy to replace the original target with the next state target or a future random state target, generating enhanced training samples; The policy update module, comprising a truncated quantile critic network and a policy network, is used to calculate the loss function and update the network weights based on the augmented training samples, and output the optimal control policy.

Images