A multi-sub-target task implementation method based on a double-coupled intelligent agent network

Abstract

The application discloses a kind of based on double coupling agent network's multiple sub-target task implementation method, comprising: the state of current environment and desired target of real-time acquisition are input into trained double coupling agent network, obtain corresponding multiple sub-target task implementation strategy;Double coupling agent network includes: using agent: main network A obtains corresponding action and its evaluation value and external reward according to current environment state and target, and is stored to experience playback area;Target network A completes the optimization of main network A according to evaluation value;Exploration agent: main network B is randomly obtained from experience playback area sample, predicts the next state of this sample state, and calculates internal reward, and target network B is optimized to main network B according to predicted state and internal reward, coupling network generates action and its external reward according to current environment state and target, and external reward is used to update using agent, and completes the optimization of main network B according to internal reward.

Description

A method for implementing multi-sub-objective tasks based on dual-coupled agent networks Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and specifically to a method for implementing multi-sub-objective tasks based on dual-coupled agent networks. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, robotic arms are playing an increasingly important role on production lines. They are widely used in complex tasks such as material handling, assembly, stacking, and sorting. In these tasks, the flexibility, intelligence, and autonomous decision-making capabilities of robotic arms are key factors in improving production efficiency, reducing labor costs, and ensuring product quality.

[0003] However, in complex multi-object tasks (such as multi-block stacking and multi-part assembly), traditional robotic arm control methods face exponentially increasing interactions and spatial constraints between objects, making it difficult for traditional methods to effectively handle combinatorial complexity. Multi-block stacking tasks, in particular, require control strategies that are far more complex than simply grasping and placing single blocks, and are fundamentally different in nature. Specifically, they face the following key challenges:

[0004] 1. Sparse Reward Dilemma: In multi-block stacking tasks, a positive reward is only obtained when all blocks are correctly stacked to the target position, with almost no feedback signal in intermediate states. As the number of blocks increases (e.g., from 2 to 4), the success probability decreases exponentially, resulting in the agent being unable to obtain effective learning signals in the early stages of training, rendering traditional reinforcement learning algorithms completely ineffective.

[0005] 2. Explosion Space Explosion: In multi-block tasks, the target space dimension increases linearly with the number of blocks (N blocks correspond to 3N-dimensional target coordinates), and the possible stacking combinations increase factorially, forming a vast exploration space. The sample efficiency of existing algorithms cannot support completing effective exploration within a reasonable time.

[0006] 3. Temporal dependence and constraint satisfaction: Stacking tasks require a specific order of operations (such as placing the bottom block first) and must satisfy physical constraints (such as stability and collision avoidance). This strong temporal dependence and multiple constraints greatly increase the difficulty of policy learning.

[0007] 4. Coupling between subtasks: There are complex interactions between the stacking subtasks of different blocks. For example, the position of one block can affect the accessibility and stacking stability of other blocks. Traditional methods are difficult to effectively model and utilize this coupling structure. Summary of the Invention

[0008] In view of this, the present invention provides a method for implementing multi-sub-target tasks based on dual-coupled agent networks, which at least solves the problems of low agent learning efficiency and agent learning failure as the number of sub-targets increases in the prior art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A method for implementing multi-sub-objective tasks based on dual-coupled agent networks includes the following steps:

[0011] The current environmental state and desired objective, acquired in real time, are input into a pre-trained dual-coupled agent network to obtain corresponding multi-sub-objective task implementation strategies; the dual-coupled agent network includes: an exploit agent and an exploration agent;

[0012] The use of intelligent agents includes: the main network A obtains the corresponding actions, their evaluation values, and external rewards based on the current environmental state and the goal, and stores them in the experience replay area; the target network A optimizes the main network A based on the evaluation values;

[0013] The exploration agent includes: the main network B randomly obtains samples from the experience replay area, predicts the next state of the sample state through the forward dynamics model FD, and calculates the internal reward; the target network B optimizes the main network B based on the predicted state and the internal reward; and the coupled network generates actions to interact with the environment and their external rewards based on the current environmental state and the target, where the external rewards are used to update the utilizing agent and complete the optimization of the main network B based on the internal rewards.

[0014] Preferably, the main network A includes:

[0015] The first actor network actor1 is used to obtain (from the environment) , ),in, The state at time step t, This represents the expected objective of achieving the current multi-sub-objective task, based on ( , Output the corresponding action ;

[0016] The first noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0017] Critics networks C1 and C2 are dual-network frameworks using TD3, and are independently based on ( , , Estimate the state-action-goal value and output Q1( , , ) and Q2( , , );

[0018] External reward unit, used to calculate the external reward corresponding to the action at time step t. and output external rewards To target network A.

[0019] Preferably, the target network A includes:

[0020] The second actor network, actor2, is used to obtain the execution... The state of the environment at the next time step t+1 ,according to( , Generate target action ;

[0021] The second noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0022] Critics networks C'1 and C'2 are dual-network frameworks using TD3, and are independently based on ( , , Estimate the state-action-goal value and output Q'1( , , ) and Q'2( , , );

[0023] The first TD target acquisition unit is used to acquire targets based on Q'1 ( , , ) and Q'2( , , ) and external rewards Calculate the target value y of TD, where , , Discount factor;

[0024] Storage unit, used to store Stored in the experience replay area, where and They represent and and The state after splicing.

[0025] Preferably, the main network B includes:

[0026] The third actor network, actor3, is used to update the target network B via soft updates.

[0027] Forward dynamics model FD is used to randomly obtain arbitrary samples from the empirical replay area. , ), thus obtaining the predicted state at the next time step t+1. According to the sample The corresponding next time step With predicted state The error between them multiplied by the scaling factor Receive internal rewards ;

[0028] Critics networks C3 and C4 are dual-network frameworks using TD3, and are independently based on ( , Based on internal rewards Estimate the state-action value and output Q1( , ) and Q2( , ).

[0029] Preferably, the target network B includes:

[0030] The fourth actor network, actor4, is used to obtain samples. The corresponding next time step +1 Output the corresponding action ;

[0031] The third noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0032] Critics Network C'3 and Critics Network C'4 are dual-network frameworks using TD3, and are independently based on ( , Estimate the state-action value and output Q'1( , ) and Q'2( , );

[0033] The second TD target acquisition unit is used to acquire targets based on Q'1 ( , ) and Q'2( , ) and internal rewards Calculate the target value y of TD, where , .

[0034] Preferably, the coupling network includes:

[0035] The fifth actor network, actor5, is used to obtain the environment state. and their corresponding objectives Output the corresponding action ;

[0036] The fourth noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0037] External reward unit, used for calculation Corresponding external rewards ;

[0038] Internal commentator network C in Reuse the critic network C3 for use based on ( , Estimate and evaluate the value of actions in exploration, and output the results. ( , );

[0039] External commentator network C ex Reuse the critic network C1 for use based on ( , , Evaluate the value of actions in task completion and output... ( , , );

[0040] Q-value weighted average unit, used to calculate based on ( , )and ( , , The weighted average is obtained ( , , ).

[0041] Preferably, the training process of a dual-coupled agent network includes:

[0042] A distributed training architecture is adopted, with 2n processes executing the training process in parallel. The agent that interacts with the environment to collect training data is determined according to the process number. Even-numbered processes use exploratory agents, and odd-numbered processes use exploit agents. The exploratory agents and exploit agents share the same sample data in the same experience playback area.

[0043] In both the exploiting agent and the exploring agent, the critic network of the main network constructs the critic loss function based on the temporal difference TD objective in each training iteration and updates the parameters through gradient descent.

[0044] The actor network of the main network adopts a delayed update mechanism: whenever the critic network of the main network completes a certain number of preset updates, the parameters of the actor network of the main network are updated once according to the deterministic policy gradient theorem, and at the same time, the actor network and critic network in the target network are simultaneously soft-updated according to the Polyak average.

[0045] The forward dynamics model FD uses the mean squared error of state prediction as the loss function and uses gradient descent for updating.

[0046] In the coupled network, only the actor network is updated, and its parameters are updated based on the weighted average Q-value of its own output according to the deterministic policy gradient theorem.

[0047] Preferably, the training process of a dual-coupled agent network includes:

[0048] (1) After the first actor network or the fifth actor network interacts with the environment for one round, the experience data is stored in the experience replay area;

[0049] (2) Call the DSHER algorithm to enhance the small batch of data randomly sampled from the experience replay area to obtain the small batch of experience data B after DSHER enhancement, and update the parameters of the dual-coupled agent network based on the small batch of experience data B;

[0050] (3) Use the first actor network to output the action, without using any noise to interact with the environment for one round;

[0051] The training cycle is defined as: executing step (1) p times and then executing step (2) q times, where q is greater than p;

[0052] The training epoch is defined as follows: after completing m cycles, step (3) is executed n times and the success rate is recorded, where n is greater than m;

[0053] Repeat the process f times until the recorded success rate converges.

[0054] Preferably, the specific enhancements to the DSHER algorithm include:

[0055] Multiple groups were randomly sampled from the experience replay area. , , , As a batch, obtain the corresponding success rate. Adaptive HER probability calculate:

[0056] ;

[0057] ;

[0058] ;

[0059] In the formula, As the attenuation factor, To adjust the dynamic coefficients, k is the number of augmented samples, a is the scaling factor, and controls the exponential term. The overall magnitude determines the maximum value of the decay factor. When the success rate suc_rate is low, the decay factor is close to a+c, which has an amplifying effect. b is the decay rate, which controls the sensitivity of the exponential function to the success rate. The larger the value of b, the more drastic the impact of the success rate change on the decay factor, and the faster the system response. c is the offset, which provides a lower limit for the decay factor, ensuring that even if the success rate is very high, the decay factor will not drop to zero, maintaining the minimum exploration activity of the system.

[0060] In a batch, iterate through the targets corresponding to the sampled samples. For each sub-target, generate a random number between (0, 1). If this random number is less than the currently calculated number... If the target is not found, then replace the current sub-target with any sub-target that has been achieved after time step t; otherwise, leave the atomic target unchanged.

[0061] When a sub-objective is replaced, the replaced sub-objective is reused to calculate the external reward.

[0062] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for implementing multi-sub-objective tasks based on dual-coupled intelligent agent networks, which has the following beneficial effects:

[0063] This invention improves sample utilization and effective training signal density in multi-sub-target tasks. During the DSHER algorithm enhancement process, the overall target is split into sub-targets, allowing independent target replacement and recombination of sub-targets. This expands the number of "achieved target combinations" that can be generated along with the sub-target dimension, enabling more efficient construction of learnable samples from failure trajectories and alleviating the sample scarcity problem caused by target space expansion in multi-sub-target tasks.

[0064] This invention alleviates the sparse reward dilemma and supports progressive achievement of complex tasks. It employs a sub-goal-level reward calculation method: rewards are calculated separately for each sub-goal and then aggregated. This allows the agent to receive feedback even when only some sub-goals are completed, reducing the risk of training stagnation caused by "rewards only upon overall completion" and improving the learning differentiability and training stability of the strategy for complex tasks.

[0065] This invention adaptively balances "replay target training" and "real target generalization" to reduce the risk of bias in later stages. It introduces a dynamic HER probability based on success rate into the DSHER algorithm: maintaining a high target replacement probability in the early stages of training when the success rate is low to provide a denser learning signal; gradually decreasing the target replacement probability as the success rate increases, allowing the policy attention to return to the real target distribution, thereby helping to improve the generalization ability to real targets in later stages and avoiding long-term training bias towards replay targets.

[0066] This invention improves exploration efficiency and establishes a collaborative "exploration-exploitation" training mechanism. The dual-coupled agent network of this invention simultaneously comprises an exploit agent and an exploration agent: the exploit agent is driven by external rewards to achieve the desired goal, while the exploration agent uses a forward dynamics model to predict errors and generate internal rewards to drive exploration. The coupling network weights the exploration value and the task value, ensuring that the strategy for interacting with the environment maintains exploration capabilities while also considering task achievement. This helps to discover effective trajectories faster and improve convergence speed in multi-sub-goal tasks.

[0067] The invention demonstrates a significant improvement in success rate in multi-objective task scenarios (simulation verification). Under relevant simulation experimental settings, the success rate of the proposed solution can be observed to increase from close to 0% to over 99% in some multi-objective tasks where the HER algorithm struggles to make training progress. Furthermore, it achieves high success rate performance in various multi-objective environments, demonstrating the effectiveness and robustness of the proposed solution in multi-objective tasks. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 is a schematic diagram of the specific structure of the dual-coupled agent network in a multi-sub-target task implementation method based on a dual-coupled agent network provided by the present invention.

[0070] Figure 2 is a schematic diagram of the parallel training architecture of the dual-coupled intelligent agent network provided by the present invention;

[0071] Figure 3 shows the average success rate learning curves of D2-HER under 11 different complexities provided by the embodiment of the present invention. Figures 3(a) to (d) represent the curves corresponding to push1 to push4, Figures 3(e) to (h) represent the curves corresponding to PickAndPlace1 to PickAndPlace4, and Figures 3(i) to (k) represent the curves corresponding to stack2 to stack4. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] This invention provides a method for implementing multi-sub-objective tasks based on dual-coupled agent networks, comprising the following steps:

[0074] The current environment state and desired target are obtained in real time and input into the trained dual-coupled agent network to obtain the corresponding multi-sub-target task implementation strategy; as shown in Figure 1, the dual-coupled agent network includes: an exploit agent and an exploration agent;

[0075] The use of intelligent agents includes: the main network A obtains the corresponding actions, their evaluation values, and external rewards based on the current environmental state and the goal, and stores them in the experience replay area; the target network A optimizes the main network A based on the evaluation values;

[0076] The exploration agent includes: the main network B randomly obtains samples from the experience replay area, predicts the next state of the sample state through the forward dynamics model FD, and calculates the internal reward; the target network B optimizes the main network B based on the predicted state and the internal reward; and the coupled network generates actions to interact with the environment and their external rewards based on the current environmental state and the target, where the external rewards are used to update the utilizing agent and complete the optimization of the main network B based on the internal rewards.

[0077] It should be noted that:

[0078] The dual-coupled agent architecture consists of a pair of specialized agents: an explorer agent (Figure 1) and an exploit agent (Figure 1). Each agent plays a different role: the exploit agent is responsible for achieving the desired goal, while the explorer agent focuses on exploring the environment and discovering new states. To guide their behavior according to their respective tasks, the agents are driven by corresponding reward signals, which consist of external rewards provided by the environment and internal rewards generated by intrinsic motivation mechanisms.

[0079] To implement the above technical solution, the main network A includes:

[0080] The first actor network actor1 is used to obtain (from the environment) , ),in, The state at time step t, This represents the expected objective of achieving the current multi-sub-objective task, based on ( , Output the corresponding action ;

[0081] The first noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0082] Critics networks C1 and C2 are dual-network frameworks using TD3, and are independently based on ( , , Estimate the state-action-goal value and output Q1( , , ) and Q2( , , );

[0083] External reward unit, used to calculate the external reward corresponding to the action at time step t. and output external rewards To target network A.

[0084] To implement the above technical solution, the target network A includes:

[0085] The second actor network, actor2, is used to obtain the execution... The state of the environment at the next time step t+1 ,according to( , Generate target action ;

[0086] The second noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0087] Critics networks C'1 and C'2 are dual-network frameworks using TD3, and are independently based on ( , , Estimate the state-action-goal value and output Q'1( , , ) and Q'2( , , );

[0088] The first TD target acquisition unit is used to acquire targets based on Q'1 ( , , ) and Q'2( , , ) and external rewards Calculate the target value y of TD, where , , Discount factor;

[0089] Storage unit, used to store Stored in the experience replay area, where and They represent and and The state after splicing.

[0090] To implement the above technical solution, the main network B includes:

[0091] The third actor network, actor3, is used to update the target network B via soft updates.

[0092] Forward dynamics model FD is used to randomly obtain arbitrary samples from the empirical replay area. , ), thus obtaining the predicted state at the next time step t+1. According to the sample The corresponding next time step With predicted state The error between them multiplied by the scaling factor Receive internal rewards ;

[0093] Critics networks C3 and C4 are dual-network frameworks using TD3, and are independently based on ( , Based on internal rewards Estimate the state-action value and output Q1( , ) and Q2( , ).

[0094] To implement the above technical solution, the target network B includes:

[0095] The fourth actor network, actor4, is used to obtain samples. The corresponding next time step Output the corresponding action ;

[0096] The third noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0097] Critics Network C'3 and Critics Network C'4 are dual-network frameworks using TD3, and are independently based on ( , Estimate the state-action value and output Q'1( , ) and Q'2( , );

[0098] The second TD target acquisition unit is used to acquire targets based on Q'1 ( , ) and Q'2( , ) and internal rewards Calculate the target value y of TD, where , .

[0099] To implement the above technical solution, the coupling network includes:

[0100] The fifth actor network, actor5, is used to obtain the environment state. and their corresponding objectives Output the corresponding action ;

[0101] The fourth noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ;

[0102] External reward unit, used for calculation Corresponding external rewards ;

[0103] Internal commentator network C in Reuse the critic network C3 for use based on ( , Estimate and evaluate the value of actions in exploration, and output the results. ( , );

[0104] External commentator network C ex Reuse the critic network C1 for use based on ( , , Evaluate the value of actions in task completion and output... ( , , );

[0105] Q-value weighted average unit, used to calculate based on ( , )and ( , , The weighted average is obtained ( , , ).

[0106] It should be noted that:

[0107] The coupled network receives two Q-values ​​from the outputs of the main networks of the two agents, which are then weighted and averaged to optimize the parameters of the actor network in the coupled network, so that the actor network can achieve the desired goal while maintaining its exploratory ability.

[0108] To implement the above technical solution, the training process of the dual-coupled agent network includes:

[0109] A distributed training architecture is adopted, with 2n processes executing the training process in parallel. The agent that interacts with the environment to collect training data is determined according to the process number. Even-numbered processes use exploratory agents, and odd-numbered processes use exploit agents. The exploratory agents and exploit agents share the same sample data in the same experience playback area.

[0110] In both the exploiting agent and the exploring agent, the critic network of the main network constructs the critic loss function based on the temporal difference TD objective in each training iteration and updates the parameters through gradient descent.

[0111] The actor network of the main network adopts a delayed update mechanism: whenever the critic network of the main network completes a certain number of preset updates, the parameters of the actor network of the main network are updated once according to the deterministic policy gradient theorem, and at the same time, the actor network and critic network in the target network are simultaneously soft-updated according to the Polyak average.

[0112] The forward dynamics model FD uses the mean squared error of state prediction as the loss function and uses gradient descent for updating.

[0113] In the coupled network, only the actor network is updated, and its parameters are updated based on the weighted average Q-value of its own output according to the deterministic policy gradient theorem.

[0114] It should be noted that:

[0115] To facilitate effective collaboration and parallel training between two agents, this invention employs a distributed training architecture based on the Message Passing Interface (MPI) standard. A schematic diagram of this architecture is shown in Figure 2. By utilizing multi-process parallel data sampling and gradient aggregation, this framework improves data collection throughput and training efficiency.

[0116] The architecture consists of 2n parallel MPI processes, each maintaining an independent experience replay buffer. In this setup, odd-numbered processes choose to interact with the environment using the agent's actor network; even-numbered processes choose to interact with the environment using a coupled network of actor networks. To encourage diverse exploration, parameter-space noise is applied to the actor networks interacting with the environment. Agents within the same process share the same transformation data in the same experience replay buffer.

[0117] Each process samples transformed data from its own experience replay buffer and computes local gradients. Through MPI, the gradients of all processes are aggregated and averaged. This represents the gradient of the network in the i-th process; subsequently, the global gradient... Broadcast to all processes to synchronize parameters;

[0118] As shown in Figure 2, the transformation data used to train the critic network is sampled from the interactions between the coupled actor network and the environment, rather than from the actor network utilizing the agent. This design allows the agent to receive gradient signals from two different transformation distributions. Gradients from odd-numbered processes drive the agent to achieve the desired goal, while gradients from even-numbered processes introduce exploration motivation, ultimately forming a strategy that can both complete the task and maintain exploration capabilities. As training progresses, the agent's familiarity with environmental dynamics continuously improves. Simultaneously, the relative influence of intrinsic motivation gradually weakens. This leads to a natural transition in the agent's behavior, becoming increasingly dominated by external rewards, and the strategy gradually refocuses on goal achievement.

[0119] To implement the above technical solution, the training process of the dual-coupled agent network includes:

[0120] (1) After the first actor network or the fifth actor network interacts with the environment for one round, the experience data is stored in the experience replay area;

[0121] (2) Call the DSHER algorithm to enhance the small batch of data randomly sampled from the experience replay area to obtain the small batch of experience data B after DSHER enhancement, and update the parameters of the dual-coupled agent network based on the small batch of experience data B;

[0122] (3) Use the first actor network to output the action, without using any noise to interact with the environment for one round;

[0123] The training cycle is defined as: executing step (1) p times and then executing step (2) q times, where q is greater than p;

[0124] The training epoch is defined as follows: after completing m cycles, step (3) is executed n times and the success rate is recorded, where n is greater than m;

[0125] Repeat the process f times until the recorded success rate converges.

[0126] To implement the above technical solution, the specific enhancements to the DSHER algorithm include:

[0127] Multiple groups were randomly sampled from the experience replay area. , , , As a batch, obtain the corresponding success rate. Adaptive HER probability calculate:

[0128] ;

[0129] ;

[0130] ;

[0131] In the formula, As the attenuation factor, To adjust the dynamic coefficients, k is the number of augmented samples, a is the scaling factor, and controls the exponential term. The overall magnitude determines the maximum value of the decay factor. When the success rate suc_rate is low, the decay factor is close to a+c, which has an amplifying effect. b is the decay rate, which controls the sensitivity of the exponential function to the success rate. The larger the value of b, the more drastic the impact of the success rate change on the decay factor, and the faster the system response. c is the offset, which provides a lower limit for the decay factor, ensuring that even if the success rate is very high, the decay factor will not drop to zero, maintaining the minimum exploration activity of the system.

[0132] In a batch, iterate through each sub-target in the target g corresponding to the sampled sample, and generate a random number between (0, 1). If this random number is less than the currently calculated value... If the target is not found, then replace the current sub-target with any sub-target that has been achieved after time step t; otherwise, leave the atomic target unchanged.

[0133] When a sub-objective is replaced, the replaced sub-objective is reused to calculate the external reward.

[0134] The invention will be further illustrated below through specific examples:

[0135] This experiment focuses on multi-block robot grasping tasks in the MultiGoal-FetchEnv simulation environment. The task types include three categories: PickAndPlace, Push, and Stack, involving 1-4 blocks.

[0136] Environments are combined based on three dimensions: task type, number of blocks, and reward format. Task types include PickAndPlace, Stack, and Push; the number of blocks N ranges from 1 to 4 (with Stack having a minimum of 2); reward formats include incremental, sparse, and dense. Environments are referred to in the following text using the abbreviation "Task Name + Quantity + Reward," for example, Stack3Sparse represents a 3-block stacking task with a sparse reward. The maximum number of turns for each environment increases linearly with the number of blocks, and in this embodiment, it is set to 50 × N.

[0137] The action space has a dimension of 4 and is represented by the action vector a = [a_x, a_y, a_z, a_g]. Here, a_x, a_y, and a_z ∈ [-1, 1], representing the displacement components in the x, y, and z directions, corresponding to the displacement direction and magnitude of the end effector in the global coordinate system. The internal scaling is 0.05. This scaling is related to the fact that each action is repeatedly applied within 20 physical substeps, used to avoid excessive cumulative displacement in a single step and maintain control stability. a_g is the gripper component, controlling the symmetrical opening and closing of the gripper; positive values ​​tend to open, and negative values ​​tend to close. During the Push task, when the gripper is locked, this component is ineffective.

[0138] Attitude control: The end effector attitude remains constant and no attitude degrees of freedom are provided.

[0139] Control frequency: The single-step control frequency is approximately 25Hz, meaning approximately 25 decisions are made within 1 second. This frequency is determined by the physical time step dt=0.002s and the number of substeps n_substeps=20. Each action is repeated over 20 substeps, forming an actual control cycle of 0.002×20=0.04s.

[0140] For the observation space, its observation structure is: observation, achieved_goal, and desired_goal. The observation includes the gripper's state information (position, velocity, finger opening / closing, etc., a total of 10 dimensions) and the state of each cube (position, relative displacement, attitude, linear velocity, and angular velocity, etc., a total of 15 dimensions), therefore the observation dimension is 10 + 15 × N. achieved_goal and desired_goal represent the current position and target position of all cubes, respectively, each with a dimension of 3 × N.

[0141] Target positions are randomly generated at the start of each round, typically sampled within a certain horizontal range around the initial position of the gripper, with the height based on the tabletop settings. For Stack tasks, target positions overlap on the plane and are stacked layer by layer at fixed intervals along the height direction, forming a stable tower-like structure. For non-Stack tasks, targets are randomly distributed on the plane, maintaining a certain interval to prevent task degradation. Notably, the PickAndPlace task allows a target to appear in the air to train grasping and lifting capabilities; the target for the Push task is always located on the tabletop plane.

[0142] Regarding the definition of rewards, in this embodiment, incremental rewards are calculated by determining whether each block has reached the target and accumulating the scores of each block; sparse rewards are given as zero only when all blocks have reached the target, otherwise -1; dense rewards are calculated as the sum of the negative distances from each block to its target position when the target has not been fully reached, and the reward is zero when the target has been fully reached.

[0143] For the initial state, at the start of the round, the robot's base rail joints are fixed at the initial position (0.405, 0.48, 0), and the base remains stationary in global coordinates. The gripper is moved to a standard initial pose with a pose quaternion of (1, 0, 1, 0), the position of which is determined internally by the simulation based on the initial joint and gripper height parameters, located in the work area above the tabletop. The initial height of the cube is fixed above the tabletop (approximately 0.425m), and its horizontal position is sampled by adding a uniform offset of [-0.15, 0.15] to the gripper's planar position, requiring a minimum planar distance (approximately 0.1m) from the gripper and other cubes. In the Stack task, the initial cubes are not pre-placed in the stack position but are randomly scattered from the tabletop. The target position is sampled near the initial gripper position, with a horizontal offset range of approximately [-0.15, 0.15]. The default height is the same as the initial height of the block; Stack targets are stacked at 0.05m intervals along the height direction and are not lifted into the air; PickAndPlace allows a target to be lifted into the air; Push targets are always located on the table plane.

[0144] The main content of each task includes: PickAndPlace: No stacked targets, emphasizing grasping and positioning, which may include aerial targets. Stack: Targets are always stacked, used to evaluate the upper limit of stacking capabilities. Push: Gripper locked, only pushing is allowed, the target is within the table plane.

[0145] In the experiment, the initialization and reset process for each training or evaluation round is the starting point of the task loop and is crucial for ensuring the diversity and generalization of learning. This process is executed strictly according to a fixed logical sequence. Specifically, it includes:

[0146] The environment first randomly generates the target location and arranges the blocks in an initial state, then returns the initial observations and target information to the agent. The agent calculates and outputs actions based on the observations. The actions are interpreted as end effector displacement and gripper opening and closing commands, and are applied to the simulation in several physical substeps.

[0147] After the environment performs an action, it updates the pose of the robotic arm and the block, calculates the new observation and achieved_goal, and provides an immediate reward based on the distance between the block and the target. This process loops within a round until the maximum number of moves is reached and the task is truncated or continues.

[0148] For Stack, the agent must first grab and align the blocks, and then place them layer by layer to meet the stacking goal; for Push, the agent can only change the position of the blocks by pushing.

[0149] At the end of the round, the environment can be reset to generate a new target and initial state, and then proceed to the next round.

[0150] All networks in the experiment were designed with three fully connected hidden layers, each containing 256 ReLU neurons. The output layer of the actor network used the tanh activation function.

[0151] Discount factor The value is determined by the number of time steps T in one round, i.e. ; Calculate the scaling factor for internal rewards The weight parameter for the Q-value weighted average unit is 0.5, 0.5; the weight parameter for the Polyak average is... It is 0.001 in the exploit agent and 0.05 in the explore agent.

[0152] In the training, this embodiment uses a size of 10. 6 An experience replay buffer is used, from which minibatch size 4096 is uniformly sampled. A training cycle consists of 10 rollouts and 40 optimization steps. For process i, the exploit agent is evaluated every 50 cycles, with 100 evaluation rounds. The evaluation metric is calculated by averaging the success rates across all processes.

[0153] Given that the stack-n task is more complex than other fetch tasks, this embodiment employs curriculum learning to assist training. This process is divided into multiple stages, each designed to help the agent master prerequisite stacking skills. Specifically, for the stack-n task, this experiment uses a pre-trained pick & place-n model to initialize the agent, based on the rationale that accurate block picking and placement skills are essential before learning stacking.

[0154] Subsequently, the basic agent is trained in a curriculum environment, gradually learning to handle partially stacked structures. Once the agent reaches a success rate threshold in the curriculum environment—between 90% and 95% depending on the number of blocks—it enters the final test environment. In this environment, all blocks start from an unstacked state on the table. Experimental results show that the D2-HER algorithm struggles to successfully learn stacking behavior without curriculum learning. Therefore, all reported results for the stack-n task were obtained using a curriculum-based training method. This experiment evaluated the performance of D2-HER on 11 multi-sub-objective tasks of varying complexity. Table 1 summarizes the comparison results of the D2-HER algorithm of this invention and the original TD3+HER algorithm in all environments. The corresponding average success rate learning curves are shown in Figure 3, where Figures 3(a) to (d) represent the curves corresponding to push1 to push4, Figures 3(e) to (h) represent the curves corresponding to PickAndPlace1 to PickAndPlace4, and Figures 3(i) to (k) represent the curves corresponding to stack2 to stack4.

[0155] Experimental results demonstrate that D2-HER achieves significant improvements across the entire task set. As shown in Table 1, D2-HER equipped with a forward dynamics model achieves a success rate exceeding 90% across all tasks. Notably, it achieves a perfect 100% success rate on several tasks, including push1, push2, pick&place1, pick&place2, and stack2. These results highlight the effectiveness and robustness of the proposed method.

[0156] It's important to note that for single-block fetch tasks, such as push1 and pick&place1, the TD3+HER algorithm shows an average success rate close to zero. This result reflects the limitation of the number of training epochs. As the number of training epochs increases, the agent can learn effective strategies to complete these tasks. However, for multi-block manipulation tasks, the baseline algorithm faces inherent complexity, and simply increasing the number of training epochs does not bring significant improvement.

[0157] Table 1

[0158] ;

[0159] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for implementing multi-sub-target tasks based on a dual-coupled intelligent agent network, used for multi-block robot grasping tasks, the task types including picking up and placing, pushing and stacking of 1-4 blocks, characterized in that, Includes the following steps: The current environment state and the desired objective are obtained in real time and then input into the trained dual-coupled agent network to obtain the corresponding multi-sub-objective task implementation strategy. The dual-coupled agent network comprises an exploiting agent and an exploratory agent. The exploiting agent consists of: a main network A acquiring corresponding actions, their evaluation values, and external rewards based on the current environmental state and the objective, and storing these in an experience replay area; and a target network A optimizing the main network A based on the evaluation values. The exploratory agent consists of: a main network B randomly acquiring samples from the experience replay area, predicting the next state of the sample state using the forward dynamics model (FD), and calculating internal rewards; a target network B optimizing the main network B based on the predicted state and internal rewards; and the coupled network generating actions and external rewards for interacting with the environment based on the current environmental state and the objective. The external rewards are used to update the exploiting agent, which optimizes the main network B based on the internal rewards. The training process of the dual-coupled agent network includes: a distributed training architecture with 2n processes executing the training process in parallel. Agents interacting with the environment to collect training data are trained according to their process numbers. It is determined that even-numbered processes use exploratory agents, while odd-numbered processes use exploit agents. Both exploratory and exploit agents share sample data from the same experience replay region. In both the exploratory and exploit agents, the main network's critic network constructs a critic loss function based on the temporal difference (TD) objective in each training iteration and updates its parameters via gradient descent. The main network's actor network employs a delayed update mechanism: after the main network's critic network completes a predetermined number of updates, the actor network parameters are updated once according to the deterministic policy gradient theorem, while the actor and critic networks in the target network are simultaneously soft-updated using Polyak averaging. The forward dynamics model (FD) uses the mean squared error of state prediction as its loss function and updates it using gradient descent. In the coupled network, only the actor network is updated, with its parameters updated based on the weighted average Q-value of its output according to the deterministic policy gradient theorem.

2. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 1, characterized in that, Main network A includes: the first actor network actor1, used to obtain (from the environment) ， ),in, For time step The following state, This represents the expected objective of achieving the current multi-sub-objective task, based on ( ， Output the corresponding action The first noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as Critics networks C1 and C2 are dual-network frameworks using TD3, and are independently based on ( ， ， Estimate the state-action-goal value and output Q1( ， ， ) and Q2( ， ， External reward unit, used to calculate time steps. External reward corresponding to the next action and output external rewards To target network A.

3. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 2, characterized in that, Target network A includes: a second actor network, actor2, used to obtain execution... Next step +1 Environment Status ,according to( ， Generate target action The second noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as ; Critics network C'1 and critic network C'2 are dual-network frameworks using TD3, and are independently based on ( ， ， Estimate the state-action-goal value and output Q'1( ， ， ) and Q'2 ， ， The first TD target acquisition unit is used to acquire targets based on Q'1 ( ， ， ) and Q'2 ， ， ) and external rewards Calculate the target value y of TD, where ， ， Discount factor; storage unit, used to... Stored in the experience replay area, where and They represent and and The state after splicing.

4. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 3, characterized in that, The main network B includes: a third actor network (actor3), used to update the target network B via soft updates; and a forward dynamics model (FD), used to randomly obtain arbitrary samples from the experience replay area. ， ), to obtain the next time step Predicted state under +1 According to the sample The corresponding next time step With predicted state The error between them multiplied by the scaling factor Receive internal rewards Critics networks C3 and C4 are dual-network frameworks using TD3, and are independently based on ( ， Based on internal rewards Estimate the state-action value and output Q1( ， ) and Q2( ， )。 5. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 4, characterized in that, Target network B includes: a fourth actor network (actor4), used to acquire samples. The corresponding next time step Output the corresponding action The third noise-adding unit is used for... Add noise To obtain the noise-added action , recorded as Critics Network C'3 and Critics Network C'4 are dual-network frameworks using TD3, and are independently based on ( ， Estimate the state-action value and output Q'1( ， ) and Q'2 ， The second TD target acquisition unit is used to acquire targets based on Q'1 ( ， ) and Q'2 ， ) and internal rewards Calculate the target value y of TD, where ， 。 6. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 4, characterized in that, The coupled network includes: actor5, the fifth actor network, used to obtain the environment state. and their corresponding objectives Output the corresponding action The fourth noise-adding unit is used for... Add noise To obtain the noise-added action + , recorded as External reward unit, used for calculation Corresponding external rewards Internal commentator network C in Reuse the critic network C3 for use based on ( ， Estimate and evaluate the value of actions in exploration, and output the results. ( ， External commentator network C ex Reuse the critic network C1 for use based on ( ， ， Evaluate the value of actions in task completion and output... ( ， ， Q-value weighted average unit, used to calculate based on... ( ， )and ( ， ， The weighted average is obtained ( ， ， )。 7. The method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 1, characterized in that, The training process of the dual-coupled agent network includes: (1) after the first actor network or the fifth actor network interacts with the environment for one round, the experience data is stored in the experience playback area; (2) the DSHER algorithm is called to enhance the small batch of data randomly sampled from the experience playback area, and the small batch of experience data B with DSHER enhancement is obtained. The parameters of the dual-coupled agent network are updated according to the small batch of experience data B; (3) the first actor network outputs the action and interacts with the environment for one round without using any noise; where the training cycle is defined as: after executing step (1) p times, step (2) is executed q times, where q is greater than p; the training epoch is defined as: after executing m cycles, step (3) is executed n times and the success rate is recorded, where n is greater than m; the epoch is executed f times until the recorded success rate converges.

8. A method for implementing multi-sub-objective tasks based on a dual-coupled agent network according to claim 7, characterized in that, The specific enhancements to the DSHER algorithm include: randomly sampling multiple groups from the experience replay area ( ， ， ， As a batch, obtain the corresponding success rate. Adaptive HER probability calculate: ； ； In the formula, As the attenuation factor, To adjust the dynamic coefficients, k is the number of augmented samples, a is the scaling factor, and controls the exponential term. The overall magnitude determines the maximum value of the decay factor. When the success rate (suc_rate) is low, the decay factor is close to a+c, which amplifies the effect. b is the decay rate, which controls the sensitivity of the exponential function to the success rate. The larger the value of b, the more drastic the impact of success rate changes on the decay factor, and the faster the system response. c is the offset, which provides a lower limit for the decay factor, ensuring that even if the success rate is high, the decay factor will not drop to zero, maintaining the minimum exploration activity of the system. In a batch, the targets corresponding to the sampled samples are traversed. For each sub-target, generate a random number between (0, 1). If this random number is less than the currently calculated number... If the sub-goal is replaced by any sub-goal that has been achieved after time step t, then the current sub-goal is replaced; otherwise, the atomic goal remains unchanged. After a sub-goal is replaced, the replaced sub-goal is reused to calculate the external reward.

Images