Double-arm collaborative planning method, system and device based on reinforcement learning and medium

Abstract

The invention discloses a double-arm collaborative planning method, system and device based on reinforcement learning and a medium, and belongs to the technical field of mechanical arm control. According to the current state in the state space, a control action is generated, and three-dimensional displacement increment instructions of the left arm end effector and the right arm end effector are obtained; after the three-dimensional displacement increment instruction is responded to and double-arm cooperative control is executed, a mixed reward function is calculated; experience enhancement processing is carried out on the execution track, target resetting is carried out on the failure track, pseudo target experience is generated, and the original experience and the pseudo target experience are stored in a playback buffer; and updating parameters of the strategy network and the Q value network according to the empirical samples in the playback buffer, and completing optimization of the double-arm collaborative trajectory planning strategy. According to the method, the problems of sparse reward and local optimum in two-arm collaborative planning are effectively solved by fusing maximum entropy reinforcement learning and an experience playback mechanism, and the training efficiency and the strategy generalization ability are improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a dual-arm cooperative planning method, system, device, and medium based on reinforcement learning. Background Technology

[0002] Robotic arm trajectory planning typically employs sensor-based artificial potential field methods and sampling-based algorithms (such as probabilistic roadmaps and fast stochastic expansion trees). However, these methods may struggle with local minima and face challenges in achieving optimal convergence. They may also lack intelligence, exhibiting poor dynamic programming capabilities and a lack of self-learning ability, thus limiting their applicability in unstructured environments and hindering their ability to empirically augment execution trajectories.

[0003] Currently, dual-arm robots are widely used in various fields such as industry, service, and healthcare, and their path planning technology is key to achieving collaborative operations. However, traditional path planning methods face many challenges in terms of efficiency and accuracy in multi-dimensional spaces, especially in dynamic environments where they are prone to getting trapped in local optima and struggling to converge. Although deep reinforcement learning (DRL) has been applied to the trajectory control of single-arm robots in recent years, problems still exist in dual-arm collaborative planning tasks, such as the difficulty in unifying obstacle avoidance and collaboration, and the current reward function design being too simplistic, unable to simultaneously guide the robot to complete the task and avoid obstacles. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is: how to achieve adaptive cooperative trajectory planning, obstacle avoidance accuracy improvement, and efficient learning in sparse reward environments for dual-arm robots by constructing Markov decision models and using maximum entropy reinforcement learning algorithms, while avoiding getting trapped in local optima and improving policy generalization ability.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a dual-arm cooperative planning method based on reinforcement learning, comprising: constructing a Markov decision model; wherein the Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increment of the end effector space; generating control actions based on a policy network using the Soft Actor-Critic algorithm according to the current state in the state space, obtaining three-dimensional displacement increment commands for the left and right arm end effectors; executing dual-arm cooperative control in response to the three-dimensional displacement increment commands, and calculating a hybrid reward function; wherein the hybrid reward function integrates attraction and repulsion rewards based on an artificial potential field and guidance rewards based on end effector posture, outputting an instantaneous reward value; performing experience enhancement processing on the execution trajectory, resetting the target of failed trajectories through the Hindsight Experience Replay mechanism, generating pseudo-target experience, and storing the original experience and the pseudo-target experience in a replay cache; updating the policy network and Q-value network parameters according to the experience samples in the replay cache, completing the update of the dual-arm cooperative trajectory planning strategy.

[0007] As a preferred embodiment of the dual-arm cooperative planning method based on reinforcement learning described in this invention, the step of constructing the state space includes: obtaining the three-dimensional position vectors and velocity vectors of the left and right arm end effectors; obtaining the relative position and relative velocity of the target object relative to the left and right arm end effectors; obtaining the relative position and relative velocity of the obstacle relative to the left and right arm end effectors; and normalizing the three-dimensional position vectors, the velocity vectors, the relative positions, and the relative velocities to obtain the state space.

[0008] As a preferred embodiment of the dual-arm collaborative planning method based on reinforcement learning described in this invention, the generation of control actions by the policy network based on the Soft Actor-Critic algorithm includes: inputting the current state into the policy network; the policy network being the Actor component in the Soft Actor-Critic algorithm; outputting action distribution parameters through the policy network; wherein the action distribution parameters include a mean vector and a standard deviation vector; performing random sampling based on the action distribution parameters to generate continuous action vectors; and applying amplitude constraints to the continuous action vectors to obtain the three-dimensional displacement increment commands for the left arm end effector and the right arm end effector.

[0009] As a preferred embodiment of the dual-arm collaborative planning method based on reinforcement learning described in this invention, the following are included: the attraction and repulsion reward based on the artificial potential field includes: a target attraction term, constructed based on the distance between the end effector and the target object; the value of the target attraction term increases as the distance between the end effector and the target object decreases; an obstacle repulsion term, constructed based on the distance between the end effector and the obstacle, the value of the obstacle repulsion term increases as the distance between the end effector and the obstacle decreases when the distance between the end effector and the obstacle is less than a safety threshold; the guidance reward based on the end effector posture includes: a position guidance term, determined based on the straight-line distance between the end effector and the target, taking a positive value when the grasping threshold is reached; and a direction guidance term, determined by calculating the angle between the end effector's movement direction and the direction of the line connecting the target, the value of the direction guidance term being negatively correlated with the angle.

[0010] As a preferred embodiment of the dual-arm collaborative planning method based on reinforcement learning described in this invention, the step of resetting the target of the failed trajectory through the Hindsight Experience Replay mechanism includes: identifying the failed trajectory that did not reach the original target; selecting a state from the failed trajectory as a pseudo-target according to a preset sampling strategy; wherein the preset sampling strategy includes a future strategy for selecting future states in the trajectory and a final strategy for selecting the end state of the trajectory; recalculating the immediate reward value of each time step in the failed trajectory based on the pseudo-target; and combining the pseudo-target and the immediate reward value recalculated based on the pseudo-target with the state and action information of the failed trajectory to store the pseudo-target experience in the replay cache.

[0011] As a preferred embodiment of the dual-arm collaborative planning method based on reinforcement learning described in this invention, updating the parameters of the policy network and the Q-value network includes: constructing a complete Soft Actor-Critic algorithm network architecture; wherein the network architecture includes the policy network, the dual Q-value network, and the target network; sampling a batch of experience samples from the replay buffer; updating the parameters of the dual Q-value network by minimizing the temporal difference error; updating the parameters of the policy network by maximizing the sum of the expected reward and the policy entropy; and synchronizing the parameters of the target network using a soft update mechanism.

[0012] As a preferred embodiment of the dual-arm cooperative planning method based on reinforcement learning described in this invention, the control of the dual-arm robot by the three-dimensional displacement increment command includes: restricting the three-dimensional displacement increment command within a set boundary range; converting the three-dimensional displacement increment command into joint angle control commands through an inverse kinematics solver; wherein the inverse kinematics solver completes the mapping from the end-effector space to the joint space based on the kinematic model of the dual-arm robot; and driving the dual-arm robot to perform cooperative actions according to the joint angle control commands.

[0013] This invention provides a dual-arm collaborative planning system based on reinforcement learning.

[0014] To address the aforementioned technical problems, this invention further provides the following technical solution: a dual-arm cooperative planning system based on reinforcement learning, comprising: a model building module for constructing a Markov decision model; the Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increment of the end effector space; a policy generation module for generating control actions based on the current state in the state space and a policy network using the SoftActor-Critic algorithm, thereby obtaining three-dimensional displacement increment commands for the left and right arm end effectors; a reward calculation module for calculating a hybrid reward function after executing dual-arm cooperative control in response to the three-dimensional displacement increment commands; an experience enhancement module for resetting the target of failed trajectories through the Hindsight Experience Replay mechanism, generating pseudo-target experience, and storing the original experience and the pseudo-target experience in a replay cache; and a parameter optimization module for updating the parameters of the policy network and the Q-value network based on the experience samples in the replay cache, thereby updating the dual-arm cooperative trajectory planning strategy.

[0015] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the reinforcement learning-based dual-arm cooperative planning method.

[0016] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the reinforcement learning-based dual-arm cooperative planning method.

[0017] The beneficial effects of this invention are as follows: By constructing a low-dimensional state space based on end-effector information to replace the traditional high-dimensional joint space, this invention reduces computational complexity and improves convergence speed; by adopting the maximum entropy framework of the Soft Actor-Critic algorithm, it achieves stable policy optimization while maintaining policy exploration, effectively avoiding the problem of traditional methods easily getting trapped in local optima; the hybrid reward function of this invention organically integrates the artificial potential field mechanism and the posture guidance mechanism, achieving unified optimization of obstacle avoidance safety and trajectory smoothness, solving the technical problem that existing single reward functions cannot take into account multi-objective optimization; the introduction of the Hindsight Experience Replay mechanism resets the target and reconstructs the experience for failed trajectories, transforming invalid samples in sparse reward environments into effective learning experience, improving sample utilization efficiency and training speed; the overall solution realizes adaptive collaborative planning of dual-arm robots in various environments, overcoming the fundamental defects of traditional methods lacking intelligence and self-learning ability, and providing an efficient and reliable solution for dual-arm collaborative operations in unstructured environments. In summary, this invention employs the Soft Actor-Critic (SAC) algorithm combined with the Hindsight Experience Replay (HER) algorithm and designs a hybrid reward function, thereby improving training efficiency and policy effectiveness, and enhancing the overall performance of trajectory planning for dual-arm robots. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The above is a flowchart of a dual-arm collaborative planning method based on reinforcement learning, provided as an embodiment of the present invention.

[0020] Figure 2 This invention provides a flowchart of a dual-arm cooperative path planning method based on the Soft Actor-Critic algorithm, which is a dual-arm cooperative planning method based on reinforcement learning, as an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0022] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a two-arm cooperative planning method based on reinforcement learning, including: S100: Construct a Markov decision model. The Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increments of the end effector space. S200: Based on the current state in the state space, generate control actions using a policy network based on the maximum entropy policy gradient algorithm, namely the SoftActor-Critic algorithm (i.e., the SAC algorithm, which solves the problem of reinforcement learning in discrete action space and continuous action space), and obtain the three-dimensional displacement increment commands for the left and right arm end effectors. S300: After executing dual-arm cooperative control in response to the three-dimensional displacement increment command, calculate the hybrid reward function. The hybrid reward function integrates the attraction / repulsion reward based on the artificial potential field and the guidance reward based on the end-effector posture, outputting an instantaneous reward value. S400: Performs experience enhancement processing on the execution trajectory. Through the Hindsight Experience Replay mechanism (Hindsight Experience Replay is a reinforcement learning technique designed to solve the sparse reward problem, suitable for goal-oriented tasks such as robot control and object grasping), it resets the target of the failed trajectory, generates pseudo-target experience, and stores the original experience and pseudo-target experience in the replay cache. S500: Based on the experience samples in the playback cache, update the parameters of the policy network and Q-value network to complete the update of the dual-arm cooperative trajectory planning strategy.

[0023] It should be noted that dual-arm robots are increasingly widely used in industrial manufacturing, service robots, and medical assistance, and their collaborative operation capability is key to achieving relatively complex tasks. In actual operation, the two arms need to move collaboratively in a three-dimensional space with obstacles, while avoiding collisions with each other and with the environment. This presents multiple challenges to trajectory planning, including high-dimensional state space, dynamic environment adaptation, and real-time requirements. Traditional path planning methods based on sampling or optimization are prone to getting trapped in local minima when facing dynamic environments and lack adaptive learning capabilities. Existing reinforcement learning methods suffer from problems such as sparse rewards, low training efficiency, and difficulty in simultaneously considering task completion and obstacle avoidance in dual-arm collaborative scenarios, resulting in slow policy convergence and insufficient generalization ability.

[0024] Therefore, to address the aforementioned issues of collaborative planning and learning efficiency, a low-dimensional Markov decision model based on end-effector information is constructed through steps S100 to S500 to reduce state space complexity; a maximum entropy reinforcement learning framework is adopted to maintain policy exploration and avoid premature convergence to local optima; a hybrid reward function integrating artificial potential field and posture guidance is designed to achieve unified optimization of task objectives and safety constraints; an experience replay mechanism is introduced to transform failure experiences into effective learning samples, improving training efficiency in sparse reward environments; and finally, adaptive collaborative trajectory planning for the dual-arm robot in various environments is achieved.

[0025] Example 2, refer to Figure 1 and Figure 2 This is the second embodiment of the present invention, which provides a dual-arm collaborative planning method based on reinforcement learning.

[0026] S100: Construct a Markov decision model. The Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increments of the end effector space.

[0027] Specifically, the state space construction steps in step S100 include steps A1 to A4: Step A1: Obtain the three-dimensional position and velocity vectors of the left and right arm end effectors; Step A2: Obtain the relative position and relative velocity of the target object with respect to the left and right end effectors; Step A3: Obtain the relative position and relative velocity of the obstacle with respect to the left and right end effectors; Step A4: Normalize the three-dimensional position vector, velocity vector, relative position, and relative velocity to obtain the state space.

[0028] It should be noted that, in order to achieve efficient cooperative trajectory planning for dual-arm robots in various dynamic environments, this invention is first formalized as a Markov Decision Process (MDP), which uses a quintuple... It means that among them For state space, For the action space, Let be the state transition probability function. For the reward function, This is the discount factor.

[0029] In terms of state space design, this invention abandons the traditional method of using high-dimensional redundant joint angles, and instead constructs a state space based on end-effector information and task-related geometric relationships, improving task orientation and model convergence efficiency. The state space includes the positions (represented as three-dimensional position vectors) and velocity vectors of the left and right end-effectors, as well as the relative positions and velocities of the target object and obstacles relative to each end-effector, specifically including: (1) Among them, subscript e This refers to the end-effector. t Indicates the target. o The subscript indicates an obstacle, with subscript 1 indicating its relation to the left arm and subscript 2 indicating its relation to the right arm. For example, the subscript... e 1. e 2 represents the left arm end effector and the right arm end effector, respectively; These represent the three-dimensional position vectors of the left and right arm end effectors, respectively. This represents the velocity vectors corresponding to the left and right arm end effectors. This is the relative position vector of the target object with respect to the left and right end effectors; This represents the relative position vector of the obstacle with respect to the left and right end effectors; all state variables have been normalized, and the overall state dimension is 24-dimensional.

[0030] In terms of motion space design, this invention uses three-dimensional incremental displacement of the end-effector space as the motion command, i.e., motion space. Defined as: (2) in, Left ( i =1) and right ( i =2) The end of the robotic arm is at x ,y , z The incremental movement in the axial direction.

[0031] It is important to note that the motion space design of this invention avoids relatively complex posture or joint angle control, thereby effectively improving the continuity and executability of motion expression. Furthermore, to ensure motion stability, displacement in each direction is limited to the range of [-0.1, 0.1] meters to prevent excessive single-step movements from causing path oscillations.

[0032] Through the above-described state and action space construction, this invention provides a clear and efficient modeling foundation for subsequent policy training based on deep reinforcement learning, and can adapt to the needs of high-dimensional collaborative control and has good practical deployment capabilities.

[0033] S200: Based on the current state in the state space, a policy network based on the Soft Actor-Critic algorithm generates control actions, resulting in three-dimensional displacement increment commands for the left and right end effectors.

[0034] Specifically, in step S200, the policy network based on the Soft Actor-Critic algorithm generates control actions, including steps B1 to B4: Step B1: Input the current state into the policy network; the policy network is the Actor component in the Soft Actor-Critic algorithm; Step B2: Output action distribution parameters through the policy network; whereby the action distribution parameters include the mean vector and the standard deviation vector; Step B3: Randomly sample based on action distribution parameters to generate continuous action vectors; Step B4: Apply amplitude constraints to the continuous motion vector to obtain the three-dimensional displacement increment commands for the left and right end effectors.

[0035] It should be noted that, in this invention, to achieve stable learning and effective trajectory optimization of the dual-arm robot in a continuous high-dimensional action space, the maximum entropy policy gradient algorithm—Soft Actor-Critic (SAC)—is used as the core reinforcement learning framework. This algorithm introduces a policy entropy term into the traditional Actor-Critic structure, maximizing the sum of reward and policy entropy to improve the policy's exploration capability and training stability.

[0036] Specifically, the SAC network structure mainly consists of the following parts: 1. Policy Network (Actor): Used to determine policy based on time steps. t status Generate corresponding actions The network outputs the probability distribution parameters (mean and variance) of actions, which are sampled using a random reparameterization technique to obtain continuous control commands. The action generation method is represented as follows: (3) in, Indicates time step t The state; Indicates time step t The action; and These are the mean and standard deviation of the policy network output, respectively. It is a random variable used in the reparameterization process; It follows a standard normal distribution; express Follows a standard normal distribution .

[0037] 2. Two independent Q-networks, also called Q-value networks (Critic): denoted as... and This is used to evaluate the long-term reward of the current state-action pair, and a dual-network structure can alleviate the problem of Q-value overestimation. In the SAC algorithm, the two Q-value networks are parameterized as follows: , , and Specifically Representative Policy Network This determines the probability of the action; and They are used in state-value networks and target-value networks, respectively, to estimate state values ​​and provide stable target values; These are the parameters of the Q-value network. The loss function of the Q-value network is defined as the mean squared error between the predicted Q-value and the target Q-value. (4) in, For parameters The target state value function is calculated by the target network; It is a reward discount factor; Represents the empirical tuples sampled from the empirical playback buffer. The expectations, among which This is the current state. In order to perform the action, For immediate reward, s' represents the next state; The radius of motion of the robotic arm; For parameters Action value function; Based on parameters The objective function is specifically the first... i indivual Q Value network ( i The objective function is ∈{1,2}).

[0038] 3. Target Q-Network and Target Policy Networks: These are used to delay updating target values, improving training stability. Their parameters are updated using a "soft update" method. (5) in, It is the target value; This is the soft update coefficient, typically set to 0.005.

[0039] 4. State-valued function network: used to directly estimate the expected value of a state. .

[0040] To optimize the policy network, this invention maximizes the objective function that includes the entropy term as follows: (6) in, Based on parameters The objective function; It represents the log probability density of the action distribution, encouraging the strategy to remain random and avoid getting trapped in local optima; Let the expectation operator be denoted as , where the time step is . t status From the experience replay buffer Mid-sampling, random variables used in the reparameterization process From the standard normal distribution Mid-sampling; For parameters Q-value network for state-action pairs Value estimation; For time steps t The action.

[0041] Furthermore, to further improve the performance of the policy network, an automatic entropy adjustment mechanism is used during training to adaptively adjust the entropy temperature coefficient based on the target entropy value. Its optimization objective is: (7) in, The preset target entropy value; Based on entropy temperature coefficient The objective function; From The average value of the sampling action in the sample.

[0042] Through the construction and joint training of the above SAC structure, the system can stably converge to a high-quality strategy in a complex state-action space, achieving efficient planning, obstacle avoidance and collaborative control of the dual-arm trajectory.

[0043] Furthermore, in the specific implementation of this invention, after the simulation environment is initialized, the state information of the dual-arm robot is first collected in real time and input into the constructed policy network to generate the current action decision. The state information includes the position and velocity of the end effectors of the left and right robotic arms, as well as their spatial relationships relative to the target object and obstacles, all of which have been normalized to a uniform scale during the training initialization phase. The policy network, as the Actor module in the SAC architecture, outputs a continuous action vector, representing the incremental displacement required by the end effectors of the left and right arms in three-dimensional Cartesian space. To enhance the policy's exploratory ability in the early training phase, Gaussian-distributed noise perturbation is introduced into the output actions, giving each action a certain degree of randomness, thereby avoiding getting trapped in local optima.

[0044] The generated motion vectors are not directly applied to the robot joints, but are instead used as end-effector displacement commands, which are fed into the motion control module and transformed into a sequence of target joint angles by the inverse kinematics solver. In the CoppeliaSim robot simulation platform, the robot model (Baxter) receives this sequence and immediately executes the corresponding action, enabling the robot's end-effector to complete one motion iteration in the task space. Within each control cycle, the robot updates its state based on the current action. The simulation platform simultaneously records the new state and immediate reward, and these, along with the original state, executed action, and current target information, form empirical data, which is stored in the playback buffer. If the action causes the robot to collide with an obstacle or cross the task boundary, a corresponding negative reward feedback is triggered, and that training round is marked as a failed round.

[0045] Furthermore, to ensure the safety and physical feasibility of the robot's movements, boundary constraints are set for the single-step amplitude of the movements in this invention. The displacement of each dimension's end effector is limited to between -0.1 meters and +0.1 meters, thereby ensuring that each step of the robot's movement is stable, the trajectory is smooth, and it is easy to execute on a physical platform. Throughout the training process, the simulation platform continuously repeats the interactive loop of state acquisition, policy generation, action execution, and environmental feedback, thereby driving the policy network to continuously adjust parameters and gradually learn the optimal path planning strategy, realizing the adaptive generation of action sequences such as dual-arm collaboration, obstacle avoidance, and target grasping.

[0046] S300: After responding to the three-dimensional displacement increment command and executing dual-arm cooperative control, it calculates a hybrid reward function. This hybrid reward function integrates attraction and repulsion rewards based on the artificial potential field and guidance rewards based on the end-effector attitude, outputting an instantaneous reward value.

[0047] Specifically, the attraction-repulsion reward based on the artificial potential field includes a target attraction term and an obstacle repulsion term: The target attraction term is constructed based on the distance between the end effector and the target object; the value of the target attraction term increases as the distance between the end effector and the target object decreases. The obstacle repulsion term is constructed based on the distance between the end effector and the obstacle. When the distance between the end effector and the obstacle is less than a safety threshold, the obstacle repulsion term increases. End-effector attitude-based guided rewards include position guidance and orientation guidance: The position guidance item is determined based on the straight-line distance between the end effector and the target. The position guidance item takes a positive value when the grasping threshold is reached. The direction guidance term is determined by calculating the angle between the end-movement direction and the direction of the line connecting the target. The value of the direction guidance term is negatively correlated with the angle.

[0048] It should be noted that the control of the dual-arm robot by the three-dimensional displacement increment command in the motion space includes steps C1 to C3: Step C1: Limit the three-dimensional displacement increment command to the set boundary range; Step C2: Convert the three-dimensional displacement increment command into joint angle control command through the inverse kinematics solver; wherein, the inverse kinematics solver completes the mapping from the end-effector space to the joint space based on the kinematic model of the dual-arm robot; Step C3: Drive the dual-arm robot to perform coordinated movements according to the joint angle control commands.

[0049] Furthermore, in the training process of this invention, in order to effectively guide the dual-arm robot strategy towards the task objective through gradual optimization, while taking into account obstacle avoidance safety and action rationality, a hybrid reward function structure integrating an artificial potential field mechanism and an end-effector posture guidance mechanism is proposed. This reward function not only provides continuous and directional feedback signals, but also alleviates the training convergence difficulty caused by sparse rewards in reinforcement learning.

[0050] Specifically, the hybrid reward function consists of two main components: an attraction-repulsion potential energy reward based on an artificial potential field model, and an attitude reward function considering the relative position and orientation between the end effector and the target. The artificial potential field is designed based on the classical attraction-repulsion model, where the attraction term encourages the end effector to approach the target point. Its functional form employs an improved expression with Huber loss (also known as Smooth L1 Loss, a commonly used loss function in regression tasks, combining squared loss and absolute loss) to enhance the robustness of near-range guidance and suppress far-range oscillations. The repulsion term is constructed based on the Euclidean distance between the robot's end effector and obstacles in the environment, rapidly increasing when the distance is less than a safe threshold to effectively prevent the path from approaching obstacles. The combination of these two components constitutes the artificial potential field sub-reward value. By adjusting the ratio of target attraction to obstacle repulsion, a reasonable path can be guided while maintaining a safe space.

[0051] The posture reward section focuses on guiding the end-effector to smoothly approach the target direction, avoiding unnecessary movements such as detours and swaying. It mainly includes two parts: position and direction. Position reward... Based on the definition of the straight-line distance between the end point and the target, when the grabbing threshold distance (denoted as ) is reached... β A high reward is given if the value is within a certain range, otherwise it is zero; directional reward. By calculating the angle between the current movement direction of the end effector and the direction of the line connecting to the target, and normalizing it to the range of 0 to 1, the smaller the angle, the higher the reward, thus encouraging the end effector to maintain positive movement as it approaches the target. The overall attitude reward consists of the difference between the position and direction terms, ensuring that the strategy not only approaches the target but also maintains a reasonable attitude.

[0052] Finally, this invention weighted and fused the artificial potential field reward and the posture reward to construct a complete hybrid reward function, which takes the following form: (8) in, It is a hybrid reward function; and Hyperparameters set for experience control the relative importance of the two types of rewards during the learning process. In actual training, this hybrid reward function is used to update the loss functions of the policy network and the Q-value network, serving as the primary feedback signal for reinforcement learning. Experimental results show that the strategy using this reward design significantly outperforms the traditional single-reward structure in terms of convergence speed, trajectory smoothness, and task completion rate, and can stably achieve dual-arm collaborative grasping tasks.

[0053] S400: Performs experience enhancement processing on the execution trajectory, resets the target of the failed trajectory through the Hindsight Experience Replay mechanism, generates pseudo-target experience, and stores the original experience and pseudo-target experience in the replay cache.

[0054] Specifically, the Hindsight Experience Replay mechanism is used to reset the target of the failed trajectory, including steps D1 to D4: Step D1: Identify failure trajectories that did not achieve the original objective; Step D2: Select a state from the failed trajectory as a pseudo-target according to a preset sampling strategy; wherein, the preset sampling strategy includes a future strategy for selecting future states in the trajectory and a final strategy for selecting the end state of the trajectory; Step D3: Recalculate the instantaneous reward value at each time step in the failed trajectory based on the pseudo-target; Step D4: Combine the pseudo-target and the instant reward value recalculated based on the pseudo-target with the status and action information of the failure trajectory, and store it as pseudo-target experience in the replay cache.

[0055] It should be noted that, in the reinforcement learning training process of this invention, in order to address the sparse reward problem in the dual-arm trajectory planning task and further improve the utilization efficiency of training samples and the convergence speed of the policy, a Hindsight Experience Replay (HER) mechanism is adopted to enhance and reconstruct the experience data. The core idea of ​​HER is to replace the target of interactive trajectories that failed to complete the task under the original objective with a state actually reached in the trajectory, thereby reconstructing the reward function so that the experience is considered successfully executed in the new context, thus obtaining positive feedback. This mechanism effectively alleviates the problems of weak training gradients and slow policy optimization caused by the scarcity of real successful samples.

[0056] In practice, during each training round, the simulation environment records the complete trajectory sequence generated by the current policy execution, including state, action, actual target, immediate reward, and next state information. For trajectories that fail to achieve the original target, the system selects one or more states from the trajectory as "pseudo-targets" to replace the original target according to certain rules. For example, the final state of the trajectory, a future state, or a random intermediate state can be selected as the new target, and the reward value is recalculated based on the replaced target. For instance, if a trajectory fails to capture the target object, but the end position of one frame is close to the pseudo-target position, the HER treats that frame as a task-achieved state and assigns a positive reward, thus reconstructing a valid experience. The data after pseudo-target replacement and reward recalculation is stored together with the original experience in the replay buffer D for subsequent training of the policy network and value network.

[0057] During the implementation of HER, to avoid distortion of reconstructed experience or deviation of guidance direction from the essence of the task, this invention sets an upper limit k for the number of pseudo-target sampling strategies and selects "future" and "final" target replacement mechanisms as the main strategies to ensure that most of the generated pseudo-experiences have physical feasibility and policy learning value. At the same time, to maintain the diversity of data distribution, some original failed samples are also retained in the experience pool to maintain training stability.

[0058] S500: Based on the experience samples in the playback cache, update the parameters of the policy network and Q-value network to complete the optimization of the dual-arm cooperative trajectory planning strategy.

[0059] Specifically, update the parameters of the policy network and the Q-value network, including steps E1 to E5: Step E1: Construct the complete Soft Actor-Critic algorithm network architecture; the network architecture includes a policy network, a double Q-value network, and a target network; Step E2: Sample a batch of empirical samples from the replay buffer; Step E3: Update the dual-Q value network parameters by minimizing the time difference error; Step E4: Update the policy network parameters by maximizing the sum of expected reward and policy entropy; Step E5: Synchronize the target network parameters using a soft update mechanism.

[0060] It should be noted that in the reinforcement learning-based dual-arm collaborative planning method proposed in this invention, policy learning relies on the continuous optimization process of neural network parameters. To this end, the constructed Soft Actor-Critic (SAC) architecture achieves policy updates by alternately optimizing the policy network and the Q-value network, and combines an experience replay mechanism and entropy regularization objective to improve training stability. In each training round, when the number of samples accumulated in the experience replay buffer D reaches a safe threshold, the system begins the parameter update process.

[0061] First, a fixed batch of empirical samples is randomly sampled from D, including information such as state, action, reward, next state, and target. For updating the Q-value network, the system uses a double-Q structure to mitigate the overestimation bias of the value function, optimizing the parameters of the two Q-networks by minimizing the time difference error (TD error). Specifically, the target Q-value is calculated by the target network, expressed as: (9) in, For parameter-based The objective function; It is a reward discount factor; The radius of motion of the robotic arm; To find the minimum value between the two Q-networks, where This represents the first Q network. Indicates the second Q network; The state at the next time step; For the action of the next time step; For parameters The policy network in state Select action The probability density; It is the natural logarithm; The entropy temperature coefficient is used to control the randomness weights of the strategy. For parameters The target Q network, This represents the target Q-network parameters. It should be noted that... Equation (7) is used consistently throughout the SAC algorithm for adaptive optimization. The value of formula (9) will be optimized. Used for Q network updates.

[0062] The Q network updates its parameters according to the loss function defined in formula (4), and optimizes the parameters of the dual Q network by minimizing the time difference error.

[0063] Subsequently, the policy network is updated to maximize the sum of expected reward and policy entropy, encouraging the policy to remain exploratory. Actions are sampled from the policy distribution using a reparameterization technique, and the gradients of the policy network are passed to the loss function. The goal is to minimize the policy optimization objective. The following expression: (10) in, This refers to actions generated through reparameterization techniques, specifically involving reparameterizing random variables. and the state at time step t Through parameters The action output is mapped to the policy network. This reparameterization technique allows the policy network to be updated with gradients via backpropagation. Similarly, I will not elaborate further; Let represent the expectation operator, where represents the state at time step t. From the experience replay buffer Mid-sampling, random variables used in the reparameterization process From the standard normal distribution Mid-sampling; This represents the experience replay buffer; Represents a standard normal distribution; For parameters Q-value network.

[0064] After the parameter update is completed, this invention uses a soft update mechanism to synchronize the parameters of the target network, in order to maintain the smoothness of the target value estimation and the stability of the learning. The update formula is as follows: (11) in, It is the target value; The soft update rate is typically set to 0.005. Throughout the training process, parameter updates and trajectory acquisition alternate, forming a closed-loop learning process. As the number of training rounds increases, the policy network gradually learns the optimal action selection rules under different states, thereby achieving efficient planning, stable obstacle avoidance, and successful grasping of the dual-arm collaborative trajectory, meeting the practical application needs in various environments.

[0065] Following steps S100 to S500, the following steps are included: After the policy network completes a certain number of training rounds and reaches a convergence trend, the system will enter the policy performance evaluation and physical deployment verification phase. Policy evaluation is first conducted in a simulation environment, primarily by observing indicators such as task completion rate, average reward value, path smoothness, and end-point obstacle avoidance performance in several consecutive evaluation rounds to determine the stability and robustness of the current policy. The evaluation process uses the same simulated task scenario as the training, but without introducing exploration noise to ensure the policy's true performance under deterministic execution conditions. If the policy exhibits a high task success rate, stable path output, and good obstacle avoidance capability in the simulation, it is considered ready for deployment.

[0066] While verifying the practical feasibility of the strategy, this invention also exports and deploys the trained strategy network to a physical Baxter dual-arm robot platform. During deployment, the state variables recorded in the simulation environment are first mapped to the sensors and end-effector positioning system of the physical platform to ensure that the input states remain consistent in coordinate space and dimensions. The strategy network receives real-time state input from the robot's end effector and generates continuous motion outputs. The motion data is converted into end-effector displacement commands by a motion control module interfaced with the physical platform, and then converted into control signals for each joint by an inverse kinematics solver and transmitted to the drive system to execute the robot's physical actions.

[0067] In the deployment and verification experiments, this invention tested several typical task scenarios, including obstacle-free target grasping, static obstacle avoidance, and dynamic interference traversal. Each experiment was repeated several times to eliminate random errors, and key indicators such as end trajectory, task completion time, grasping accuracy, and whether a collision occurred were recorded after each round. Experimental results show that the deployed strategy can stably complete the task objectives without human intervention and maintain good adaptability when facing new obstacles or changes in target position. This indicates that the reinforcement learning method proposed in this invention not only performs excellently in simulation environments but also has engineering feasibility for reliable operation on actual physical platforms.

[0068] In summary, this invention effectively trains continuous control behavior within a reasonable timeframe based solely on state observations. This highlights the effectiveness of the learning strategy in handling new scenarios, demonstrating its ability to adapt to significant environmental changes with minimal additional training. Secondly, this invention effectively addresses the sparse reward problem through the HER algorithm and a hybrid reward function, significantly improving data utilization efficiency and learning speed. Finally, this invention optimizes the reward structure in reinforcement learning, enabling it to guide a dual-arm robot to effectively avoid obstacles and converge quickly.

[0069] Example 3 is the third embodiment of the present invention. This embodiment provides a dual-arm cooperative planning system based on reinforcement learning, including: The model building module is used to build Markov decision models. The Markov decision model includes a state space built based on the position and velocity information of the dual-arm end effectors, and an action space built based on the three-dimensional displacement increment of the end effector space. The strategy generation module is used to generate control actions based on the current state in the state space and the strategy network based on the Soft Actor-Critic algorithm, so as to obtain the three-dimensional displacement increment commands of the left and right arm end effectors. The reward calculation module is used to calculate the hybrid reward function after the dual-arm cooperative control is executed in response to the three-dimensional displacement increment command; The experience enhancement module is used to reset the target of the failed trajectory through the Hindsight Experience Replay mechanism, generate pseudo-target experience, and store the original experience and pseudo-target experience in the replay cache. The parameter optimization module is used to update the parameters of the policy network and Q-value network based on the experience samples in the playback cache, thereby completing the update of the dual-arm cooperative trajectory planning strategy.

[0070] Example 4, the fourth embodiment of the present invention, differs from the previous three embodiments in that: if the function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0072] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0073] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination of all three. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0074] Example 5, the fifth embodiment of the present invention, provides a dual-arm collaborative planning method based on reinforcement learning. To verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0075] In this invention, to train and verify the dual-arm cooperative trajectory planning method, a highly realistic robot operating environment and training platform must first be constructed to support the policy learning process based on reinforcement learning algorithms. Specifically, this invention uses CoppeliaSim as the main simulation platform and imports the Baxter dual-arm robot model into this platform as the physical simulation basis for the intelligent agent. By configuring the end effector and joint parameters of the robot model, it is ensured that it has complete 6-DOF maneuverability and basic control interfaces for coordinated left and right arm movements.

[0076] In the simulation environment, multiple typical task scenarios are preset, including static target grasping, obstacle avoidance grasping, and alternating grasping of multiple targets. Various types of typical interference objects are set as obstacles to simulate common situations in real industrial scenarios, such as workbench interference and dynamic path interference. At the initialization of each training round, the position of the target object is randomly generated within a set range, while the position, size, and number of obstacles are configured according to task requirements to improve the generalization ability and robustness of the strategy.

[0077] In terms of control interaction, the simulation platform communicates with the reinforcement learning training program in real time via a Python interface. The system collects current state variables (including end-effector position, velocity, target relative information, etc.) at a fixed frequency and passes them to the policy network to generate action decisions. The continuous action vectors output by the policy are converted into end-effector spatial displacement commands, which are then converted into specific joint angle commands by the robot inverse kinematics solver. Finally, the simulation engine executes the robotic arm's actions, thus completing one state-action-state interaction loop.

[0078] To support the reinforcement learning training process, an experience replay buffer D needs to be initialized to store the state, action, reward, next state, and current target information for each interaction time step. During training, the policy network and Q-value network periodically sample batch data from D for updates. Simultaneously, a HER mechanism is introduced to "redefine the target" for failed trajectories, and fake experiences are also stored in the buffer, improving training efficiency and policy convergence speed.

[0079] Furthermore, in this embodiment, a target grasping task in a fixed obstacle environment using a dual-arm robot is taken as the actual application scenario to fully deploy and test the reinforcement learning-based dual-arm cooperative trajectory planning method proposed in this invention. The simulation platform used is CoppeliaSim, and the robot model is the Baxter dual-arm robot, which has two robotic arms, each with 7 degrees of freedom. The workbench is set to a size of 1.2 meters × 0.8 meters, and the target object is a cube with a side length of 6 centimeters, with its placement randomly generated within the workbench area. The obstacles are set as three cylinders, each with a diameter of 10 centimeters and a height of 15 centimeters, fixedly distributed at key positions on the path between the target and the initial position, simulating interference structures in typical industrial operations.

[0080] In the initial simulation phase, the initial positions of the dual-arm end effectors were set to (−0.3, 0.0, 0.2) m and (0.3, 0.0, 0.2) m, respectively. The state space was designed to be 24-dimensional, including the three-dimensional position and velocity of the left and right arm end effectors, as well as the position and velocity information of the target and obstacles relative to the end effectors. The motion space was 6-dimensional, controlling the displacement increments of the left and right arm end effectors in three-dimensional Cartesian space, with the motion range of each dimension set to [−0.1 m, 0.1 m]. The motion was output by the policy network and then converted into actual joint control commands by the inverse kinematics module and executed.

[0081] Both the policy network and the Q-value network employ two fully connected neural networks, with a hidden layer size of 256 nodes each. The learning rate is set to 3×10⁻⁶. -4The discount factor γ was set to 0.99, and the experience replay batch size was 128. The Soft Actor-Critic (SAC) algorithm was used for training, combined with the Hindsight Experience Replay (HER) mechanism to improve sample utilization under sparse rewards. Each failed trajectory can generate up to four pseudo-target trajectories based on the "future" strategy and store them in the experience pool D.

[0082] The reward function is designed as a hybrid structure, integrating artificial potential field and attitude guidance mechanism. The attraction term uses Huber loss to construct the target attraction, the repulsion term is the inverse function of obstacle distance, and the attitude guidance part includes position proximity term and target direction angle term. The final expression is shown in the aforementioned formula (8): ; in, It is a hybrid reward function; and The hyperparameters set for experience control the relative importance of the two types of rewards during the learning process. In this embodiment... =1.0, =0.5.

[0083] The training process involved 500,000 simulation interaction steps. After the policy training was completed, the resulting model was deployed on the Baxter physical platform for validation. During deployment, the state mapping module ensured that the data collected by the physical sensors was consistent with the training input, and the control system generated actions in real time to execute the grasping task.

[0084] Through the above implementation process, the reinforcement learning trajectory planning strategy described in this invention exhibits excellent execution performance in both simulation and actual physical platforms. On the simulation platform, training convergence is stable, and the robot can plan continuous, smooth, and obstacle-avoiding paths even with multiple obstacles, successfully completing the target grasping and transport tasks. During physical platform verification, the robot completed collision-free grasping in multiple task rounds, with good end-effector path continuity and no path oscillations or action conflicts. The dual-arm cooperative behavior is natural, with no interference between the left and right arms, and strong trajectory synchronization and stability, meeting the safety and real-time requirements of industrial applications.

[0085] Therefore, the dual-arm cooperative trajectory planning method provided by this invention has good task completion ability, obstacle avoidance ability and trajectory quality, and can be applied to efficient trajectory control tasks in actual multi-arm operating systems. It has significant engineering deployability and promotion value.

[0086] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dual-arm collaborative planning method based on reinforcement learning, characterized in that: include, Construct a Markov decision model; wherein the Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increment of the end effector space; Based on the current state in the state space, a policy network based on the Soft Actor-Critic algorithm generates control actions to obtain three-dimensional displacement increment commands for the left and right arm end effectors. After executing dual-arm cooperative control in response to the three-dimensional displacement increment command, a hybrid reward function is calculated; wherein, the hybrid reward function integrates the attraction and repulsion reward based on the artificial potential field and the guidance reward based on the end posture, and outputs an instantaneous reward value; The execution trajectory is enhanced with experience. The failed trajectory is reset through the Hindsight Experience Replay mechanism to generate pseudo-target experience and store the original experience and the pseudo-target experience in the replay cache. Based on the experience samples in the replay cache, update the parameters of the policy network and Q-value network to complete the update of the dual-arm cooperative trajectory planning strategy.

2. The dual-arm collaborative planning method based on reinforcement learning as described in claim 1, characterized in that: The steps for constructing the state space include: Obtain the three-dimensional position and velocity vectors of the left and right arm end effectors; Obtain the relative position and relative velocity of the target object with respect to the left and right end effectors; The relative position and relative velocity of the obstacle with respect to the left and right end effectors are obtained; The state space is obtained by normalizing the three-dimensional position vector, the velocity vector, the relative position, and the relative velocity.

3. The dual-arm collaborative planning method based on reinforcement learning as described in claim 2, characterized in that: The policy network based on the Soft Actor-Critic algorithm generates control actions, including: The current state is input into the policy network; the policy network is the Actor component in the Soft Actor-Critic algorithm. The policy network outputs action distribution parameters, which include a mean vector and a standard deviation vector. Random sampling is performed based on the action distribution parameters to generate continuous action vectors; Amplitude constraints are applied to the continuous motion vector to obtain the three-dimensional displacement increment commands for the left arm end effector and the right arm end effector.

4. The dual-arm cooperative planning method based on reinforcement learning as described in claim 3, characterized in that: The attraction-repulsion reward based on the artificial potential field includes: The target attraction term is constructed based on the distance between the end effector and the target object; the value of the target attraction term increases as the distance between the end effector and the target object decreases; the obstacle repulsion term is constructed based on the distance between the end effector and the obstacle; when the distance between the end effector and the obstacle is less than a safety threshold, the value of the obstacle repulsion term increases as the distance between the end effector and the obstacle decreases. The end-effector-based guided reward includes: The position guidance item is determined based on the straight-line distance between the end effector and the target, and the position guidance item takes a positive value when the grasping threshold is reached; The direction guidance term is determined by calculating the angle between the end-movement direction and the direction of the line connecting the target, and the value of the direction guidance term is negatively correlated with the angle.

5. The dual-arm collaborative planning method based on reinforcement learning as described in claim 4, characterized in that: The method of resetting the target for the failed trajectory through the Hindsight Experience Replay mechanism includes: Identify the failure trajectories that did not achieve the original objective; From the failed trajectories, states are selected as pseudo-targets according to a preset sampling strategy; wherein, the preset sampling strategy includes a future strategy for selecting future states in the trajectory and a final strategy for selecting the end state of the trajectory; Based on the pseudo-target, the instantaneous reward value of each time step in the failed trajectory is recalculated; The pseudo-target and the instantaneous reward value recalculated based on the pseudo-target are combined with the status and action information of the failure trajectory and stored as the pseudo-target experience in the replay cache.

6. The dual-arm cooperative planning method based on reinforcement learning as described in claim 5, characterized in that: Update the parameters of the policy network and Q-value network, including: Construct a complete network architecture for the Soft Actor-Critic algorithm; wherein the network architecture includes the policy network, the double Q-value network, and the target network; Sample a batch of empirical samples from the replay buffer; The dual-Q value network parameters are updated by minimizing the time difference error; The policy network parameters are updated by maximizing the sum of expected reward and policy entropy; The target network parameters are synchronized using a soft update mechanism.

7. The dual-arm cooperative planning method based on reinforcement learning as described in claim 6, characterized in that: The control of the dual-arm robot by the three-dimensional displacement increment command includes: The three-dimensional displacement increment command is restricted to a set boundary range; The three-dimensional displacement increment command is converted into joint angle control command by an inverse kinematics solver; wherein, the inverse kinematics solver completes the mapping from the end-effector space to the joint space based on the kinematic model of the dual-arm robot; The dual-arm robot is driven to perform coordinated movements according to the joint angle control commands.

8. A dual-arm cooperative planning system based on reinforcement learning, employing the dual-arm cooperative planning method based on reinforcement learning as described in any one of claims 1 to 7, characterized in that, include: The model building module is used to build Markov decision models; The Markov decision model includes a state space constructed based on the position and velocity information of the dual-arm end effectors, and an action space constructed based on the three-dimensional displacement increment of the end effector space. The strategy generation module is used to generate control actions based on the current state in the state space and the strategy network of the Soft Actor-Critic algorithm, so as to obtain the three-dimensional displacement increment commands of the left arm end effector and the right arm end effector. The reward calculation module is used to calculate the hybrid reward function after executing dual-arm cooperative control in response to the three-dimensional displacement increment command; The experience enhancement module is used to reset the target of the failed trajectory through the Hindsight Experience Replay mechanism, generate pseudo-target experience, and store the original experience and the pseudo-target experience into the replay cache. The parameter optimization module is used to update the parameters of the policy network and Q-value network based on the experience samples in the playback cache, thereby completing the update of the dual-arm cooperative trajectory planning strategy.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the two-arm collaborative planning method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the two-arm collaborative planning method based on reinforcement learning as described in any one of claims 1 to 7.

Images