Multi-robot collaborative trajectory generation method and device based on deep reinforcement learning

Abstract

The invention provides a multi-robot cooperative trajectory generation method and device based on deep reinforcement learning. The method comprises the following steps: acquiring state data of each robot in a current time step in a target area; processing the state data by adopting a cross-robot attention mechanism in a pre-trained deep reinforcement learning strategy network to obtain structured action parameters of a mechanical arm of each robot; and on the basis of the structured action parameters of the mechanical arm and the configured constraint conditions, a local track section of the next time step is determined. According to the method, through combination of deep reinforcement learning and a structured action space, collision-free and high-smoothness real-time cooperative trajectory generation of multiple robots is realized.

Description

Technical Field

[0001] This application relates to the field of multi-robot cooperative trajectory planning technology, and more specifically, to a method and apparatus for generating multi-robot cooperative trajectories based on deep reinforcement learning. Background Technology

[0002] In fields such as industrial manufacturing, logistics warehousing, and exploration of complex environments, collaborative operation of multi-robot systems (MRS) has become a key technology for improving the level of automation.

[0003] In existing technologies, multi-robot cooperative trajectory planning mainly employs centralized optimization methods (such as model predictive control), geometric interpolation methods (such as B-spline curves), or deep reinforcement learning methods. Centralized optimization methods solve for the optimal control quantity through global modeling, but their computational complexity is high, making it difficult to meet the real-time requirements of dynamic environments. Geometric interpolation methods focus on the feasibility and safety of the trajectory, but lack guarantees of high-order continuity. Deep reinforcement learning methods treat the robot as an intelligent agent, mapping low-dimensional control commands through a policy network, but their action space definition is simple and cannot effectively handle high-order smoothness and high-dimensional coupling constraints in cooperative motion.

[0004] It is evident that the existing methods mentioned above face challenges in industrial applications, such as collaborative Jerk mutation, the contradiction between real-time performance and robustness, and insufficient handling of complex constraints. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for generating multi-robot cooperative trajectories based on deep reinforcement learning, which combines deep reinforcement learning with a structured motion space to achieve collision-free, high-smoothness real-time cooperative trajectory generation for multiple robots.

[0006] Firstly, a method for generating cooperative trajectories of multiple robots based on deep reinforcement learning is provided, which may include: Acquire the status data of each robot within the target area in the current time step; The state data is processed using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. Based on the structured motion parameters and configured constraints of the robotic arm, the local trajectory segment for the next time step is determined.

[0007] In one possible implementation, the state data is processed using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain structured motion parameters for each robot's arm, including: The state data is normalized and converted into high-dimensional feature vectors, which are then input into a pre-trained deep reinforcement learning policy network. The state data includes the body state of each robotic arm, environmental information, and the cooperative state between robots. The high-dimensional feature vector is processed by the deep reinforcement learning policy network through a cross-robot attention mechanism, the interaction weights between robot states are calculated, and global features containing global collaborative information are aggregated. Based on the output layer of the deep reinforcement learning policy network, the global features are mapped to the structured motion parameters of each robotic arm.

[0008] In one possible implementation, the local trajectory segment is represented by a polynomial of a preset order; the constraints include boundary constraints and the corresponding physical constraints of the robotic arm. Based on the structured motion parameters and configured constraints of the robotic arm, the local trajectory segment for the next time step is determined, including: The structured motion parameters are used as the coefficient vector of a polynomial of a preset order to construct a polynomial trajectory function of each robotic arm joint with respect to the time variable. Based on the aforementioned boundary constraints, an initial local trajectory segment satisfying higher-order continuity is generated by solving the continuity equation of the polynomial trajectory function at the connection point. If the initial local trajectory segment satisfies the physical constraint conditions, then the initial local trajectory segment is determined to be the local trajectory segment for the next time step.

[0009] In one possible implementation, based on the boundary constraints, an initial local trajectory segment satisfying higher-order continuity is generated by solving the continuity equation of the polynomial trajectory function at the connection point, including: A system of continuity equations for the running state of a polynomial trajectory function at continuous connection points is constructed, wherein the running state includes derivatives of four orders: position, velocity, acceleration, and jerk. Solve the continuity equations to obtain the polynomial coefficients that satisfy the boundary constraints; Substituting the polynomial coefficients into the polynomial trajectory function generates an initial local trajectory segment that satisfies continuity in the time domain.

[0010] In one possible implementation, the method further includes: The local trajectory segment is sampled at fixed time intervals to obtain the trajectory position at multiple time points; Convert the position status of trajectory location points at multiple time points into corresponding trajectory point commands; The controller converts the corresponding trajectory point commands into motor torque signals to drive the movement of each robotic arm joint.

[0011] In one possible implementation, the method further includes: Based on the composite reward function, the state data corresponding to the local trajectory segment is processed to calculate the composite reward value and store it in the experience playback buffer. If K local trajectory segments at different time steps are cumulatively determined, then empirical data is randomly sampled from the empirical replay buffer to update the deep reinforcement learning policy network offline. The empirical data includes multiple empirical tuples, where K is an integer greater than 5.

[0012] In one possible implementation, the composite reward function includes task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward; Based on the composite reward function, the state data corresponding to the local trajectory segment is processed to calculate the composite reward value, including: The state data corresponding to the local trajectory segment is processed in parallel to calculate four sub-items of reward: task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward. Based on preset weighting coefficients, the rewards of each sub-item are weighted and summed to obtain a composite reward value that comprehensively evaluates the quality of the trajectory segment; The composite reward value, along with the corresponding state data, action data, and next state data, is encapsulated into an experience tuple. The experience tuples are stored in the experience replay buffer for subsequent batch training and optimization updates of the policy network.

[0013] Secondly, a multi-robot cooperative trajectory generation device based on deep reinforcement learning is provided, the device may include: The acquisition unit is used to acquire the state data of each robot within the target area in the current time step. The processing unit is used to process the state data using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. The determining unit is used to determine the local trajectory segment of the next time step based on the structured motion parameters and configured constraints of the robotic arm.

[0014] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0016] This application provides a method and apparatus for generating cooperative trajectories for multiple robots based on deep reinforcement learning. The method acquires the state data of each robot within the target area at the current time step; it processes the state data using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of each robot's robotic arm; based on the structured motion parameters of the robotic arm and the configured constraints, it determines the local trajectory segment for the next time step. This method, by combining deep reinforcement learning with a structured motion space, achieves collision-free, high-smoothness real-time cooperative trajectory generation for multiple robots. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a multi-robot cooperative trajectory generation method based on deep reinforcement learning, provided for an embodiment of this application; Figure 2 A schematic diagram of the acceleration curve of a robotic arm joint changing over time during cooperative obstacle avoidance using a C3 continuous trajectory, provided as an embodiment of this application; Figure 3 A schematic diagram of a multi-robot cooperative trajectory generation device based on deep reinforcement learning provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects. The terms "connection," "coupled," or "linked," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0020] Existing technologies cannot guarantee the high-order continuity of trajectories in multi-robot cooperative motion, leading to Jerk mutations at cooperative points that cause system vibrations; centralized optimization computation is time-consuming, making it difficult to achieve millisecond-level real-time replanning; and the simplified processing of high-dimensional coupling constraints (such as collision avoidance and singularity avoidance) reduces trajectory accuracy and robustness.

[0021] The multi-robot cooperative trajectory generation method based on deep reinforcement learning provided in this application is particularly suitable for generating high-order continuous (e.g., C3) cooperative trajectories of multiple redundant robotic arms. The core of this method lies in constructing a reinforcement learning policy for a structured motion space, using a Transformer network to handle the cooperative relationships between multiple robots, and driving policy learning through a composite reward function to achieve real-time, safe, and smooth trajectory planning. The entire system completes the closed loop from state perception to trajectory execution within a short control cycle, meeting the high real-time requirements of industrial applications.

[0022] The multi-robot cooperative trajectory generation method based on deep reinforcement learning provided in this application can be applied to a multi-robot cooperative trajectory generation system based on deep reinforcement learning. This system may include: a distributed perception and state observation unit, a cooperative policy generation unit, a trajectory constraint and execution unit, and an offline training and update unit. Wherein: 1) Distributed sensing and state observation units, including: Body state acquisition module: used to acquire the joint position, speed and torque information of each robotic arm in real time through joint encoder, and to calculate the current end pose and Jacobian matrix.

[0023] Environmental perception module: Uses a global vision system (such as a depth camera or LiDAR) to collect environmental point cloud data and extract the position and velocity information of dynamic obstacles.

[0024] Cooperative state analysis module: used to calculate the relative distance, relative pose error, and deviation relative to the cooperative task target among multiple robots.

[0025] 2) Cooperative strategy generation unit (core controller), including: Feature encoding module: used to convert the observed ontology state, environmental information and cooperative state into high-dimensional feature vectors.

[0026] Transformer Collaborative Reasoning Module: It has a built-in pre-trained Transformer policy network that uses a cross-robot attention mechanism to process the feature interactions between robots and outputs global features containing collaborative information.

[0027] Structured motion generation module: Based on global features, it directly outputs the 7th order polynomial coefficients of each robotic arm in the next time step, instead of the original speed command, defining local high-order trajectory segments from the source.

[0028] 3) Trajectory constraint and execution unit, including: Higher-order continuity constraint module: Combines the boundary conditions of the current joint state (such as the position, velocity, acceleration, and jerk conditions of the position state) and uses the output polynomial coefficients to generate a joint trajectory sequence that satisfies C3 continuity.

[0029] The underlying servo control module converts the generated smooth trajectory sequence into high-frequency motor commands to drive the movement of each robotic arm joint.

[0030] 4) Offline training and update units, including: Composite reward calculation module: responsible for calculating the total reward value, including task efficiency, safety penalty, singularity penalty, and smoothness (Jerk) penalty.

[0031] Policy optimization and update module: Using PPO or SAC algorithm, the parameters of the policy network are iteratively updated based on the cumulative reward, and the trained model parameters are deployed to the collaborative policy generation unit.

[0032] Furthermore, the following description will elaborate on the hardware configuration, software implementation, and detailed steps.

[0033] (1) The hardware environment required to implement the method of this application includes: Robot system: A collaborative workstation consisting of at least two redundant robotic arms (such as a 7-DOF robotic arm), each equipped with a high-precision absolute encoder (sampling frequency 1kHz) and a torque sensor, communicating with the central processing unit via a CAN bus.

[0034] Perception system: Multiple depth cameras or LiDARs are deployed at the top of the workspace to collect environmental point cloud data in real time and detect the outlines of static obstacles and the movement trajectories of dynamic obstacles (such as AGVs and personnel).

[0035] Computing units: High-performance GPU servers (such as NVIDIA A100) are used for offline training of DRL policy networks; embedded GPUs (such as NVIDIA Jetson series) are used for online real-time inference, ensuring decision latency of 1 millisecond.

[0036] (2) The software environment required to implement the method of this application includes: Simulation environment: A multi-robot collaborative simulation environment is built using a high-fidelity physics engine (such as MuJoCo or Isaac Gym) to verify the physical accuracy of the DRL strategy.

[0037] Policy network architecture: It adopts an Actor-Critic framework (such as MAPPO or MADDPG) with centralized training and distributed execution (CTDE), with Transformer as the main body of the network, and uses its cross-robot attention mechanism to handle multi-agent interactions.

[0038] Training algorithm: The policy network is trained using the Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) algorithm, with data stored and sampled through an experience replay buffer.

[0039] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0040] Figure 1 This is a flowchart illustrating a multi-robot cooperative trajectory generation method based on deep reinforcement learning, provided as an embodiment of this application. Figure 1 As shown, the method may include: Step S110: Obtain the status data of each robot within the current time step in the target area.

[0041] Status data can include the body status of each robotic arm (such as joint position and speed), environmental information (such as obstacle position), and the cooperative status between robots (such as relative distance).

[0042] Specifically, the body state can be acquired by using the joint encoders of each robotic arm to collect joint angle position signals at a frequency of 1kHz, and then calculating the instantaneous angular velocity of the joints through differential calculation. Torque sensors monitor the joint output torque in real time, and the data is transmitted to the central processing unit via the CAN bus. This allows the joint encoders to collect the joint position, velocity, and torque of each robotic arm in real time, and calculate the end effector pose and Jacobian matrix (for singularity detection). The body state vector can include joint position, velocity, torque, and end effector pose.

[0043] For environmental information, depth camera point cloud data can be processed through voxel mesh downsampling and statistical outlier removal to extract feature points on obstacle surfaces. Multi-camera data is fused using point cloud registration algorithms to generate a 3D environmental model, which tracks the trajectories of dynamic obstacles in real time, thus extracting the position and velocity information of dynamic obstacles.

[0044] The cooperative state can be obtained by acquiring the relative geometric relationships between computational robots, including the Euclidean distance between end effectors, the relative posture transformation matrix, and cooperative task progress indicators (such as deviation from the target point).

[0045] Furthermore, the acquired information can be preprocessed: the state data can be timestamped to ensure that the time synchronization of multi-source data is within 0.1ms. The preprocessed data is then encapsulated into a standardized data structure, which may include timestamps, robot IDs, joint state vectors, environmental feature matrices, and cooperative state flags.

[0046] Step S120: The state data is processed using the cross-robot attention mechanism in a pre-trained deep reinforcement learning strategy network to obtain the structured motion parameters of the robotic arms of each robot.

[0047] In practice, the state data is normalized and converted into high-dimensional feature vectors, which are then input into a pre-trained deep reinforcement learning policy network. Specifically: the ontology state is mapped to a 512-dimensional hidden feature space through fully connected layers, and nonlinearity is introduced using the GeLU activation function. Environmental information (environmental point cloud data) is processed by a PointNet++ network to extract multi-scale geometric features, outputting a 256-dimensional environmental context vector. The cooperative state is processed by a pose encoding network to generate 128-dimensional relational features. All feature vectors are layer-normalized and then concatenated to form a comprehensive feature representation for each robot, i.e., a high-dimensional feature vector input into the pre-trained deep reinforcement learning policy network.

[0048] The high-dimensional feature vector is processed through a cross-robot attention mechanism using a deep reinforcement learning policy network. This process calculates the interaction weights between robot states and aggregates them to generate global features containing global collaborative information. Based on the output layer of the deep reinforcement learning policy network, these global features are mapped to structured motion parameters for each robot arm. Specifically, each robot's high-dimensional feature vector is transformed linearly to generate a query vector Q, a key vector K, and a value vector V. Attention scores are calculated using scaled dot product attention. A spatial distance-based attention bias is configurable, with higher weights for robots closer together. Eight attention heads are computed in parallel to capture interactions such as collision avoidance and task division. The attention mechanism output undergoes residual connections and layer normalization, and a feedforward network further processes the features to generate a 1024-dimensional global feature vector. Finally, the global feature vector is used to generate structured motion parameters through the policy network's output layer. The output layer is configured with a fully connected network for each robot, mapping the 1024-dimensional feature vector to the motion space. For a 7-DOF robot arm, the output is 56-dimensional parameters (7 joints × 8 coefficients), corresponding to the coefficients of a 7th-order polynomial.

[0049] Step S130: Based on the structured motion parameters and configured constraints of the robotic arm, determine the local trajectory segment for the next time step.

[0050] The local trajectory segment is represented by a polynomial of a preset order; the configured constraints may include boundary constraints and the physical constraints of the corresponding robotic arm (such as joint limits, collision avoidance, etc.).

[0051] In practice, structured motion parameters are used as coefficient vectors of a polynomial of a preset order to construct a polynomial trajectory function of each robotic arm joint with respect to the time variable. For each robotic arm joint, the polynomial trajectory function with respect to the time variable is represented by a 7th-order polynomial: Where τ∈[0,1] is the normalized time variable, which maps the actual time interval (current time step) [t,t+Δt] to the unit interval; arrive This is the coefficient vector output by the policy network. The polynomial order is chosen to be 7 because the 8 coefficients can satisfy the boundary condition degrees of freedom required for C3 continuity (continuity of the four orders of position, velocity, acceleration, and jerk).

[0052] To verify continuity, the derivatives of each order need to be calculated: speed: ; Acceleration: ; Jerk: .

[0053] Among them, actual time ∈[t,t+Δt] through linear transformation τ=( The time interval (t) / Δt is converted to normalized time to ensure the consistency of the trajectory segment across different time scales. Normalization simplifies derivative calculations and the application of boundary conditions.

[0054] Subsequently, based on boundary constraints, an initial local trajectory segment satisfying higher-order continuity is generated by solving the continuity equation of the polynomial trajectory function at the connection points. The boundary constraints require the trajectory segment to satisfy C3 continuity (position, velocity, acceleration, and jerk continuity) at the connection points (i.e., the end point of the current segment and the start point of the next segment). The initial local trajectory segment is generated by constructing and solving the continuity equation.

[0055] For constructing the continuity equation: Let the state at the end of the current trajectory segment at τ=1 be... The state at the next starting time τ=0 must be continuous with it. Therefore, a system of 8 linear equations is established: Starting point constraint (τ=0):

[0056] Connection point continuity constraint (τ=1):

[0057] The system of equations can be represented in matrix form as Ax=b, where A is an 8×8 coefficient matrix (a variant of the Vandermonde matrix). is the coefficient vector, and b is the boundary condition vector.

[0058] For a well-formed matrix (boundary condition number less than 1), The solution is obtained directly using Gaussian elimination with partial pivoting. For ill-conditioned matrices (where the number of boundary conditions is greater than...), the solution is more complex. This triggers singular value decomposition (SVD) to improve numerical stability. After solving, polynomial coefficients satisfying continuity constraints are obtained, generating initial local trajectory segments. The solution process is completed in real time on the embedded system, taking less than 0.1 ms.

[0059] After generating the initial local trajectory segment, the continuity of C3 is verified by numerical differentiation: the differences in position, velocity, acceleration, and jerk at the connection point are calculated. If the difference is less than the tolerance (e.g., ... If the condition is met, then the continuity is confirmed. Verification ensures that the trajectory is smooth and without abrupt changes, avoiding vibrations in cooperative motion.

[0060] This method ensures the high-order continuity of the trajectory through mathematical constraints, thus solving the Jerk mutation problem at its source. Figure 2 As shown.

[0061] Finally, the generated initial local trajectory segment is verified through physical constraints; the physical constraints ensure that the trajectory segment is executed safely within the physical limits of the robotic arm joints, including joint limit constraints and collision avoidance constraints.

[0062] If the initial local trajectory segment meets the physical constraints, it is determined as the local trajectory segment for the next time step. If the initial local trajectory segment does not meet the physical constraints, a hierarchical processing mechanism is triggered to ensure real-time performance and safety.

[0063] Specifically, in the joint limit verification, (1) position limit check: traverse all time points on the trajectory segment (discrete at 1ms intervals) and check whether the position of each joint is within the design limits of the robotic arm joint (e.g., If the limit is exceeded, it will be marked as a violation. (2) Speed ​​limit check: Calculate the joint speed v(τ) and ensure that it is lower than the maximum allowable value (e.g., Exceeding the speed limit may cause motor overload.

[0064] In the collision avoidance verification, (1) geometry-based collision detection: continuous collision detection is performed using an axis-aligned bounding box hierarchy tree (AABBTree). The robotic arm links and obstacles are modeled as geometric bounding boxes, and the minimum Euclidean distance at all time points on the trajectory segment is calculated. If the distance is less than the safety threshold (e.g., 1.2 times the robot radius), a collision warning is triggered. (2) dynamic obstacle detection: obstacle positions are updated in real time by combining environmental perception data (e.g., depth camera point clouds). The collision detection algorithm considers the obstacle's motion trajectory and avoids future collisions through linear prediction. (3) self-collision detection: interference between links between multiple robots or within the robotic arm itself is checked to ensure safe cooperative motion.

[0065] The tiered handling mechanism can include handling minor violations (such as small joint overruns or near-threshold distances), moderate violations (such as approaching joint limits or high collision risk), and severe violations (such as impending collisions). Minor violation handling can include online trajectory correction and local replanning (preserving the basic shape of the current trajectory segment and adjusting only the overrun portion). Moderate violation handling can include progressive deceleration (interrupting current trajectory execution and generating a smooth deceleration trajectory (such as using a fifth-order polynomial) to bring the robotic arm to a smooth stop at a safe position) and rolling optimization replanning (regenerating trajectory segments based on the latest state using a policy network). Severe violation handling can include emergency braking (immediately cutting off motor power and triggering a hardware braking program) and safe trajectory generation (activating an emergency avoidance strategy to generate a shortest escape trajectory. This trajectory may sacrifice smoothness but ensures immediate escape from the danger zone).

[0066] In some embodiments, after the local trajectory segment is determined in the next time step, the local trajectory segment can be continuously sampled at fixed time intervals to obtain the trajectory positions at multiple time points. The position states corresponding to the trajectory positions at multiple time points are converted into corresponding trajectory point commands. The position states may include timestamps, target positions, target velocities, and target accelerations. Then, the controller converts the corresponding trajectory point commands into motor torque signals to drive the movement of each robotic arm joint. This controller is a low-level proportional-integral-derivative (PID) controller.

[0067] Specifically, local trajectory segments can be discretized with a 1ms control cycle. At each interval, the target position, velocity, and acceleration are calculated using the 5th-order Runge-Kutta method to form a command sequence. Simultaneously, the Jacobian matrix for each point is calculated for Cartesian space error compensation. The command sequence is encapsulated into a standard communication protocol, including timestamps, control modes, target values, and security check codes, and sent to each joint actuator via Ethernet.

[0068] In some embodiments, the quality of a trajectory segment and a deep reinforcement learning policy network can be evaluated based on a composite reward function, including: processing the state data corresponding to a local trajectory segment based on the composite reward function, calculating a composite reward value, and storing it in an experience replay buffer. The composite reward function may include task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward. Specifically: the state data corresponding to the local trajectory segment is processed in parallel to calculate four sub-items of reward: task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward; based on preset weight coefficients, the sub-item rewards are weighted and summed to obtain a composite reward value that comprehensively evaluates the quality of the trajectory segment; the composite reward value, along with the corresponding state data, action data, and next state data, is encapsulated into an experience tuple; the experience tuple is stored in the experience replay buffer for subsequent batch training and optimization updates of the policy network.

[0069] Task efficiency reward ( ): Encourages quick and accurate task completion, including rewards for approaching and reaching the goal. The formula is: .in, is the task efficiency weighting coefficient, a positive real number used to adjust the importance of task completion speed. The larger the value, the more the system tends to approach the goal quickly; N is the total number of collaborative robots. The actuator pose (position and orientation) of robot i at the end of time is a three-dimensional or six-dimensional vector (depending on the task space dimension). Let i be the target pose of robot i, and The same dimension represents the expected final state of the task; The reward for reaching the target is a scalar value. It is triggered when the distance between the robot's end effector and the target pose is less than a set threshold, and is usually set to a normal value to encourage task completion; Calculate the L2 norm (Euclidean norm) of a vector. The modulus represents the straight-line distance between the robot's end effector and the target point.

[0070] Safety penalties ( ): Used to avoid collisions and singularities, inflicting a huge negative reward upon collision. The formula is: .in, This is a collision penalty term, a negative scalar. When a collision between the robot and an obstacle or another robot is detected, a large negative value (e.g., -1000) is assigned to force collision avoidance. The singularity penalty term is a negative scalar, calculated based on the operability index, and is used to guide the robot away from the kinematic singular configuration.

[0071] Singularity penalty ( Based on operability indicators Actively guide strategies away from singular regions. The formula is: .in, This is the singularity weighting coefficient, a positive real number that adjusts the importance of avoiding singular configurations. A larger value indicates a more sensitive system to singularities. The operability index function takes the joint position vector of robot i as input. The output is a positive scalar. It is usually defined as the condition number or determinant of the Jacobian matrix. The smaller the value, the closer it is to a singularity (and the worse the operability). For very small positive numbers (e.g.) (), used to prevent the denominator from being zero and to enhance numerical stability; Let be the joint angle vector of robot i, with dimensions equal to the degrees of freedom of the robotic arm (e.g., 7-dimensional).

[0072] Smoothness penalty ( ): Explicitly minimize jerk (Jerk) to ensure the generation of a continuous C3 trajectory and suppress vibration. The formula is: .in, This is the smoothness weighting coefficient, a positive real number that controls the strength of the trajectory acceleration penalty. A larger value results in a smoother trajectory. For robot i in time The jerk vector (Jerk) is the joint position. The third derivative with respect to time has the same dimension as the number of joints. A large Jerk value indicates that the motion is not smooth and is prone to vibration. The variable is the integral variable, representing continuous time, with a value range of [t, t+Δt]; t is the current time point, i.e., the start time of the trajectory segment; Δt is the time step of the local trajectory segment, which is a positive real number, usually in the millisecond range (e.g., 0.001 seconds). The square of the L2 norm is used to calculate the square of the Euclidean norm of the jerk vector, which is used to quantize the size of Jerk (to avoid negative values ​​canceling each other out).

[0073] Synergistic Coupling Rewards ( ): This penalizes the deviation from the expected relative pose, ensuring the accuracy of the collaborative task. The formula is: .in, This is the collaborative coupling weight coefficient, a positive real number that adjusts the importance of relative pose accuracy among multiple robots. A larger value indicates higher accuracy requirements for the collaborative task. and Let represent the end-effector poses of robot 1 and robot 2 at time t, respectively; Define the relative position and orientation deviation that the robots should maintain (such as the relative gripping points of the workpiece in a handling task) for the desired relative pose vector.

[0074] Subsequently, if K local trajectory segments at different time steps are cumulatively identified, empirical data is randomly sampled from the empirical replay buffer to update the deep reinforcement learning policy network offline. That is, once sufficient data has accumulated in the empirical buffer, the policy network update process is initiated. Small batches of empirical data are randomly sampled from the buffer, and the advantage function estimate is calculated. The PPO algorithm is used to update the policy network, pruning the policy ratio to prevent excessive updates. The value function network is updated by minimizing the mean squared temporal difference error. The updated network parameters are validated on a test set and then deployed to the online inference system. The empirical data includes multiple empirical tuples.

[0075] Furthermore, this application can also be based on higher-order smoothness constraints of Snap (acceleration), specifically, the smoothness penalty term in the reward function can be... The penalty for Jerk is upgraded to a penalty for Snap (where Snap is the fourth derivative of position, i.e., the rate of change of jerk). Specifically, the L2 norm or square integral of the trajectory Snap is calculated and used as a negative reward. The formula for the upgraded smoothness penalty is: Simultaneously, a requirement for fourth-order derivative continuity is added to the boundary condition constraints to ensure that the generated trajectory achieves C4 continuity. Since C4 continuity provides higher smoothness than C3, it can further suppress Jerk's abrupt changes (i.e., limit the rate of change of force). This can significantly reduce motor current fluctuations and mechanical vibrations in ultra-precision manufacturing (such as optical lens polishing and wafer transfer) or high-speed, high-dynamic motion scenarios, thus extending the robot's lifespan.

[0076] Furthermore, this application can also model topological interactions based on graph neural networks (GNN / GCN). Specifically, graph convolutional networks (GCN) or graph attention networks (GAT) can be used instead of Transformers as the backbone architecture of the policy network. Each robot is modeled as a node in the graph, and the communication or physical proximity relationships between robots are modeled as edges. Feature information of neighboring nodes is aggregated through a message passing mechanism to extract collaborative features. For dynamic environments, a dynamic graph structure can be constructed, and the connection relationships are updated in real time according to the Euclidean distance between robots. Because GCN / GNN has natural advantages in processing non-Euclidean data and graph structure data, it is particularly suitable for describing the topology of multi-robot systems. Compared with fully connected Transformers, GNN can more effectively handle inputs from different numbers of robots (with better permutation invariance), improve the generalization ability of the policy network when the number of robots changes (e.g., training with 3 robots and testing with 5 robots), and has higher computational efficiency in sparse interaction scenarios.

[0077] Furthermore, this application can also simplify the motion space based on joint acceleration. Specifically, the output motion space of the policy network can be defined as the instantaneous joint acceleration at the next moment. or joint torque In this scheme, instead of directly generating a polynomial curve, the position and velocity at the next moment are obtained through kinematic integration. To constrain smoothness, a penalty term for the change in acceleration (i.e., approximate Jerk) is introduced into the reward function: ,in, Let represent the joint acceleration vector at time t+1, which is the second derivative of position with respect to time; This provides the joint acceleration vector at the current moment, thus indirectly suppressing Jerk reactions. This approach significantly reduces the dimensionality of the action space (from multiple sets of polynomial coefficients to a single acceleration vector), simplifies the output layer of the neural network, and typically accelerates the convergence speed of reinforcement learning training, reducing training difficulty. Although the mathematical guarantee of C3 continuity is slightly weaker than the polynomial coefficient method (which is a soft constraint), it is an efficient compromise in applications where smoothness requirements are lower or real-time control frequency requirements are extremely high.

[0078] The method described in this application ensures that multi-robot systems can achieve efficient, safe, and smooth collaborative operation in complex dynamic environments, with all technical indicators meeting the requirements for industrial applications. The entire system completes a closed loop from state perception to trajectory execution within a millisecond-level control cycle, demonstrating a deep integration of algorithm design and engineering implementation.

[0079] Corresponding to the above method, this application also provides a multi-robot cooperative trajectory generation device based on deep reinforcement learning, such as... Figure 3 As shown, the device includes: Acquisition unit 310 is used to acquire the state data of each robot within the target area in the current time step; The processing unit 320 is used to process the state data using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. The determining unit 330 is used to determine the local trajectory segment of the next time step based on the structured motion parameters and configured constraints of the robotic arm.

[0080] The functions of each functional unit in the multi-robot cooperative trajectory generation device based on deep reinforcement learning provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the multi-robot cooperative trajectory generation device based on deep reinforcement learning provided in the embodiments of this application will not be repeated here.

[0081] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440.

[0082] Memory 430 is used to store computer programs; When the processor 410 executes the program stored in the memory 430, it performs the following steps: Acquire the status data of each robot within the target area in the current time step; The state data is processed using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. Based on the structured motion parameters and configured constraints of the robotic arm, the local trajectory segment for the next time step is determined.

[0083] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0084] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0085] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0086] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0088] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the deep reinforcement learning-based multi-robot cooperative trajectory generation methods described in the above embodiments.

[0089] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the deep reinforcement learning-based multi-robot cooperative trajectory generation methods described in the above embodiments.

[0090] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0095] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for generating cooperative trajectories of multiple robots based on deep reinforcement learning, characterized in that, The method includes: Acquire the status data of each robot within the target area in the current time step; The state data is processed using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. Based on the structured motion parameters and configured constraints of the robotic arm, the local trajectory segment for the next time step is determined.

2. The method as described in claim 1, characterized in that, The state data is processed using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot, including: The state data is normalized and converted into high-dimensional feature vectors, which are then input into a pre-trained deep reinforcement learning policy network. The state data includes the body state of each robotic arm, environmental information, and the cooperative state between robots. The high-dimensional feature vector is processed by the deep reinforcement learning policy network through a cross-robot attention mechanism, the interaction weights between robot states are calculated, and global features containing global collaborative information are aggregated. Based on the output layer of the deep reinforcement learning policy network, the global features are mapped to the structured motion parameters of each robotic arm.

3. The method as described in claim 1, characterized in that, The local trajectory segment is represented by a polynomial of a preset order; the constraints include boundary constraints and the physical constraints of the corresponding robotic arm. Based on the structured motion parameters and configured constraints of the robotic arm, the local trajectory segment for the next time step is determined, including: The structured motion parameters are used as the coefficient vector of a polynomial of a preset order to construct a polynomial trajectory function of each robotic arm joint with respect to the time variable. Based on the aforementioned boundary constraints, an initial local trajectory segment satisfying higher-order continuity is generated by solving the continuity equation of the polynomial trajectory function at the connection point. If the initial local trajectory segment satisfies the physical constraint conditions, then the initial local trajectory segment is determined to be the local trajectory segment for the next time step.

4. The method as described in claim 3, characterized in that, Based on the aforementioned boundary constraints, by solving the continuity equation of the polynomial trajectory function at the connection point, an initial local trajectory segment satisfying higher-order continuity is generated, including: A system of continuity equations for the running state of a polynomial trajectory function at continuous connection points is constructed, wherein the running state includes derivatives of four orders: position, velocity, acceleration, and jerk. Solve the continuity equations to obtain the polynomial coefficients that satisfy the boundary constraints; Substituting the polynomial coefficients into the polynomial trajectory function generates an initial local trajectory segment that satisfies continuity in the time domain.

5. The method as described in claim 1, characterized in that, The method further includes: The local trajectory segment is sampled at fixed time intervals to obtain the trajectory position at multiple time points; Convert the position status of trajectory location points at multiple time points into corresponding trajectory point commands; The controller converts the corresponding trajectory point commands into motor torque signals to drive the movement of each robotic arm joint.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the composite reward function, the state data corresponding to the local trajectory segment is processed to calculate the composite reward value and store it in the experience playback buffer. If K local trajectory segments at different time steps are cumulatively determined, then empirical data is randomly sampled from the empirical replay buffer to update the deep reinforcement learning policy network offline. The empirical data includes multiple empirical tuples, where K is an integer greater than 5.

7. The method as described in claim 6, characterized in that, The composite reward function includes task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward; Based on the composite reward function, the state data corresponding to the local trajectory segment is processed to calculate the composite reward value, including: The state data corresponding to the local trajectory segment is processed in parallel to calculate four sub-items of reward: task efficiency reward, safety penalty, smoothness penalty, and cooperative coupling reward. Based on preset weighting coefficients, the rewards of each sub-item are weighted and summed to obtain a composite reward value that comprehensively evaluates the quality of the trajectory segment; The composite reward value, along with the corresponding state data, action data, and next state data, is encapsulated into an experience tuple. The experience tuples are stored in the experience replay buffer for subsequent batch training and optimization updates of the policy network.

8. A multi-robot cooperative trajectory generation device based on deep reinforcement learning, characterized in that, The device includes: The acquisition unit is used to acquire the state data of each robot within the target area in the current time step. The processing unit is used to process the state data using a cross-robot attention mechanism in a pre-trained deep reinforcement learning policy network to obtain the structured motion parameters of the robotic arms of each robot. The determining unit is used to determine the local trajectory segment of the next time step based on the structured motion parameters and configured constraints of the robotic arm.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images