Multi-robot cooperative path planning and dynamic obstacle avoidance decision method based on deep reinforcement learning

Abstract

The application belongs to the technical field of non-electric variable regulation, and relates to a multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning, which comprises the following steps: acquiring a pose deviation vector, a relative obstacle distance and a transient acceleration vector of a controlled robot, and calculating an interactive conflict degree feature; using a deep reinforcement learning model to extract mapping correlation of the above parameters in a multi-machine coupling environment, and outputting a feedback gain correction coefficient; establishing a pose closed-loop regulation loop and mapping the feedback gain correction coefficient to the loop in real time; adjusting a differential item feedback gain to constrain a control signal output gradient, and inhibiting non-continuous steps of a controlled variable; and driving a motor adjustment unit to move the robot according to the corrected control signal, wherein the application constructs a regulation architecture based on a dynamic virtual damping, absorbs non-continuous energy jumps generated by path re-planning by using gain correction, eliminates regulation oscillation induced by instruction steps, keeps a pose evolution curve stable, and improves motion stability.

Description

Technical Field

[0001] This invention belongs to the field of non-electric variable regulation technology, and in particular relates to a multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning. Background Technology

[0002] Currently, in automated sorting and flexible manufacturing scenarios, multi-robot collaborative control systems ensure operational efficiency and safety by adjusting non-electrical physical variables such as pose, speed, and acceleration. This mainstream technology utilizes deep reinforcement learning models to collect environmental states and output obstacle avoidance commands or trajectory planning sequences. However, when these planning commands drive the underlying control loops, a mismatch in physical response dimensions arises. Because the decision commands output by the model have discrete characteristics, and the controlled robot's adjustment process is constrained by the physical inertia of the actuator and the system's frequency response bandwidth, the planning logic and physical evolution become decoupled. In high-frequency dynamic obstacle avoidance environments, the control quantity undergoes a step change, inducing the actuator to enter the power saturation region.

[0003] Besides the limitations imposed by the intrinsic physical properties of the actuator, existing technologies are insufficient in coordinating logical steps and physical responses at the control logic level. For example, Chinese invention patent CN118769255B discloses a method and system for identifying robot dynamic parameters based on physical consistency constraints and load-related friction models. Although the technology introduces load-related friction models and physical consistency constraints to improve the accuracy of dynamic parameter identification, it is essentially still within the scope of static or quasi-static modeling of the intrinsic properties of the controlled object. It does not establish a real-time physical damping constraint mechanism for the discrete planning instructions generated by the intelligent decision-making layer. If the above contradictions are alleviated by increasing the sampling frequency or increasing the actuator power, it will not only increase the computational overhead and hardware cost, but also introduce high-frequency control noise. This planning mode without physical constraints causes the controlled physical quantities to oscillate, resulting in reduced trajectory tracking accuracy and accelerated fatigue of mechanical components. This design approach, which limits path planning to motion geometry optimization while ignoring the consistency of the dynamic characteristics of the system, forms a common bottleneck in the industry that restricts the smooth control of the system under high dynamic conditions.

[0004] Therefore, how to construct a coordination mechanism that smoothly maps intelligent decision-making logic to underlying physical adjustment parameters, thereby solving the mismatch between discrete planning instructions and the continuous physical evolution of the controlled object, becomes the technical problem to be solved by this invention. Summary of the Invention

[0005] This invention aims to solve the problems of physical quantity regulation oscillation and controlled variable instability caused by discrete path planning instructions driving the underlying actuator.

[0006] In this technical solution, a multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning includes:

[0007] Step 101: Obtain the pose deviation vector, relative obstacle distance, and transient acceleration vector fed back by the inertial sensor of the controlled multi-robot, and calculate the interaction conflict degree feature that characterizes the deviation intensity of the controlled variables in the pose closed-loop adjustment loop.

[0008] Step 102: Use a pre-trained deep reinforcement learning model to extract the pose deviation vector, interaction conflict degree features, and transient acceleration vector mapping correlation in a multi-machine coupled environment, and calculate and output the feedback gain correction coefficient containing logical weights.

[0009] Step 103: Establish a pose closed-loop control loop that includes proportional, integral and derivative terms. The pose closed-loop control loop uses the pose of multiple robots as the controlled variable.

[0010] Step 104: The feedback gain correction coefficient is mapped to the pose closed-loop adjustment loop in real time. By adjusting the feedback gain of the differential term, the output gradient of the control signal is physically constrained to suppress the discontinuous step of the controlled variable induced by path replanning.

[0011] Step 105: Based on the control signal after differential gain correction, a drive current is generated to drive the motor adjustment unit to move the multiple robots, so that when the multiple robots respond to obstacle avoidance actions, the cumulative variance of their pose evolution curve relative to the preset trajectory is less than 0.05 within a 10ms sampling period.

[0012] Preferably, step 101 is further refined as follows: step 1011, collecting the relative pose vector and relative velocity vector between each controlled robot; step 1012, calculating the interaction conflict degree feature reflecting the adjustment pressure based on the rate of change of the relative pose vector and relative velocity vector in the time series; step 1013, determining whether the interaction conflict degree feature exceeds the preset steady-state threshold of 0.75.

[0013] Preferably, when calculating and outputting the feedback gain correction coefficient: if the interaction conflict degree feature exceeds the steady-state threshold of 0.75, the gain coefficient of the differential term in the pose closed-loop adjustment loop is increased proportionally according to the logical weights output by the deep reinforcement learning model.

[0014] Preferably, when the drive motor adjustment unit drives multiple robots to move, the gradient of the drive signal of the drive motor adjustment unit is limited to within 10%.

[0015] Preferably, the reward function of the deep reinforcement learning model includes a power compensation term, which is used to quantitatively compensate for the nonlinear friction loss of the motor regulating unit in the low-speed regulation range.

[0016] Preferably, the process of adjusting the drive motor unit also includes link redundancy monitoring: monitoring the communication delay of the data link between each robot; if the communication delay exceeds 50ms, the pose closed-loop adjustment loop automatically switches to the local adjustment mode based on vehicle environment perception.

[0017] Preferably, obtaining the transient acceleration vector fed back by the inertial sensor includes: collecting the three-axis acceleration data of each robot using the vehicle-mounted inertial measurement unit; performing gravity component compensation and low-pass filtering on the three-axis acceleration data to extract the transient acceleration vector reflecting the motion trend of multiple robots.

[0018] Preferably, when outputting the feedback gain correction coefficient, the method also includes obstacle environment adaptation: calculating the environmental obstacle density based on the relative obstacle distance; in areas where the environmental obstacle density is lower than a preset density threshold, reducing the exploration parameters of the deep reinforcement learning model to shorten the pose adjustment response time; and in areas where the environmental obstacle density is higher than a preset density threshold, increasing the exploration parameters to enhance the stability of the adjustment loop.

[0019] Preferably, after driving the movement of multiple robots, the steady-state deviation of the controlled variables relative to the preset trajectory is calculated in real time, and the integral term of the pose closed-loop adjustment loop is corrected based on the steady-state deviation.

[0020] Compared with existing technologies, the multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning in this invention has the following advantages:

[0021] 1. In multi-robot collaborative path planning, a closed-loop adjustment architecture based on dynamic virtual damping is constructed. A deep reinforcement learning model is used to extract the spatiotemporal features of the multi-robot coupled environment and directly map them into the feedback gain correction coefficient in the adjustment loop. This absorbs the discontinuous energy jumps caused by path replanning instructions at the physical level. This mechanism changes the mismatch between traditional planning instructions and the underlying actuators, so that when the controlled robot performs high-frequency obstacle avoidance actions, the evolution curve of its pose variables is always constrained within the preset dynamic envelope. This effectively eliminates the adjustment oscillations induced by instruction steps and the actuator saturation phenomenon, and improves the motion stability of the system in a precision industrial environment.

[0022] 2. By calculating the information entropy components in the multi-machine interaction field, the adjustment stiffness between each controlled node is adjusted in real time, transforming the conflict avoidance behavior between robots from simple geometric avoidance to the parameterized response within the adjustment system. Under this multi-mechanism synergy, the system can automatically trigger nonlinear compensation of the feedback gain based on the velocity gradient tensor in the local neighborhood, so that the robot naturally generates a deceleration trend with physical damping characteristics when approaching obstacles. This process does not rely on the forced modification of discrete given values, but rather on reconstructing the response characteristics of the adjustment loop to transform the dynamic conflict between multiple machines into the smooth dissipation of energy within the system, ensuring that the controlled physical quantities still have low steady-state errors under high-density collaborative conditions.

[0023] 3. By deeply coupling the robustness of classical control theory with the perception capabilities of intelligent algorithms, and by setting communication redundancy and physical layer obstacle avoidance redundancy mechanisms, the reliability of the control system in dealing with non-ideal working conditions is enhanced. When there is a delay or instantaneous packet loss in the data link, the bottom adjustment layer can automatically switch to independent operation mode according to the preset potential field adjustment mode. It uses the transient acceleration vector fed back by the vehicle sensor to maintain basic pose adjustment capability. This layered architecture ensures that the weight matrix generated by the intelligent decision layer is only used as incremental information for optimizing adjustment performance, rather than the only prerequisite for system operation. Thus, while improving the system response accuracy, it avoids the risk of control failure caused by the jitter of a single intelligent model inference to physical entities. Attached Figure Description

[0024] Figure 1 This is a flowchart of the multi-robot cooperative obstacle avoidance method based on deep reinforcement learning of the present invention;

[0025] Figure 2 This is a diagram showing the modular interaction architecture of the multi-robot collaborative obstacle avoidance decision-making system of the present invention. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0027] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0028] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0029] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0030] A multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning includes:

[0031] Step 101: Obtain the pose deviation vector, relative obstacle distance, and transient acceleration vector fed back by the inertial sensor of the controlled multi-robot, and calculate the interaction conflict degree feature that characterizes the deviation intensity of the controlled variables in the pose closed-loop adjustment loop.

[0032] Step 102: Use a pre-trained deep reinforcement learning model to extract the pose deviation vector, interaction conflict degree features, and transient acceleration vector mapping correlation in a multi-machine coupled environment, and calculate and output the feedback gain correction coefficient containing logical weights.

[0033] Step 103: Establish a pose closed-loop control loop that includes proportional, integral and derivative terms. The pose closed-loop control loop uses the pose of multiple robots as the controlled variable.

[0034] Step 104: The feedback gain correction coefficient is mapped to the pose closed-loop adjustment loop in real time. By adjusting the feedback gain of the differential term, the output gradient of the control signal is physically constrained to suppress the discontinuous step of the controlled variable induced by path replanning.

[0035] Step 105: Based on the control signal after differential gain correction, a drive current is generated to drive the motor adjustment unit to move the multiple robots, so that when the multiple robots respond to obstacle avoidance actions, the cumulative variance of their pose evolution curve relative to the preset trajectory is less than 0.05 within a 10ms sampling period.

[0036] Preferably, step 101 is further refined as follows: step 1011, collecting the relative pose vector and relative velocity vector between each controlled robot; step 1012, calculating the interaction conflict degree feature reflecting the adjustment pressure based on the rate of change of the relative pose vector and relative velocity vector in the time series; step 1013, determining whether the interaction conflict degree feature exceeds the preset steady-state threshold of 0.75.

[0037] Preferably, when calculating and outputting the feedback gain correction coefficient: if the interaction conflict degree feature exceeds the steady-state threshold of 0.75, the gain coefficient of the differential term in the pose closed-loop adjustment loop is increased proportionally according to the logical weights output by the deep reinforcement learning model.

[0038] Preferably, when the drive motor adjustment unit drives multiple robots to move, the gradient of the drive signal of the drive motor adjustment unit is limited to within 10%.

[0039] Preferably, the reward function of the deep reinforcement learning model includes a power compensation term, which is used to quantitatively compensate for the nonlinear friction loss of the motor regulating unit in the low-speed regulation range.

[0040] Preferably, the process of adjusting the drive motor unit also includes link redundancy monitoring: monitoring the communication delay of the data link between each robot; if the communication delay exceeds 50ms, the pose closed-loop adjustment loop automatically switches to the local adjustment mode based on vehicle environment perception.

[0041] Preferably, obtaining the transient acceleration vector fed back by the inertial sensor includes: collecting the three-axis acceleration data of each robot using the vehicle-mounted inertial measurement unit; performing gravity component compensation and low-pass filtering on the three-axis acceleration data to extract the transient acceleration vector reflecting the motion trend of multiple robots.

[0042] Preferably, when outputting the feedback gain correction coefficient, the method also includes obstacle environment adaptation: calculating the environmental obstacle density based on the relative obstacle distance; in areas where the environmental obstacle density is lower than a preset density threshold, reducing the exploration parameters of the deep reinforcement learning model to shorten the pose adjustment response time; and in areas where the environmental obstacle density is higher than a preset density threshold, increasing the exploration parameters to enhance the stability of the adjustment loop.

[0043] Preferably, after driving the movement of multiple robots, the steady-state deviation of the controlled variables relative to the preset trajectory is calculated in real time, and the integral term of the pose closed-loop adjustment loop is corrected based on the steady-state deviation.

[0044] Example 1: In a high-density logistics sorting center scenario with more than 50 autonomous mobile robots, the system faces a dynamically evolving environment consisting of frequently moving forklifts, randomly moving personnel, and narrow aisles. Multiple robots need to frequently execute transient start-up, high-frequency obstacle avoidance, and emergency braking commands within a confined space of 1.2m width. Because discrete planning commands cause discontinuous steps in the speed setpoint when driving the underlying control loop, the drive motor enters the power saturation region, causing oscillations in the controlled pose variables. To solve this technical problem, the method claimed in this invention collects the pose deviation vector of the controlled robot, the distance to relative obstacles, and the transient acceleration vector fed back by inertial sensors using sensors, and calculates the interaction conflict degree characteristic reflecting the deviation intensity of the controlled variables in the pose closed-loop control loop. ,in , which is a dimensionless eigenvalue characterizing the strength of deviation of the controlled variable.

[0045] pose deviation vector and interaction conflict degree features The model uses a pre-trained deep reinforcement learning model with transient acceleration vector input. The model outputs a feedback gain correction coefficient containing logical weights, which is then mapped in real-time to the pose closed-loop control loop. The feedback gain of the differential term in the pose closed-loop control loop physically constrains the output gradient of the control signal. The feedback gain of the differential term is determined by dynamic virtual damping parameters. Decide, This system is designed to absorb discontinuous energy jumps caused by path replanning, ensuring that the cumulative variance of the pose evolution curve of the controlled robot relative to the preset trajectory is less than 0.05 within a 10ms sampling period when responding to obstacle avoidance actions. Furthermore, the gradient of the drive signal of the motor adjustment unit is limited to within 10%. This transforms the logical conflict avoidance behavior between multiple robots into a parameterized response within the adjustment system, achieving smooth adjustment of pose and velocity vectors under physical inertial constraints.

[0046] Example 2: In a closed test field equipped with a 4-wheel omnidirectional mobile platform, the platform weighs 50 kg and is equipped with a servo motor adjustment unit with a maximum speed of 3000 rpm. A three-axis acceleration sampling unit with a sampling accuracy of [missing information] is mounted at the center. An inertial measurement unit with a sampling frequency of 100Hz was used. The test environment was divided into a 20m × 20m rectangular area, within which dynamic obstacles with random motion vectors were deployed to simulate a high-density sorting center. Three levels of problem intensity gradients were constructed by changing the number of dynamic obstacles per unit area: low-density gradient (2 obstacles per 10 square meters), medium-density gradient (5 obstacles per 10 square meters), and high-density gradient (10 obstacles per 10 square meters). The sampling period was set to balance the real-time performance of non-electrical variable adjustment with the computational load of the controller, and was set to 10ms under the conditions of this experiment. For the acquisition of pose deviation vectors of the controlled multi-robot, an optical positioning system with a measurement accuracy of 5mm was pre-deployed above the test field as an external reference to obtain the real-time position of the controlled robots during movement. With speed Interaction Conflict Characteristics The steady-state threshold of 0.75 is determined according to the calibration process. A unit step velocity command is given to the mobile platform, and the overshoot of its pose evolution curve is measured. When the overshoot reaches 25%, the deviation change rate at this point is recorded, and the dimensionless value is determined as 0.75. During the dimensionless calculation process, the controller extracts the maximum inherent deviation change rate of the system that can be measured when the mobile platform performs emergency braking under rated full load conditions, using it as a standardized divisor benchmark. The controller divides the actual deviation change rate recorded when the overshoot reaches 25% by this benchmark value. The resulting relative ratio directly represents the actual pressure level of the current system approaching the physical control instability critical point, and outputs 0.75 as the threshold for the system to enter steady-state constraints. Boundary conditions were set for the deep reinforcement learning model used in the experiment. This model was built on a residual network architecture, containing three fully connected hidden layers, each with 256 neurons. The input vector was a 12-dimensional tensor containing pose deviation and transient acceleration vectors, and the output was a 1-dimensional feedback gain correction coefficient. This 12-dimensional tensor was precisely constructed from multiple heterogeneous physical parameters through 1-dimensional flattening and vector concatenation operations. Its internal structure included: a pose deviation vector occupying six dimensions, representing the three-dimensional spatial translation error and three-axis Euler angle rotation error in Cartesian coordinates; a transient acceleration vector occupying three dimensions; and scalars representing relative obstacle distance, relative velocity amplitude, and interaction conflict degree features, each occupying one dimension. The extraction and calculation procedure is as follows: obtain the relative distance vector formed by the physical coordinates of the current controlled robot center and the physical coordinates of the nearest dynamic obstacle center, and calculate the vector magnitude. Simultaneously acquire the projection magnitude of the relative velocity vectors of the two objects in the direction of the relative distance vector. The controller sets a conflict time determination constant. The numerical calibration is the theoretical braking time required for the controlled robot to brake from its maximum operating speed to a standstill under maximum workload. The controller's arithmetic logic unit calculates this time using the formula... Obtain the reciprocal of the collision time and the constant. The reciprocal is taken as the difference, and the logic comparator checks the difference. If the difference is greater than zero, the difference is compared with the dimensionless penalty coefficient. Perform a product operation to obtain Output a scalar; if it is less than or equal to zero, Assigning a value of zero, penalty coefficient The value is the ratio of the robot's current total mass to its unloaded mass. The calculation process is executed periodically every 10ms sampling period, and the output is... Scalar data is stored in a buffer register, forming the independent dimension of the input tensor of the deep reinforcement learning model.

[0047] The experimental group used the method claimed in this invention to determine the interaction conflict degree feature in a low-density gradient environment. When fluctuating between 0.15 and 0.35, the feedback gain correction coefficient of the deep reinforcement learning model output affects the dynamic virtual damping parameter in the proportional-derivative control loop. Maintaining a value between 0.22 and 0.31, the cumulative variance of the robot's pose evolution curve relative to the preset trajectory is 0.016. As the environment switches to a medium-density gradient, frequent intersections between obstacles lead to increased interaction conflict characteristics. After repeatedly touching the steady-state threshold of 0.75, the system increased the physical constraint strength of the control loop by proportionally increasing the feedback gain of the differential term. The driving current gradient of the motor control unit was observed to be stably limited to within 7.2%, and the cumulative variance rose to 0.028, demonstrating the steady-state maintenance capability of the controlled physical quantity under increasing conflict. The physical mapping and constraint steps from the feedback gain correction coefficient to the pose closed-loop control loop are as follows: the deep reinforcement learning model's end is processed by a hyperbolic tangent function, and the output value is limited to the original logic weight scalar between -1 and 1. The controller calls the internal multiplier to multiply the logic weight scalar with the servo motor driver damping gain safety factor, extracting the actual correction increment. The damping gain safety factor is set to 10% of the differential gain extreme value corresponding to the maximum allowable driving current gradient limit of the motor. The actual correction increment is numerically summed with the initial feedback gain of the differential term in the pose closed-loop control loop. During the above gain summation process, a first-order low-pass filter is essentially connected in series at the front end of the differential control branch of the pose closed-loop control loop. The filtering algorithm module blocks and filters out high-frequency displacement noise components induced by high-frequency path replanning command jumps and higher than the inherent mechanical resonant frequency of the motor before the pose deviation signal is introduced into the pure differential operation stage. Based on the smooth error rate of change vector of the filtered output, the adjustment loop applies the updated differential term feedback gain for multiplication and accumulation, thereby simply changing the damping distribution of the dominant pole of the control system. This converts discontinuous step energy into smooth frictional heat dissipation without injecting high-frequency current harmonics into the physical entity. Before the control unit executes the parameter overwrite action, the logic prediction is intervened to detect whether the theoretical gradient of the motor control signal driven by the actual differential gain parameter exceeds the 10% physical limit. If it is determined to exceed the limit, the controller cuts off the update command, keeps the differential gain parameter of the previous control cycle unchanged, and cuts off the transmission path of the control command step to the underlying mechanical mechanism. When entering the high-density gradient pressure test scenario, multiple robots and dynamic obstacles generate continuous coupling interference in narrow areas, and the interaction conflict degree characteristics are shown. If the value remains above 0.85, the system will continuously adjust the dynamic virtual damping parameter in real time. The variance of the pose evolution trajectory was reduced to 0.65 to absorb the discontinuous energy jumps caused by discrete path replanning. The measured data showed that the cumulative variance of the pose evolution trajectory eventually stabilized at 0.043, which is lower than the preset accuracy threshold of 0.05. In contrast, the control group, which lacked the dynamic virtual damping compensation mechanism, experienced multiple gradient jumps of more than 35% in the motor drive signal under the same high-density environment because the underlying adjustment loop could not suppress the control signal step induced by the high-frequency obstacle avoidance command. This caused physical vibration in the mechanical transmission mechanism of the controlled robot, and the cumulative variance of its pose evolution curve increased to 0.142. Moreover, an average end position deviation of 12cm appeared during precision docking. This data confirms the effectiveness of the present invention in solving the mismatch between logical decision-making and physical execution at the physical mechanism level.

[0048] Performance inflection point tests show that when the interaction conflict degree characteristic After exceeding the overload zone of 1.15, although the deep reinforcement learning model continued to output commands to increase damping, the cumulative variance of the pose evolution curve began to show a non-linear growth trend due to the servo motor adjustment unit approaching its maximum rated torque output limit. This confirmed that the feedback gain correction logic is based on the optimal operating window within the physical power envelope of the actuator. When the simulated communication delay reached 65ms, the system triggered the link redundancy monitoring mechanism, and the pose closed-loop adjustment loop automatically switched to the local adjustment mode based on the vehicle-mounted LiDAR. At this time, the transient acceleration vector fed back by the inertial measurement unit was used to maintain basic obstacle avoidance capability through the physical potential field compensation algorithm. The measured cumulative variance of the pose recovered to 0.075 but no physical collision occurred, verifying the safety of the control scheme in a non-ideal communication environment. By monitoring the power loss of the motor adjustment unit in the low-speed adjustment range below 0.2m / s, it was found that the non-linear friction loss of the experimental group with the power compensation term was reduced by 12.8% compared with the uncompensated control group. The power compensation term in the reward function of the deep reinforcement learning model... The quantitative calibration procedure is as follows: During the offline initialization phase of the motor regulating unit, an initial value of 0.01A is injected into the stator winding, and the test current is increased in steps according to the set step size. The amplitude of the drive current when the rotor transitions from a stationary state to the critical state of physical displacement is recorded. During the process of the controlled robot responding to obstacle avoidance commands, when the transient velocity amplitude reported by the inertial measurement unit is lower than the upper limit of the set low-speed range of 0.2 m / s, the controller reads the actual driving current from the current sensor in real time. If the actual driving current The value is less than the critical drive current amplitude The controller is based on the relational formula The non-negative friction loss scaling parameter is calculated and added to the negative feedback term of the reward function of the deep reinforcement learning model. The parameter is introduced to shift the logic weight of the model output. The command pose closed-loop adjustment loop increases the output current integral compensation in this low-speed range until the continuous position displacement pulses fed back by the motor encoder are detected. This shows that adding a mechanical loss constraint term to the reward function of the deep reinforcement learning model can induce the system to generate control commands that are more in line with the laws of physical evolution, thereby improving the pose adjustment accuracy while reducing the second-order physical loss of the actuator.

[0049] This experiment, through gradient comparisons at three intensities and comparisons with targeted component missing parameters, confirms the dynamic virtual damping parameters based on deep reinforcement learning output. It can absorb the kinetic energy mutation caused by path replanning, and maintain the smoothness of the pose evolution curve of the controlled non-electric variables under dynamic high-frequency intervention. It realizes the mechanism fusion of discrete decision logic and continuous physical adjustment loop, and transforms the conflict avoidance in the multi-robot cooperation process from geometric avoidance to the parameterized energy balance process inside the adjustment system. Thus, it improves the control stability and trajectory tracking accuracy of the multi-robot cooperative system without changing the hardware frequency response bandwidth.

[0050] Example 3: This example combines Figures 1 to 2 This section describes a multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning, such as... Figure 1 As shown, step 101 is to collect the pose and acceleration of the multi-robot and calculate the conflict features; step 102 is to output the gain correction coefficient using the reinforcement learning model; step 103 is to establish an adjustment loop with the pose of the multi-robot as the controlled variable; step 104 is to adjust the gradient of the gain constraint signal by mapping the correction coefficient; and step 105 is to generate a control current to drive the multi-robot to move and avoid obstacles so that the cumulative variance of the pose evolution curve of the multi-robot relative to the preset trajectory when responding to the obstacle avoidance action is less than 0.05 within a 10ms sampling period.

[0051] like Figure 2 As shown, the system mainly includes an onboard inertial measurement unit and a motor adjustment unit placed within rectangular frames with double vertical lines on both sides, a deep reinforcement learning model placed within a parallelogram frame, and a pose closed-loop adjustment loop placed within a document frame with a wavy line at the bottom. The onboard inertial measurement unit is connected by solid lines to the triaxial acceleration data acquisition node and the transient acceleration vector extraction node, both using elliptical frames. The triaxial acceleration data acquisition node is pointed to the transient acceleration vector extraction node by a dashed one-way arrow. The deep reinforcement learning model is connected by solid lines to the interaction conflict degree calculation feature node, the environment mapping association node, and the output gain correction coefficient node, all using elliptical frames. The interaction conflict degree calculation feature node is pointed to the environment mapping association node by a dashed one-way arrow. Mapping associated nodes: The environment mapping associated nodes are pointed to the output gain correction coefficient node via dashed one-way arrows. The adjustment differential term feedback gain node and the constraint control signal gradient node, which are represented by elliptical boxes, are directly associated to the pose closed-loop adjustment loop via solid lines. The adjustment differential term feedback gain node is pointed to the constraint control signal gradient node via dashed one-way arrows. The compensation nonlinear friction loss node, the drive controlled multi-robot motion node, and the switch local adjustment mode node, which are represented by elliptical boxes, are directly associated to the motor adjustment unit via solid lines. The compensation nonlinear friction loss node is pointed to the drive controlled multi-robot motion node via dashed one-way arrows. The drive controlled multi-robot motion node is pointed to the switch local adjustment mode node via dashed one-way arrows.

[0052] Example 4: In a multi-robot operating environment equipped with a permanent magnet synchronous motor with a rated power of 400W and a rated torque of 1.27 N⋅m, the actuator of the controlled robot experiences increased mechanical backlash and nonlinear frictional force fluctuations with increasing service time. Furthermore, the inertial sensor it carries exhibits a standard deviation of [missing information - likely a value] due to electromagnetic interference. The random noise causes the fixed gain adjustment loop to lag in response to discrete obstacle avoidance commands, resulting in oscillations in the pose evolution curve at the obstacle avoidance entry point, and the cumulative variance is likely to exceed the 0.05 threshold. The method determines the physical boundary of the motor through an initialization process, that is, by reading the current loop sampling frequency of the motor controller (10kHz) and the speed loop adjustment period (1ms), and determining the power saturation zone coefficient based on the ratio of the motor's rated current to the maximum stall current. The deep reinforcement learning model receives a 12-dimensional normalized tensor composed of pose deviation vector, relative obstacle distance, and transient acceleration vector as input. The model is transformed through three fully connected layers, each containing 256 neurons and activated using a linear rectified function. Finally, the output layer generates an original weight scalar with a value between -1 and 1 through a hyperbolic tangent function. .

[0053] Before being deployed in the actual control loop for real-time inference, the model relies on a pre-configured proximal policy optimization algorithm to complete model training in a multi-machine collaborative offline simulation environment. In this reinforcement learning Markov decision framework, the environment state space is defined as the aforementioned 12-dimensional normalized feature tensor, and the action space is set as a continuous interval of original weight scalars. The algorithm's calculation process constructs a cumulative reward function with the objectives of minimizing the cumulative variance of the multi-machine relative to the preset trajectory and minimizing motor friction loss. The weights of neurons in the fully connected layer are continuously updated in iterations using the gradient ascent method until the change in policy gradient between adjacent training rounds is less than a set extreme value. At this point, the network topology and its internal parameters are solidified, forming a definite multi-dimensional mapping relationship. The controller adjusts the original weight scalars according to the linear mapping logic. Converted to feedback gain correction coefficient, i.e., when When the value is greater than 0, the differential term gain in the pose closed-loop adjustment circuit will be... Adjust to initial value The sum of the correction increments, where the correction increments are The product of this and a preset step size of 0.5 increases the damping strength of the regulating loop, thus determining the interaction conflict characteristics applicable to the current motor state. To determine the steady-state threshold, the system uses a gain margin scanning program to lock parameters. Specifically, when the robot is stationary, the controller sequentially injects sinusoidal disturbance signals with frequencies ranging from 1Hz to 50Hz and amplitudes of 5% of the rated voltage into the motor, and monitors the output residual of the pose closed-loop adjustment loop in real time. When the rate of change of the residual reaches a local maximum, the interaction conflict characteristic value calculated from the pose deviation vector is recorded, and multiplied by a safety margin coefficient of 0.8. This determines the steady-state threshold under the current working condition to be 0.75.

[0054] When the interaction conflict degree feature When the value exceeds 0.75, the method increases the differential feedback gain proportionally to put the system into a high-damping state. At this time, the measured motor drive current gradient drops from 25.4% to 8.8%, avoiding the problem of frequent triggering of the limit protection due to control signal jumps. Even when the noise reference of the inertial sensor increases, the cumulative variance of the pose evolution curve relative to the preset trajectory remains at 0.032 for 500 consecutive sampling periods through the extraction of multi-machine coupling features by the deep reinforcement learning model, ensuring that the controlled robot has physical adjustment stability under non-ideal hardware conditions.

[0055] Example 5: In an industrial setting where multi-robot collaborative path planning and dynamic obstacle avoidance decision-making methods are deployed to drive modules with different moments of inertia and frictional characteristics, the system faces feedback adjustment pole shifts caused by individual performance differences in the drive motors. To determine the mapping benchmark for the differential term gain in the pose closed-loop adjustment loop and establish the input normalization boundary for the deep reinforcement learning model, the method initiates an initialization physical characteristic calibration process before the robot enters the working state. The controller sends a triangular wave velocity command with a period of 500ms and an amplitude of 10% of the rated speed to the motor adjustment unit. Sensors collect the transient acceleration vector of the controlled robot in an obstacle-free environment to calculate the mechanical time constant of the motor. The interaction conflict degree feature is determined based on the collected pose deviation change rate. The dynamic range is linearly scaled, and the scaled values ​​are used as feature inputs to the deep reinforcement learning model to ensure that drive motors with different physical parameters have consistent control response gradients when receiving the same feedback gain correction coefficient.

[0056] When a controlled robot faces obstacle avoidance conflicts caused by the coupling of multiple dynamic obstacles, and the measured interaction conflict degree characteristics are... When the value reaches 0.75, the method retrieves the calibration parameters pre-stored in the register and applies them to the original weight scalar of the deep reinforcement learning model output. Implement gain limiting and obtain the mechanical time constant based on the initial calibration. The upper limit of the step size change rate of the feedback gain correction coefficient is determined, and the dynamic virtual damping parameter is increased by adjusting the feedback gain of the differential term in the pose closed-loop adjustment loop. The physical effect limits the output gradient of the control signal to within the rated current envelope of the motor adjustment unit. Data monitoring shows that after introducing the above-mentioned initial physical characteristic calibration, the motor saturation frequency of the controlled robot decreases by 15.3% when performing obstacle avoidance and turning. The cumulative variance of the pose evolution curve relative to the preset trajectory remains stable below 0.05 within a 10ms sampling period. The discrete planning instruction and the continuous physical adjustment process achieve an adaptive mechanism fusion.

[0057] Example 6: In an industrial deployment scenario involving multiple drive modules with heterogeneous physical characteristics, the controller, in order to obtain the intrinsic damping response benchmark of each actuator, executes a standardized physical property calibration procedure before operation. This involves sending a step excitation signal with a pulse width of 100ms and an amplitude of 15% of the rated speed to each motor adjustment unit. Simultaneously, the sensors acquire the transient acceleration vector of the mobile robot, and the response delay time of the actuator is determined by calculating the time difference between the peak moment of the acceleration curve and the start moment of the excitation signal. The controller determines the initial value of the differential term gain based on linear decision logic. When the response delay time When it is in the range of 5ms to 15ms, The value is set to 1.25, and this value is used as the calibration zero for calculating the feedback gain correction coefficient in subsequent deep reinforcement learning models.

[0058] When the system operates under conditions where fluctuations in the wireless communication link signal quality cause a non-linear increase in data packet transmission delay, the controller monitors the data link delay time of each controlled node in real time. Implementation status determination, among which For the time span from pose data back to control command issuance, if the logic comparator determines the delay time within 3 consecutive sampling periods... If the preset judgment threshold of 50ms is exceeded, the controller cuts off the weight optimization layer of the deep reinforcement learning model and switches the pose closed-loop adjustment loop to the vehicle environment perception adjustment mode. At this time, the system calculates the virtual repulsion gradient based on the local obstacle distance vector obtained by the vehicle lidar and directly corrects the driving command gradient, so that the cumulative variance of the controlled pose variable relative to the preset trajectory is less than 0.08 under the condition of no cooperative data input, thus realizing physical layer safety obstacle avoidance in non-ideal network environment.

[0059] In a controlled multi-robot collaborative material transfer station, the controlled robots face the physical constraint of drastic changes in obstacle distribution during the transition from a low-density storage area to a high-density sorting area. The method calculates the environmental obstacle density based on the collected relative obstacle distances. ,in To characterize the dimensionless parameter of obstacle distribution density per unit area, when When the density falls below a preset threshold of 0.5, the controller will adjust the exploration parameters of the deep reinforcement learning model. Reducing it to 0.05 improves pose adjustment speed by more than 12% by shortening the response delay of the feedback gain correction coefficient; while when When the density exceeds the aforementioned density threshold of 0.5, the system will explore parameters. Increasing it to 0.2 correspondingly limits the fluctuation range of the differential term gain in the pose closed-loop adjustment loop to the initial value. Within 20%, the controlled robot exhibits a deceleration trend with physical damping characteristics during dense obstacle avoidance, ensuring that the cumulative variance of the pose evolution curve relative to the preset trajectory is always less than 0.05 within a 10ms sampling period. When the motor adjustment unit of the controlled robot operates in the low-speed nonlinear friction range below 0.1m / s, a power compensation term determined by the power consumption monitoring operator is introduced into the reward function of the deep reinforcement learning model. ,in To quantify the characteristic operator for motor friction loss, the system obtains nonlinear friction force characteristic values ​​by online correlation of motor stator current and transient acceleration vector, and converts them into power compensation coefficients. The controller adjusts the feedback gain of the differential term based on the feedback gain correction coefficient to correct the drive signal gradient. Data monitoring shows that after performing 50 consecutive start and stop actions, the motor regulation unit with power compensation term reduces effective power consumption by 14.6%, and the instantaneous acceleration fluctuation amplitude is stably constrained to 1.5m / s². Within this framework, the obstacle avoidance decision weights of the logic layer are accurately mapped to the physical adjustment parameters of the execution layer.

[0060] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.

Claims

1. A multi-robot cooperative path planning and dynamic obstacle avoidance decision-making method based on deep reinforcement learning, characterized in that, include: Step 101: Obtain the pose deviation vector, relative obstacle distance, and transient acceleration vector fed back by the inertial sensor of the controlled multi-robot, and calculate the interaction conflict degree feature that characterizes the deviation intensity of the controlled variables of the pose closed-loop adjustment loop. Step 102: Use a pre-trained deep reinforcement learning model to extract the mapping and correlation of pose deviation vector, interaction conflict degree features and transient acceleration vector in a multi-machine coupled environment, and calculate and output feedback gain correction coefficients containing logical weights. Step 103: Establish a pose closed-loop control loop that includes proportional, integral and derivative terms. The pose closed-loop control loop uses the pose of multiple robots as the controlled variable. Step 104: The feedback gain correction coefficient is mapped to the pose closed-loop adjustment loop in real time. By adjusting the feedback gain of the differential term, the output gradient of the control signal is physically constrained to suppress the discontinuous step of the controlled variable induced by path replanning. Step 105: Based on the control signal after differential gain correction, a drive current is generated to drive the motor adjustment unit to move the multiple robots, so that when the multiple robots respond to obstacle avoidance actions, the cumulative variance of their pose evolution curve relative to the preset trajectory is less than 0.05 within a 10ms sampling period.

2. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, Step 101 is further refined as follows: Step 1011, collect the relative pose vector and relative velocity vector between each controlled robot; Step 1012, calculate the interaction conflict degree feature reflecting the adjustment pressure based on the rate of change of the relative pose vector and relative velocity vector in the time series; Step 1013, determine whether the interaction conflict degree feature exceeds the preset steady-state threshold of 0.

75.

3. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 2, characterized in that, When calculating and outputting the feedback gain correction coefficient: if the interaction conflict degree feature exceeds the steady-state threshold of 0.75, the gain coefficient of the differential term in the pose closed-loop adjustment loop is increased proportionally according to the logical weights output by the deep reinforcement learning model.

4. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, When the drive motor adjustment unit moves multiple robots, the gradient of the drive signal of the drive motor adjustment unit is limited to within 10%.

5. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, The reward function of the deep reinforcement learning model includes a power compensation term, which is used to quantify and compensate for the nonlinear friction loss of the motor regulating unit in the low-speed regulation range.

6. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, The process of adjusting the drive motor unit also includes link redundancy monitoring: monitoring the communication delay of the data link between each robot; if the communication delay exceeds 50ms, the pose closed-loop adjustment loop will automatically switch to the local adjustment mode based on the vehicle environment perception.

7. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, Acquiring transient acceleration vectors fed back by inertial sensors includes: collecting three-axis acceleration data of each robot using an onboard inertial measurement unit; performing gravity component compensation and low-pass filtering on the three-axis acceleration data to extract transient acceleration vectors that reflect the motion trends of multiple robots.

8. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, When outputting the feedback gain correction coefficient, obstacle environment adaptation is also included: calculating the environmental obstacle density based on the relative obstacle distance; in areas where the environmental obstacle density is lower than a preset density threshold, reducing the exploration parameters of the deep reinforcement learning model to shorten the pose adjustment response time; and in areas where the environmental obstacle density is higher than a preset density threshold, increasing the exploration parameters to enhance the stability of the adjustment loop.

9. The method for multi-robot cooperative path planning and dynamic obstacle avoidance decision-making based on deep reinforcement learning according to claim 1, characterized in that, After driving multiple robots to move, the steady-state deviation of the controlled variables relative to the preset trajectory is calculated in real time, and the integral term of the pose closed-loop adjustment loop is corrected based on the steady-state deviation.

Images