Transform-based reinforcement learning multi-robot navigation path planning method

Abstract

The invention discloses a reinforcement learning multi-robot navigation path planning method based on Transform, and belongs to the technical field of multi-robot autonomous navigation. The method adopts a closed-loop architecture combining global planning, state optimization and dynamic adjustment, and comprises the following steps: firstly, collecting and preprocessing environment and robot state data, and constructing a system state and an observation vector; through a Transformer network comprising a path encoder, an environment encoder and a cooperation decoder, extracting a global feature and robot cooperation relationship, and combining a reinforcement learning reward mechanism to generate an optimal initial path; then position and speed estimation is corrected in real time through a Kalman filtering model, and noise and motion deviation of the sensor are restrained; and finally, responding to environment change through a local re-planning strategy, and continuously optimizing the path in combination with closed-loop feedback. The method has decentration flexibility and global optimization capability, and is suitable for complex dynamic scenes such as warehouse logistics and intelligent manufacturing.

Description

Technical Field

[0001] This invention relates to the field of multi-robot autonomous navigation technology, and in particular to a reinforcement learning-based multi-robot navigation path planning method based on Transformer. Background Technology

[0002] Multi-robot path planning is a core technology for enabling collaborative operation of multi-robot systems. Its goal is to plan an efficient, collision-free path from the initial position to the target position for each robot. In real-world scenarios such as warehousing and logistics, intelligent manufacturing, and emergency rescue, multi-robot systems often face challenges such as complex environments, high robot density, and sensor noise interference, placing high demands on the robustness and collaborative nature of path planning methods.

[0003] Traditional multi-robot path planning methods are mainly divided into two categories: centralized and decentralized. Centralized methods plan the paths of all robots through a central controller. Although this can guarantee global optimization, the computational complexity increases exponentially with the number of robots, making it difficult to adapt to large-scale systems. Decentralized methods allow each robot to plan its path independently, reducing computational complexity, but they lack effective cooperation between robots and are prone to collisions or deadlocks in dense environments.

[0004] In recent years, path planning methods based on reinforcement learning and deep learning have received widespread attention. Reinforcement learning acquires optimal policies through trial and error and can adapt to dynamic environments, but it suffers from unstable training and slow convergence. Transformer networks, with their multi-head self-attention mechanism, can effectively capture global dependencies, enabling implicit collaboration among robots. However, when directly applied to path planning, they are susceptible to sensor noise and motion errors, leading to decreased path execution accuracy. Kalman filtering, as a classic state estimation method, can effectively fuse multi-source sensor data and suppress noise interference, but it lacks global path planning capabilities.

[0005] Therefore, how to combine the global collaborative planning capability of Transformer with the state optimization capability of Kalman filtering to solve the problems of multiple robots lacking communication and cooperation and weak path planning noise resistance in dense environments has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a Transformer-based reinforcement learning multi-robot navigation path planning method to solve the problems of weak collaboration and poor noise interference resistance in the existing technology.

[0007] To achieve the above objectives, this invention provides a Transformer-based reinforcement learning method for multi-robot navigation path planning, comprising the following steps: S1. Data Acquisition and Preprocessing: Acquire the robot's initial position, target position, environmental obstacle information, and the robot's own motion state data; construct the system state vector and observation vector; and perform noise reduction and normalization preprocessing on the raw data. S2. Initial path generation based on Transformer: Construct a Transformer path planning network, extract global environmental features and robot collaboration features through the encoder, generate several candidate paths through the decoder, and use the reward mechanism of reinforcement learning to select the optimal initial path; S3. Kalman Filter State Optimization: Establish a Kalman filter model of the robot's motion state, and correct the robot's position and velocity estimates in real time based on the observation data to compensate for sensor noise and motion deviation; S4. Dynamic Path Adjustment: Based on the optimized state output by the Kalman filter and combined with information on dynamic environmental changes, the initial path is adjusted in real time to ensure collision-free and timely path operation. S5. Path Execution and Feedback: The robot executes the optimized path, collects motion state data and environmental feedback information in real time, and forms a closed-loop iterative optimization.

[0008] Preferably, in S1, the system state vector is defined as: ; in, n For the number of robots, , They represent the first i A robot k Moment x , y Direction and position coordinates , They represent the first i A robot k Moment x , y Directional velocity component; The observation vector is defined as: ; in, , , , The first i The robot obtains position and velocity data through sensors.

[0009] Preferably, in S2, the Transformer path planning network includes a path encoder, an environment encoder, and a cooperative decoder: Path encoder: Represents the path sequence of each robot as... B × N × D The three-dimensional tensor is used to enhance the feature dimension through a multilayer perceptron, and three layers of multi-head self-attention modules are stacked to capture path global dependency features. Environmental encoder: Encodes obstacle information and robot position information into environmental feature vectors, and highlights key obstacle constraints through spatial attention mechanism; Collaborative Decoder: It integrates path features and environmental features, models the implicit collaborative relationships between robots through a cross-attention mechanism, and outputs multiple candidate paths.

[0010] Preferably, in S2, the reward function for reinforcement learning is designed as follows: ; in, The robot is rewarded when it reaches the target location, and penalized according to its distance from the target if it fails to reach it. Collision penalty: A penalty is imposed when the robot collides with an obstacle or another robot; As an efficiency bonus, rewards are given based on path length, with shorter paths yielding higher rewards; To smooth out rewards, rewards are given based on the curvature changes of adjacent path points; the smaller the curvature change, the higher the reward. The specific reward value is set according to the importance of the project.

[0011] Preferably, in S3, the Kalman filter model includes a state prediction equation and an observation update equation: State prediction equation: ; ; Observational update equation: ; ; ; in, for k The prior state estimation vector at time step is based on k The optimal state at time -1 is predicted. k The robot is in constant state and has not yet merged. k Real-time observation data; for k The posterior state estimate vector at time -1 is a fusion kThe optimal state estimate obtained after observing the data at time -1 is the prediction. k The basis of moment-state; for k The prior state covariance matrix at time t. for k The posterior state covariance matrix at time -1; F k This is the state transition matrix, i.e., the robot's motion model; B k To control the input matrix, u k To control the input vector, H k For the observation matrix, Q k The process noise covariance matrix is... R k To observe the noise covariance matrix, K k For Kalman gain, Z k for k The observation vector at time t, for k The posterior state estimate vector at time t. for k The posterior state covariance matrix at time t. I It is the identity matrix. w k-1 This is process noise.

[0012] Preferably, in S4, the dynamic path adjustment adopts a local replanning strategy. When the deviation between the position after Kalman filtering correction and the initial path exceeds a set threshold or a new obstacle is detected, local path replanning is triggered, and a locally optimal path segment is quickly generated through the Transformer network.

[0013] In accordance with the path planning methods described above, this invention also provides a Transformer-based reinforcement learning multi-robot navigation path planning system, comprising: Data acquisition module: Composed of LiDAR, which has a built-in odometer and GPS, used to collect robot motion status data and environmental obstacle data; Preprocessing module: Denoises and normalizes the collected data, and constructs the system state vector and observation vector; Transformer path generation module: Constructs a Transformer network and combines reinforcement learning to generate an initial optimal path; Kalman filter optimization module: Establishes a motion state filtering model and corrects the robot's position and velocity estimates in real time; Dynamic adjustment module: Adjusts the initial path in real time based on the filtering results and environmental changes; Execution and Feedback Module: Controls the robot's execution path and collects feedback data to form a closed-loop optimization.

[0014] A storage medium storing a computer program, which, when executed by a processor, implements the aforementioned Transformer-based reinforcement learning multi-robot navigation path planning method. A computer device including a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the aforementioned Transformer-based reinforcement learning multi-robot navigation path planning method.

[0015] Therefore, the present invention employs the above-mentioned Transformer-based reinforcement learning multi-robot navigation path planning method, which has the following beneficial effects: 1) By capturing the implicit cooperative relationships between robots through Transformer networks, collaborative obstacle avoidance can be achieved under conditions of no communication, thereby improving the success rate of path planning in multi-robot systems; 2) Introduce the Kalman filter algorithm to estimate and correct the robot's motion state in real time, suppress the influence of sensor noise and motion error, and improve the accuracy and robustness of path execution; 3) Design a multi-dimensional reward function that includes target arrival, collision penalty, efficiency reward and smoothness reward to guide the robot to generate an efficient and smooth optimal path; 4) It adopts a local replanning strategy to quickly respond to dynamic changes in the environment, ensuring the timeliness and safety of the path, and is suitable for multi-robot collaborative operations in complex dynamic environments.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is the Transformer path planning network workflow of this invention embodiment; Figure 3 This is a schematic diagram of the Kalman filter state optimization process according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example 1 This invention provides a Transformer-based reinforcement learning method for multi-robot navigation path planning, the process of which is as follows: Figure 1 As shown, the process includes five steps: data acquisition and preprocessing, initial path generation based on Transformer, Kalman filter state optimization, dynamic path adjustment, and path execution and feedback.

[0021] S1. Data Acquisition and Preprocessing: Acquire the robot's initial position, target position, environmental obstacle information, and the robot's own motion state data; construct the system state vector and observation vector; and perform noise reduction and normalization preprocessing on the raw data. In this embodiment, environmental data and robot state data of a multi-robot system are collected using LiDAR. Environmental data includes the position, shape, and size of obstacles; robot state data includes the initial position, target position, real-time position, and velocity of each robot. The collected raw data is preprocessed: Gaussian filtering is used to remove sensor noise, coordinate transformation unifies all data to a global coordinate system, and position and velocity data are normalized to construct system state vectors and observation vectors. Specifically, this embodiment constructs a 10m×10m simulation environment with 3 robots. The initial and target positions of each robot are randomly generated. The obstacles are 5 cubes with a side length of 1m, evenly distributed in the gaps between shelves to simulate a complex passage environment. All obstacles are equipped with collision detection attributes, and the safe distance threshold between the obstacles and the robots is set to 0.3m.

[0022] The system state vector is defined as: ; in, n For the number of robots, , They represent the first iA robot k Moment x , y Direction and position coordinates , They represent the first i A robot k Moment x , y Directional velocity component; The observation vector is defined as: ; in, , , , The first i The robot obtains position and velocity data through sensors.

[0023] S2. Initial Path Generation Based on Transformer: Construct a Transformer path planning network. The encoder extracts global environmental features and robot collaboration features, while the decoder generates several candidate paths. The optimal initial path is then selected using a reinforcement learning reward mechanism. The specific process is as follows: Figure 2 As shown; the Transformer path planning network consists of three parts: a path encoder, an environment encoder, and a cooperative decoder. Path encoder: Represents the path sequence of each robot as... B × N × D The three-dimensional tensor (where B For batch size, N The number of path points. D (For path point feature dimensions), the feature dimensions are transformed from... D Project to C The system uses a stacked 3-layer multi-head self-attention module to capture global dependencies between path points and output a path feature tensor.

[0024] Environmental encoder: Encodes obstacle information into M × K The feature matrix (where M The number of obstacles, K (For obstacle feature dimension), the association weight between each robot and the obstacle is calculated through a spatial attention mechanism to highlight key obstacle constraints and generate an environmental feature vector.

[0025] Collaborative Decoder: Employs a cross-attention mechanism to fuse path features and environmental features, modeling implicit collaborative relationships between robots and outputting multiple candidate paths. A reinforcement learning reward function is used to evaluate these candidate paths and select the optimal initial path. The reinforcement learning reward function is designed as follows: ; in, The robot is rewarded when it reaches the target location, and penalized according to its distance from the target if it fails to reach it. Collision penalty: A penalty is imposed when the robot collides with an obstacle or another robot; As an efficiency bonus, rewards are given based on path length, with shorter paths yielding higher rewards; To smooth out rewards, rewards are given based on the curvature changes of adjacent path points; the smaller the curvature change, the higher the reward. The specific reward value is set according to the importance of the project.

[0026] In this embodiment, both the path encoder and decoder employ a 3-layer multi-head self-attention module with 4 attention heads and a hidden layer dimension of 256; the experience replay buffer size is 100,000, the batch size is 64, and the learning rate is 1e. -4 .

[0027] S3. Kalman Filter State Optimization: Establish a Kalman filter model for the robot's motion state, and correct the robot's position and velocity estimates in real time based on observation data to compensate for sensor noise and motion deviations. The process is as follows: Figure 3 The Kalman filter model includes a state prediction equation and an observation update equation: State prediction equation: ; ; Observational update equation: ; ; ; in, for k The prior state estimation vector at time step is based on k The optimal state at time -1 is predicted. k The robot is in constant state and has not yet merged. k Real-time observation data; for k The posterior state estimate vector at time -1 is a fusion k The optimal state estimate obtained after observing the data at time -1 is the prediction. k The basis of moment-state; for k The prior state covariance matrix at time t. for k The posterior state covariance matrix at time -1; F k This is the state transition matrix, i.e., the robot's motion model; B k To control the input matrix, u k To control the input vector, H k For the observation matrix, Q k The process noise covariance matrix is... R k To observe the noise covariance matrix, K k For Kalman gain, Z k for k The observation vector at time t, for k The posterior state estimate vector at time t. for k The posterior state covariance matrix at time t. I It is the identity matrix. w k-1 This is process noise.

[0028] The specific execution process is as follows: State prediction: Based on the robot's motion model, calculate using the state prediction equation. k Prior state estimation at time 1 With the prior covariance matrix The state transition matrix is... F k A constant-velocity motion model is used for design, and the control input matrix is... B k The process noise covariance matrix is ​​determined based on the robot's motion control commands. Q k Set it to a diagonal matrix with diagonal elements of 0.01.

[0029] Observation Update: Calculating Kalman Gain K k Using observation vectors Z k The prior state estimate is corrected to obtain the posterior state estimate. With the posterior covariance matrix The observation matrix H k The identity matrix and the observation noise covariance matrix are given. Rk Set it to a diagonal matrix with diagonal elements of 0.02.

[0030] S4. Dynamic Path Adjustment: Based on the optimized state output by the Kalman filter and combined with dynamic environmental change information, the initial path is adjusted in real time to ensure collision-free and timely path adjustment. The dynamic path adjustment adopts a local replanning strategy. When the deviation between the position corrected by the Kalman filter and the initial path exceeds a set threshold or a new obstacle is detected, local path replanning is triggered, and the local optimal path segment is quickly generated through the Transformer network.

[0031] In this embodiment, based on the optimized state of the Kalman filter output... The deviation between the robot's actual position and the initial path is calculated. When the deviation exceeds the threshold δ=0.5m, or when a new obstacle is detected by the LiDAR, local path replanning is triggered: taking the robot's current optimized position as the starting point and the target position as the ending point, obstacle information of the local environment is extracted; the Transformer path planning network is used to quickly generate a locally optimal path segment; and the local path segment is smoothly spliced ​​with the original path to form a new feasible path.

[0032] S5. Path Execution and Feedback: The robot executes motion commands according to the optimized path, and collects motion state data and environmental feedback information in real time through sensors. The feedback data is input into the Kalman filter model in S3 to update the state estimate; at the same time, the feedback information is used for reinforcement learning experience playback to continuously optimize the parameters of the Transformer path planning network, forming a closed-loop iterative optimization.

[0033] In this embodiment, 30 independent experiments were conducted, with a maximum runtime of 200 seconds for each experiment. Failure to reach the target within this time was considered a failure. The final results are shown in the table below: Table 1 Performance Comparison of Different Methods

[0034] In this embodiment, the method exhibits optimal overall performance: it retains the flexibility of a decentralized architecture (without centralized computation latency) and achieves path optimization and higher execution robustness close to that of centralized methods through the combination of Transformer and Kalman filtering. It is particularly suitable for lightweight multi-robot collaborative scenarios with high requirements for path accuracy and stability, such as small warehouse cargo transfer and laboratory robot collaboration.

[0035] Therefore, this invention employs a Transformer-based reinforcement learning multi-robot navigation path planning method. By capturing the implicit cooperative relationships between robots through a Transformer network, it achieves cooperative obstacle avoidance even without communication, thus improving the success rate of path planning in multi-robot systems. A Kalman filter algorithm is introduced to estimate and correct the robot's motion state in real time, suppressing the influence of sensor noise and motion errors, and improving the accuracy and robustness of path execution. A multi-dimensional reward function is designed, including target arrival, collision penalty, efficiency reward, and smoothness reward, guiding the robot to generate an efficient and smooth optimal path. A local replanning strategy is adopted to quickly respond to dynamic environmental changes, ensuring the timeliness and safety of the path, making it suitable for multi-robot collaborative operations in complex dynamic environments.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A reinforcement learning-based multi-robot navigation path planning method based on Transformer, characterized in that, Includes the following steps: S1. Data Acquisition and Preprocessing: Acquire the robot's initial position, target position, environmental obstacle information, and the robot's own motion state data; construct the system state vector and observation vector; and perform noise reduction and normalization preprocessing on the raw data. S2. Initial path generation based on Transformer: Construct a Transformer path planning network, extract global environmental features and robot collaboration features through the encoder, generate several candidate paths through the decoder, and use the reward mechanism of reinforcement learning to select the optimal initial path; S3. Kalman Filter State Optimization: Establish a Kalman filter model of the robot's motion state, and correct the robot's position and velocity estimates in real time based on the observation data to compensate for sensor noise and motion deviation; S4. Dynamic Path Adjustment: Based on the optimized state output by the Kalman filter and combined with information on dynamic environmental changes, the initial path is adjusted in real time to ensure collision-free and timely path operation. S5. Path Execution and Feedback: The robot executes the optimized path, collects motion state data and environmental feedback information in real time, and forms a closed-loop iterative optimization.

2. The reinforcement learning-based multi-robot navigation path planning method based on Transformer according to claim 1, characterized in that, In S1, the system state vector is defined as: ； in, n For the number of robots, , They represent the first i A robot k Moment x , y Direction and position coordinates , They represent the first i A robot k Moment x , y Directional velocity component; The observation vector is defined as: ； in, , , , The first i The robot obtains position and velocity data through sensors.

3. The reinforcement learning-based multi-robot navigation path planning method based on Transformer according to claim 2, characterized in that, In S2, the Transformer path planning network includes a path encoder, an environment encoder, and a cooperative decoder: Path encoder: Represents the path sequence of each robot as... B × N × D The three-dimensional tensor is used to enhance the feature dimension through a multilayer perceptron, and three layers of multi-head self-attention modules are stacked to capture path global dependency features. Environmental encoder: Encodes obstacle information and robot position information into environmental feature vectors, and highlights key obstacle constraints through spatial attention mechanism; Collaborative Decoder: It integrates path features and environmental features, models the implicit collaborative relationships between robots through a cross-attention mechanism, and outputs multiple candidate paths.

4. The reinforcement learning-based multi-robot navigation path planning method based on Transformer according to claim 3, characterized in that, In S2, the reward function for reinforcement learning is designed as follows: ； in, The robot is rewarded when it reaches the target location, and penalized according to its distance from the target if it fails to reach it. Collision penalty: A penalty is imposed when the robot collides with an obstacle or another robot; As an efficiency bonus, rewards are given based on path length, with shorter paths yielding higher rewards; To smooth out rewards, rewards are given based on the curvature changes of adjacent path points; the smaller the curvature change, the higher the reward. The specific reward value is set according to the importance of the project.

5. The Transformer-based reinforcement learning multi-robot navigation path planning method according to claim 4, characterized in that, In S3, the Kalman filter model includes a state prediction equation and an observation update equation: State prediction equation: ； ； Observational update equation: ； ； ； in, for k The prior state estimation vector at time step is based on k The optimal state at time -1 is predicted. k The robot is in constant state and has not yet merged. k Real-time observation data; for k The posterior state estimate vector at time -1 is a fusion k The optimal state estimate obtained after observing the data at time -1 is the prediction. k The basis of moment-state; for k The prior state covariance matrix at time t. for k The posterior state covariance matrix at time -1; F k This is the state transition matrix, i.e., the robot's motion model; B k To control the input matrix, u k To control the input vector, H k For the observation matrix, Q k The process noise covariance matrix is... R k To observe the noise covariance matrix, K k For Kalman gain, Z k for k The observation vector at time t, for k The posterior state estimate vector at time t. for k The posterior state covariance matrix at time t. I It is the identity matrix. w k-1 This is process noise.

6. The reinforcement learning-based multi-robot navigation path planning method based on Transformer according to claim 5, characterized in that, In S4, the dynamic path adjustment adopts a local replanning strategy. When the deviation between the position after Kalman filtering correction and the initial path exceeds a set threshold or a new obstacle is detected, local path replanning is triggered, and a local optimal path segment is quickly generated through the Transformer network.

7. A Transformer-based reinforcement learning multi-robot navigation path planning system, characterized in that, include: Data acquisition module: Composed of LiDAR, which has a built-in odometer and GPS, used to collect robot motion status data and environmental obstacle data; Preprocessing module: Denoises and normalizes the collected data, and constructs the system state vector and observation vector; Transformer path generation module: Constructs a Transformer network and combines reinforcement learning to generate an initial optimal path; Kalman filter optimization module: Establishes a motion state filtering model and corrects the robot's position and velocity estimates in real time; Dynamic adjustment module: Adjusts the initial path in real time based on the filtering results and environmental changes; Execution and Feedback Module: Controls the robot's execution path and collects feedback data to form a closed-loop optimization.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the Transformer-based reinforcement learning multi-robot navigation path planning method as described in any one of claims 1-6.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the Transformer-based reinforcement learning multi-robot navigation path planning method according to any one of claims 1-6.

Images