Mobile robot safety corridor generation method based on deep reinforcement learning

Abstract

The invention discloses a mobile robot safety corridor generation method based on deep reinforcement learning, and the method comprises the following steps: employing a Hybrid A * algorithm to generate a discrete node sequence which accords with the incomplete constraint of a robot, and enabling the discrete node sequence to serve as a global path, and constructing an initial corridor skeleton structure; constructing a deep reinforcement learning model, defining environment state input and an action space as rectangular parameters of a safety corridor, and designing a reward function; training through a DQN algorithm; predicting and correcting safety corridor parameters; and smoothing the parameter sequence. According to the method, corridor parameters are predicted online through the deep reinforcement learning model, limitation of a traditional fixed expansion radius is eliminated, the size and direction of a corridor can be adjusted in real time according to obstacle distribution, and the problem of excessive conservative or failure is avoided; and the global path and the local obstacle information are coupled in the generation process, so that obstacle avoidance is ensured through safety detection, track redundancy is reduced through path matching reward and smooth processing, and the motion smoothness is improved.

Description

Technical Field

[0001] This invention relates to the field of mobile robot path planning technology, specifically a method for generating safe corridors for mobile robots based on deep reinforcement learning. Background Technology

[0002] Mobile robots, as the earliest type of robot to achieve commercial application, have been developed over decades and have been widely used in various fields such as industrial manufacturing, logistics and warehousing, medical services, and home services. Path planning can be said to be the foundation of mobile robot movement. Traditional robot path planning methods include A*, RRT, and Voronoi diagrams. In complex environments, path planning methods based on safe corridors are often used. Compared with traditional point-to-point path planning methods, safe corridors provide continuous and connected feasible spaces, which can directly guarantee obstacle avoidance constraints. Moreover, after the safe corridor is constructed, the path planning problem can be transformed into optimizing a continuous trajectory within the corridor, without having to search the entire environment, greatly reducing the search space.

[0003] Traditional methods for generating safe corridors include grid map dilation, Voronoi diagrams, convex hull / convex segmentation, and path expansion. These methods have significant drawbacks: the size and orientation of the generated safe corridors are fixed, making them unsuitable for dynamic environments and different obstacle deployments; when the environment is complex or the global path is close to obstacles, the generated corridors are often too conservative, leading to a shrinking feasible region and limiting robot movement; furthermore, the generated corridors do not match the global path well enough, easily causing trajectory redundancy or unevenness. Summary of the Invention

[0004] The purpose of this invention is to overcome or at least partially solve the above-mentioned problems by proposing a method for generating safe corridors for mobile robots based on deep reinforcement learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for generating safe corridors for mobile robots based on deep reinforcement learning, comprising the following steps:

[0006] S1. Obtain the global path and construct the initial corridor. In the known environment grid map, set the robot's starting pose and target pose. Use the Hybrid A* algorithm to perform global path planning and generate a discrete node sequence that meets the robot's nonholonomic constraints as the global path. Use the global path as the center line of the initial corridor and construct the initial corridor skeleton structure C0 that only contains the center line coordinates.

[0007] S2. Construct a deep reinforcement learning model and define the environment state input. The action space is a rectangle with parameters of the safety corridor. The reward function was designed; the DQN algorithm was selected as the model optimization strategy, the policy function and the value function were defined, a deep reinforcement learning model was constructed and trained;

[0008] S3. Predict and correct the safety corridor parameters. Divide the global path into sequence points according to a preset step size. Each sequence point is a corridor unit and its tangent direction is calculated. Using the policy network of the trained deep reinforcement learning model, output the parameter increment relative to the initial value of the center line to map the safety corridor rectangle parameters of each corridor unit. Calculate the minimum distance between each corridor unit and the obstacle. If the preset safety gap is not met, correct the parameters. Calculate the overlap ratio of adjacent corridor units. If the preset overlap requirement is not met, prioritize expanding the length of the narrower corridor unit until the requirement is met or the maximum length limit is reached, and perform safety detection again.

[0009] S4. Output the safe corridor sequence and smooth the corrected safe corridor rectangle parameter sequence; detect the minimum distance to obstacles again for each smoothed corridor unit to ensure that the preset safety gap is met; combine all the corridor units that finally meet the requirements into a safe corridor sequence and output it for subsequent robot trajectory optimization.

[0010] In a preferred embodiment, in step S1, the kinematic model expression of the Hybrid A* algorithm is:

[0011]

[0012] Where v is the robot speed; θ represents the heading angle; L is the vehicle wheelbase; δ represents the vehicle turning angle; and the total cost of each extended node in the HybridA* algorithm is... g(n) represents the cumulative cost from the starting point to the current node n, and h(n) represents the heuristic function. After planning, a discrete sequence of nodes is obtained.

[0013] In a preferred embodiment, in step S2, the environmental state input The expression is:

[0014]

[0015] in, This represents local environmental grid information, indicating the distribution of obstacles and feasible areas; This is the current global path segment; This is the robot's current position.

[0016] The rectangle parameter The expression is:

[0017]

[0018] in, θ represents the center position of the rectangular safety corridor; w and h represent the width and length of the safety corridor; θ represents the orientation angle of the safety corridor.

[0019] In a preferred embodiment, in step S2, the expression for the reward function is:

[0020]

[0021]

[0022]

[0023]

[0024] in, These are the weighting factors for obstacle avoidance reward, path matching reward, and smoothness reward, respectively; R safe This represents the obstacle avoidance reward, where O is the set of obstacles. The indicator function is set to 1 if the obstacle falls within the generated rectangular corridor, and 0 otherwise; R path Indicates path matching reward. R represents the average distance between the corridor centerline and the global path; smooth For smooth reward, This is the difference in direction angle between the current corridor and the previous corridor.

[0025] In a preferred embodiment, in step S2, the optimization objective expression of the DQN algorithm is:

[0026]

[0027] Where θ represents the policy network parameters; This is a probability ratio used to measure the difference between the old and new strategies on the same action. and These represent the old and new strategies in the state. Select action The probability of; The dominant function represents the action. The relative value of the average state is good or bad; clip is the truncation function.

[0028] In a preferred embodiment, in step S3, the global path is divided into sequence points according to a preset step size s. After that, each sequence point As a reference point for a corridor unit, calculate the tangent direction angle at that point. The parameter increments output by the trained policy network include: , , , ,in The increment in the x-direction from the center of the corridor unit. The increment in the y-direction of the corridor unit center. For the increase in the width of the corridor unit, For the increment of corridor unit length, The parameter increment is the direction angle increment of the corridor unit; the formula for mapping the parameter increment to the parameters of the safety corridor rectangle is:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] in,( , () represents the coordinates of the center position of the corridor unit; This is a reference value for the width of the safety corridor. These are the minimum and maximum values ​​for the width of the safety corridor, respectively. This serves as a reference value for the length of the safety corridor. These are the minimum and maximum values ​​of the safety corridor length, respectively; clip is the truncation function.

[0035] The minimum distance between the corridor unit and the obstacle is calculated as follows: The distance field function d(x) of the environment is pre-calculated, where d(x) represents the distance from any point x back to the nearest obstacle. The minimum distance between rectangle Rt and the obstacle is then calculated.

[0036]

[0037] The preset safe distance is ,like Then adjust the width, length, or center point position of rectangle Rt until the desired result is achieved. .

[0038] In a preferred embodiment, in step S3, after the safety clearance detection is completed, the component ∆s of the center projection of adjacent rectangles Rt and Rt+1 onto the principal axis of Rt is calculated. t The following conditions must be met:

[0039]

[0040] in, This is the overlap ratio control coefficient. If the preset overlap requirement is not met, the length of the narrower corridor unit will be increased first until the requirement is met or the maximum length limit is reached.

[0041] In a preferred embodiment, step S4 involves smoothing the corrected safety corridor rectangle parameter sequence, specifically including:

[0042] The smoothing of the center position of the safety corridor is achieved using a moving average method with a window size of 2k+1, smoothing the x and y coordinates of the center separately, as shown in the formula:

[0043]

[0044]

[0045] The smoothing of the direction angle of the safety corridor is achieved by taking a moving average of the sine and cosine values ​​of the direction angle, and then calculating the smoothed direction angle using the arctangent function. The formula is as follows:

[0046]

[0047] The width and length of the safety corridor are smoothed by using the same moving average method as the center position of the safety corridor for both width and length.

[0048] Compared with existing technologies, this invention uses a deep reinforcement learning model to predict corridor parameters online, breaking away from the limitations of traditional fixed expansion radii. It can adjust the size and direction of the corridor in real time according to the distribution of obstacles, avoiding overly conservative or ineffective problems. Furthermore, it couples global path and local obstacle information during the generation process, ensuring obstacle avoidance through safety detection and reducing trajectory redundancy and improving motion smoothness through path matching rewards and smoothing processing. At the same time, the model has self-learning capabilities and can continuously optimize the strategy during the training phase. It can still work stably in unknown or dynamic environments, greatly reducing the need for manual parameter tuning and adapting to the path planning needs of mobile robots in different scenarios. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the deep reinforcement learning model of the present invention;

[0050] Figure 2 This is a flowchart of the safe corridor generation process of the present invention; Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings.

[0052] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this description, those skilled in the art can make creative modifications to this embodiment as needed, but as long as they are within the scope of the claims of the present invention, they are protected by patent law.

[0053] This invention provides a method for generating safe corridors for mobile robots based on deep reinforcement learning, which solves the technical problems in the prior art. The overall idea is as follows:

[0054] Example 1:

[0055] Please see Figures 1-2 A method for generating safe corridors for mobile robots based on deep reinforcement learning includes the following steps:

[0056] S1. Obtain the global path and construct the initial corridor;

[0057] S2. Construct a deep reinforcement learning model;

[0058] S3. Predict and correct safety corridor parameters;

[0059] S4, Output the safe corridor sequence.

[0060] Specifically, for step S1, in the known environmental grid map, let the robot's starting pose be... The target point pose is The Hybrid A* algorithm is used for global path planning. The Hybrid A* algorithm introduces continuous kinematic constraints on top of the traditional A* algorithm, enabling the generation of feasible paths that meet the robot's nonholonomic constraints. Its kinematic model can be written as:

[0061]

[0062] In the formula: v is the robot speed; θ represents the heading angle; L is the vehicle wheelbase; δ represents the vehicle turning angle.

[0063] During the search process, the total cost of each expanded node is Where g(n) represents the cumulative cost from the starting point to the current node n, and h(n) represents the heuristic function. After the search, a series of discrete node sequences can be obtained:

[0064]

[0065] Using the planned global path P as the centerline of the initial corridor, the initial corridor skeleton structure is established at the key nodes of the path. C0 contains only the centerline coordinates and does not contain the specific geometric parameters (width, orientation) of the rectangular corridor.

[0066] Specifically, for step S2, a deep reinforcement learning model is established based on the initial corridor centerline to adaptively predict the rectangular parameters of the safety corridor.

[0067] remember The environmental state input at time t includes local environmental raster information. This represents the distribution of obstacles and feasible regions; the current global path segment. Robot's current location ,Will Represented as:

[0068]

[0069] In the action space, Define the rectangular parameters of the predicted safety corridor:

[0070]

[0071] In the formula: The center position of the rectangular safety corridor is denoted by θ; w and h are the width and length of the safety corridor, respectively; θ represents the orientation angle of the safety corridor.

[0072] The environmental state is updated as the action changes. The function f represents the next step in updating the local environment based on the rectangular safe corridor and obstacles. To ensure that the generated safe corridor is safe, connected, and matches the global path, obstacle avoidance, smoothness, and feasibility need to be considered simultaneously. The reward function is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] In the formula: These represent the weighting factors for each cost; R safe This represents the obstacle avoidance reward, where O is the set of obstacles. The indicator function takes a value of 1 if the obstacle falls within the generated rectangular corridor, and 0 otherwise; R path Indicates path matching reward. R represents the average distance between the corridor centerline and the global path; smooth For smooth reward, This is the difference between the direction angle of the current corridor and the previous corridor;

[0078] After defining the reward function, we need to build a deep reinforcement learning model for training, defining the model's policy function and value function:

[0079]

[0080]

[0081] In the formula: The value function, approximated by the neural network parameter ∅, represents the state s. t Next, according to strategy π θ The expected long-term cumulative return that can be obtained by continuing to execute; t+k γ represents the reward obtained after performing an action starting from time t; γ represents the discount factor, which controls the degree of decay of future rewards.

[0082] The DQN algorithm is chosen as the optimization strategy for the model, and the optimization objective is defined as follows:

[0083]

[0084] In the formula: θ represents the policy network parameters; This is a probability ratio used to measure the difference between the old and new strategies on the same action. and These represent the old and new strategies in state s, respectively. t Choose action a t The probability of; Let be the dominance function, representing action a. t The relative state average value is good or bad; clip is a cutoff function to prevent probability ratios from being manipulated. If the deviation is too large, ensure training stability.

[0085] Specifically, for step S3, the global path is divided into sequence points according to the step size s. Each As a corridor determination, calculate the tangent direction at that point. Using a pre-trained policy network :

[0086]

[0087] In the formula: This represents the increment relative to the initial value of the centerline.

[0088] Map actions to safety corridor rectangle parameters:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in,( , () represents the coordinates of the center position of the corridor unit; This is a reference value for the width of the safety corridor. These are the minimum and maximum values ​​for the width of the safety corridor, respectively. This serves as a reference value for the length of the safety corridor. These are the minimum and maximum values ​​of the safety corridor length, respectively; clip is the truncation function.

[0095] Pre-calculate the environment's d(x), return the distance to the nearest obstacle for any point x, and calculate the minimum distance between rectangle Rt and the obstacle:

[0096]

[0097] Safety clearance required If the conditions are met, the test is passed; otherwise, corrections are required.

[0098] Next, considering the overlapping portion of adjacent rectangles, calculate the component ∆s of the center projection of adjacent rectangles Rt and Rt+1 onto the principal axis of Rt. t ,,ensure:

[0099]

[0100] In the formula: This is used to control the overlap ratio; if the requirement is not met, the narrower side's h is expanded first (until the requirement is met or reached). A second safety test was conducted to ensure that the corridor segments could be spliced ​​together continuously.

[0101] Specifically, for step S4, after completing the single-step prediction and correction, the set of corridor rectangle parameters is obtained sequentially. This constitutes a complete safety corridor sequence. The prediction results of deep reinforcement learning models may exhibit slight fluctuations between different segments, requiring smoothing of the sequence. A common method is to use a moving average, first smoothing the center position of the safety corridor.

[0102]

[0103]

[0104] Smooth the direction angle:

[0105]

[0106] The same smoothing process is applied to the width (w) and height (h) of the safety corridor. For each smoothed safety corridor, the minimum distance to the obstacle is detected again using the distance field function d(x). Ensure all segments meet the requirements. The judgments that do not meet the requirements still need to be revised. The final safe corridor sequence is as follows:

[0107]

[0108] The obtained safe corridor sequence can be directly used as a constraint in the subsequent trajectory optimization process.

[0109] This invention can dynamically predict the rectangular parameters of a safety corridor based on the environment, avoiding the problems of overly conservative approaches or failures in complex environments caused by traditional methods based on fixed expansion radii. Furthermore, this invention fully considers the coupling relationship between the global path and the distribution of local obstacles during the generation of the safety corridor, and can adaptively adjust the size and direction of the safety corridor, thereby improving safety and throughput. At the same time, it utilizes the self-learning ability of deep reinforcement learning models to continuously optimize strategies during the training phase, ensuring robustness in unknown or dynamic environments, effectively reducing the need for manual parameter tuning, and improving the universality and adaptability of the algorithm.

[0110] The above description of the embodiments is provided to facilitate understanding and use of the present invention by those skilled in the art. It is obvious to those skilled in the art that various modifications can be made to the embodiments, and the general principles described herein can be applied to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. A method for generating safe corridors for mobile robots based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Obtain the global path and construct the initial corridor. In the known environment grid map, set the robot's starting pose and target pose. Use the Hybrid A* algorithm to perform global path planning and generate a discrete node sequence that meets the robot's nonholonomic constraints as the global path. Use the global path as the center line of the initial corridor and construct the initial corridor skeleton structure C0 that only contains the center line coordinates. S2. Construct a deep reinforcement learning model and define the environment state input. The action space is a rectangle with parameters of the safety corridor. The reward function was designed; the DQN algorithm was selected as the model optimization strategy, the policy function and the value function were defined, a deep reinforcement learning model was constructed and trained; S3. Predict and correct the safety corridor parameters. Divide the global path into sequence points according to a preset step size. Each sequence point is a corridor unit and its tangent direction is calculated. Using the policy network of the trained deep reinforcement learning model, output the parameter increment relative to the initial value of the center line to map the safety corridor rectangle parameters of each corridor unit. Calculate the minimum distance between each corridor unit and the obstacle. If the preset safety gap is not met, correct the parameters. Calculate the overlap ratio of adjacent corridor units. If the preset overlap requirement is not met, prioritize expanding the length of the narrower corridor unit until the requirement is met or the maximum length limit is reached, and perform safety detection again. S4. Output the safety corridor sequence and smooth the corrected safety corridor rectangle parameter sequence. The minimum distance to obstacles is detected again for each smoothed corridor unit to ensure that the preset safety gap is met; all corridor units that finally meet the requirements are combined into a safe corridor sequence and output for subsequent trajectory optimization of the robot.

2. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the kinematic model expression of the Hybrid A* algorithm is: ； Where v is the robot speed; θ represents the heading angle; L is the vehicle wheelbase; δ represents the vehicle turning angle; and the total cost of each extended node in the Hybrid A* algorithm is... g(n) represents the cumulative cost from the starting point to the current node n, and h(n) represents the heuristic function. After planning, a discrete sequence of nodes is obtained.

3. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the environmental status input The expression is: ； in, This represents local environmental grid information, indicating the distribution of obstacles and feasible areas; This is the current global path segment; This is the robot's current position. The rectangle parameter The expression is: ； in, θ represents the center position of the rectangular safety corridor; w and h represent the width and length of the safety corridor; θ represents the orientation angle of the safety corridor.

4. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the expression for the reward function is: ； ； ； ； in, These are the weighting factors for obstacle avoidance reward, path matching reward, and smoothness reward, respectively; R safe This represents the obstacle avoidance reward, where O is the set of obstacles. The indicator function is set to 1 if the obstacle falls within the generated rectangular corridor, and 0 otherwise; R path Indicates path matching reward. R represents the average distance between the corridor centerline and the global path; smooth For smooth reward, This is the difference in direction angle between the current corridor and the previous corridor.

5. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the optimization objective expression of the DQN algorithm is: ； Where θ represents the policy network parameters; This is a probability ratio used to measure the difference between the old and new strategies on the same action. and These represent the old and new strategies in the state. Select action The probability of; The dominant function represents the action. The relative value of the average state is good or bad; clip is the truncation function.

6. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the global path is divided into sequence points according to a preset step size s. After that, each sequence point As a reference point for a corridor unit, calculate the tangent direction angle at that point. ; The parameter increments output by the trained policy network include , , , ,in The increment in the x-direction from the center of the corridor unit. The increment in the y-direction of the corridor unit center. For the increase in the width of the corridor unit, For the increment of corridor unit length, This represents the increment of the corridor unit's orientation angle; The formula for mapping the parameter increment to the parameters of the safety corridor rectangle is: ； ； ； ； ； in,( , () represents the coordinates of the center position of the corridor unit; This is a reference value for the width of the safety corridor. These are the minimum and maximum values ​​for the width of the safety corridor, respectively. This serves as a reference value for the length of the safety corridor. These are the minimum and maximum values ​​of the safety corridor length, respectively; clip is the truncation function. The minimum distance between the corridor unit and the obstacle is calculated as follows: The distance field function d(x) of the environment is pre-calculated, where d(x) represents the distance from any point x back to the nearest obstacle. The minimum distance between rectangle Rt and the obstacle is then calculated. ； The preset safe distance is ,like Then adjust the width, length, or center point position of rectangle Rt until the desired result is achieved. .

7. A method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1 or 6, characterized in that: In step S3, after the safety clearance detection is completed, the component ∆s of the center projection of adjacent rectangles Rt and Rt+1 onto the principal axis of Rt is calculated. t The following conditions must be met: ； in, This is the overlap ratio control coefficient. If the preset overlap requirement is not met, the length of the narrower corridor unit will be increased first until the requirement is met or the maximum length limit is reached.

8. The method for generating a safe corridor for a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: In step S4, the corrected safety corridor rectangle parameter sequence is smoothed, specifically including: The smoothing of the center position of the safety corridor is achieved using a moving average method with a window size of 2k+1, smoothing the x and y coordinates of the center separately, as shown in the formula: ； ； The smoothing of the direction angle of the safety corridor is achieved by taking a moving average of the sine and cosine values ​​of the direction angle, and then calculating the smoothed direction angle using the arctangent function. The formula is as follows: ； The width and length of the safety corridor are smoothed by using the same moving average method as the center position of the safety corridor for both width and length.

Images