Unmanned clamping and embracing vehicle storage area path planning algorithm based on graph search guide reinforcement learning
By combining the path planning algorithm of graph search and reinforcement learning, smooth obstacle avoidance paths for unmanned clamped vehicles in complex storage environments are generated, which solves the problem of insufficient planning in diversified areas of traditional methods and achieves more efficient path generation and obstacle avoidance performance.
Patent Information
- Application Number
- CN202510315446.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2025-08-15
Smart Images
Figure CN120489117A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of autonomous driving and path planning, and in particular to a warehouse path planning algorithm for an unmanned gripping vehicle based on graph search-guided reinforcement learning. Background Art
[0002] Unmanned gripper vehicles are increasingly used in modern cotton ginning plants. Their core mission is to flexibly and efficiently plan routes across diverse drivable areas, including wide and narrow roads. However, in complex warehouse environments, conventional path planning algorithms face the challenge of insufficient planning flexibility for diverse drivable areas. For example, in environments with frequently changing road widths and narrownesses, traditional planning methods struggle to efficiently plan routes in these dynamic and ever-changing scenarios. Furthermore, many path planning algorithms rely heavily on hyperparameters, making it difficult to quickly adapt to new scenarios in practice.
[0003] In contrast, learning-based methods demonstrate greater adaptability and flexibility in these areas. Han Li et al. proposed a learning-based optimization trajectory planning (LOTP) for unstructured scenarios. This method replaces heuristic functions with a deep neural network and combines it with a Monte Carlo tree search method to generate a rough path. This path is then smoothed and optimized for velocity planning. Experiments show that LOTP improves the computational efficiency and success rate of path planning in real-world mining environments. Shirin et al., by improving a new loss function and a recurrent neural network with an adjustable safety margin, achieved the generation of a safe path in relatively constant time under arbitrary spatial complexity. The self-attention mechanism has significant advantages in modeling long-term dependencies and sequences. Lucas Lehnert et al. trained a Transformer model to mimic the A* search process, enabling it to find the optimal plan with fewer search steps, thus improving search efficiency. Fu et al. combined planning and prediction, leveraging a transformer to share global contextual reasoning across all traffic participants, enabling interconnected and fused features between planning and prediction, thus reducing collision risk. However, methods that use neural networks to learn and extract environmental features to generate paths are highly dependent on training data, lack robustness across different environments, and are often uninterpretable. Reinforcement learning has been studied by many scholars because it does not rely on environmental and model information and does not require data labeling. It continuously learns towards its goal by making a series of decisions based on rewards through continuous interaction between the intelligent agent and the environment in order to maximize the expected return. In recent years, path planning with reinforcement learning and deep reinforcement learning as the core has become a research hotspot. Summary of the Invention
[0004] The purpose of the present invention is to address the technical defects in the prior art and provide a warehouse path planning algorithm for an unmanned gripping vehicle based on graph search guided reinforcement learning.
[0005] The technical solution adopted to achieve the purpose of the present invention is:
[0006] A warehouse path planning algorithm for an unmanned gripper vehicle based on graph search-guided reinforcement learning includes the following steps:
[0007] Step 1: Use LiDAR to scan the warehouse environment, record the first point cloud data and create an environment map. At the same time, match and locate the unmanned gripping vehicle based on the real-time point cloud data to determine its current position in the environment map. Fuse the depth image generated by the depth camera image with the real-time point cloud data of LiDAR to obtain the relative position, size and dynamic information of the obstacle relative to the gripping vehicle.
[0008] Step 2: Based on the environment map in step 1 and the given target point, a series of path points are planned through the graph search algorithm. The rough path of
[0009] Step 3: The Actor network in the reinforcement learning optimization module based on the Actor-Critic framework takes the current position of the clamping vehicle, the relative position, size and dynamic information of the obstacle relative to the clamping vehicle as input, and generates a series of path points. The refined path and calculate the immediate reward r t , Critic network is based on the current state S t 、Action a t and the next moment state S t+1 , so that the immediate reward r t The output state-action value function Q(s t ,a t ) Approaching the target expected maximum reward Q target , and calculate the mean square error loss function L critic , based on L critic Update the critic network parameters θ critic , where the current state S t Including the position of the clamping vehicle, the position of the obstacle and the relative position of the obstacle, action a t Contains a series of path point coordinates The rough path in step 2 is used as the guide of the reinforcement learning module. The critic network scores the similarity between the rough path and the fine path through the path fusion strategy to obtain the similarity loss function L similarity , L critic and L similarity Weighted loss function L of the Actor network actor , based on the loss function L actor Adjust the Actor network parameters θ actor .
[0010] In the above technical solution, the loss function L of the Actor network in step 3 is actor The calculation formula is as follows:
[0011] L actor =aL critic +(1-a)L similarity
[0012] Where, L actor is the loss function of the Actor network, L critic is the mean square error loss function, L similarity is the similarity loss function between the coarse path and the fine path, and a is the weight coefficient.
[0013] In the above technical solution, the mean square error loss function L critic The calculation formula is as follows:
[0014]
[0015] Where, For the target expected maximum return, is the current state-action function of the i-th path point, is the current state of the i-th path point, is the action of the i-th path point at the current moment, The action probability distribution output by the policy network is expressed in state Next select action The probability of N is the number of path points.
[0016] In the above technical solution, the similarity loss function L between the coarse path and the fine path is similarity The calculation formula is as follows:
[0017]
[0018] Where, L similarity is the similarity loss function between the coarse path and the fine path, are the path points generated by the graph search algorithm, is the waypoint generated by the Actor network, and N is the number of waypoints.
[0019] In the above technical solution, the target expected maximum reward Q target The calculation formula is as follows:
[0020] Q target =r t +γQ(s t+1 ,a t+1 )
[0021] Where Q targetis the target expected maximum return, r t is the immediate reward of the Actor network, γ is the discount factor, Q(s t+1 ,a t+1 ) is the state-action function at the next moment.
[0022] In the above technical solution, the instant reward r of the Actor network t The calculation formula is:
[0023]
[0024] Where N is the number of path points, K i is the i-th path point on the refined path, R smooth (K i ) is the smoothness reward, R goal (K i ) is the end distance reward, R safe (K i ) is the obstacle avoidance safety reward, and w1, w2, and w3 are the corresponding weights.
[0025] In the above technical solution, the smoothness reward R smooth (K i ) is calculated as:
[0026] R smooth (K i )=-|θ i -θ i-1 | 2
[0027] Where θ i is the heading angle of the i-th path point of the refined path, θ i-1 is the heading angle of the i-1th path point;
[0028] The end distance reward R goal (K i ) is calculated as:
[0029] R goal (K i )=-|K i -s goal |
[0030] Where K i is the i-th path point of the refined path, s goal is the target endpoint;
[0031] The obstacle avoidance safety reward R safe (K i ) is calculated as:
[0032]
[0033] Where, d obs,min (K i ) is the minimum distance between the i-th path point and the nearest obstacle on the refined path, d safe The safety threshold is set.
[0034] In the above technical solution, the Critic network parameter θ in step 3 critic The update is as follows:
[0035]
[0036] Where θ critic is the Critic network parameter, is the mean square error loss function L critic For the Critic network parameter θ critic Find the partial derivative, α c is the learning rate of the Critic network.
[0037] In the above technical solution, the Actor network parameter θ in step 3 is actor The update is as follows:
[0038]
[0039] Where θ actor is the Actor network parameter, is the loss function L actor Actor network parameters θ actor Find the partial derivative, α a is the learning rate of the Actor network.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] The path planning algorithm of the unmanned clamping vehicle proposed in this invention first calculates the mean square error loss function L critic , the rough path generated by graph search is used as the guide of the reinforcement learning model. The Critic network evaluates the similarity of the rough path and the fine path generated by the Actor network through a fusion strategy to obtain the similarity loss function L similarity , L critic and L similarity Weighted loss function L of the Actor network actor , according to the loss function L of the Actor network actorBy adjusting network parameters, the algorithm is made more robust in multiple scenarios, solving the planning problem in areas with mixed wide and narrow road distances. The generated paths are smoother and have stronger obstacle avoidance performance. Graph search guidance is introduced into the reinforcement learning strategy to reduce unnecessary search behavior, significantly improving learning efficiency and reducing convergence time. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 The figure shown in the present invention searches for an initial path to generate a flow chart.
[0043] Figure 2 Shown is a schematic diagram of generating an initial path for graph search according to the present invention.
[0044] Figure 3 Shown is a comparative schematic diagram of path planning in a mixed wide and narrow road area according to the present invention.
[0045] Figure 4 This is a schematic diagram of the framework of the unmanned gripper vehicle storage area path planning algorithm based on graph search guided reinforcement learning in the present invention. DETAILED DESCRIPTION
[0046] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0047] A warehouse path planning algorithm for an unmanned gripper vehicle based on graph search-guided reinforcement learning includes the following steps:
[0048] Reference Figure 1 ,Step 1, use the lidar to scan the warehouse environment, record the first point cloud data and form an environment map, and at the same time, match and locate the unmanned gripping vehicle in the environment map based on the real-time point cloud data; fuse the depth image generated by the depth camera image with the real-time point cloud data of the lidar to obtain the relative position, size and dynamic information of the obstacle relative to the gripping vehicle.
[0049] Reference Figure 2 The black area represents obstacles, which are projected onto a plane after the radar point cloud scans the obstacle. The white area represents the drivable area on the map. The real-time and accuracy of this data have a significant impact on the efficiency and effectiveness of subsequent path planning.
[0050] Step 2: Based on the environment map in step 1 and the given target point, a series of path points are generated using the hybrid A* algorithm. The rough path is combined with the shortest path from the starting point to the end point, the vehicle's turning radius, maximum speed and other motion characteristics to ensure that the generated path meets the vehicle's kinematic constraints. At the same time, Hybrid A* supports dynamic adjustment of the search range and target direction weight in the heuristic function, thereby accelerating path generation in complex environments. The generated path provides a guiding benchmark for subsequent reinforcement learning optimization, which can not only narrow the search space but also reduce the randomness of policy learning. Figure 2 , the green part represents the path planned by the algorithm, the black arrow represents the end point of the path, and the green arrow represents the path point.
[0051] Reference Figure 3 The red line is the path planned after the destination is given. In the left scene, the road margin is wide and the curvature is relatively small, so the planning is completed. In the right scene, the road margin is yellow and narrow, and a large curvature path is planned to complete the turn.
[0052] Step 3: The Actor network in the reinforcement learning optimization module based on the Actor-Critic framework takes the current position of the clamping vehicle, the relative position, size and dynamic information of the obstacle relative to the clamping vehicle as input, and generates a series of path points. The refined path and calculate the immediate reward r t , Critic network is based on the current state S t 、Action a t and the next moment state S t+1 , so that the immediate reward r t The output state-action value function Q(s t ,a t ) Approaching the target expected maximum reward Q target , and calculate the mean square error loss function L critic , based on L critic Update the critic network parameters θ critic , where the current state S t Including the position of the clamping vehicle, the position of the obstacle and the relative position of the obstacle, action a t Contains a series of path point coordinates The rough path in step 2 is used as the guide of the reinforcement learning module. The critic network scores the similarity between the rough path and the fine path through the path fusion strategy to obtain the similarity loss function L similarity , L critic and L similarity Weighted loss function L of the Actor network actor , based on the loss function L actor Adjust the Actor network parameters θ actor .
[0053] In step 3, the Actor network in the reinforcement learning optimization module based on the Actor-Critic framework takes the current position of the clamping vehicle, the relative position, size and dynamic information of the obstacle relative to the clamping vehicle as input, and generates a series of path points. The refined path and calculate the immediate reward r t The details are as follows:
[0054] Input the coordinates of the destination point of the planned path (x, y, yaw), where x represents the horizontal position of the target point in the established map coordinates, y represents the vertical position of the target point in the established map coordinates, and yaw represents the heading angle of the target point in the global map coordinate system with the horizontal axis as 0 degrees, which is used to describe the orientation of the vehicle at the destination. The depth image and the destination coordinates are connected after feature extraction through the neural network to output the key points of the planned path. Contains n key points, each composed of (x, y, yaw), traverses each point on the path, calculates the smoothness, end distance and safety rewards respectively, and weights them to form an immediate reward r t Then, the planned path is interpolated and input into the control to execute, and the path points of the Actor network to generate a fine path are obtained. And update the state to S t+1 .
[0055] Instant reward r of the Actor network t The calculation formula is:
[0056]
[0057] Where r t is the instant reward of the Actor network, N is the number of path points, K i is the i-th path point, R smooth (K i ) is the smoothness reward, R goal (K i ) is the end distance reward, R safe (K i ) is the obstacle avoidance safety reward, and w1, w2, and w3 are the corresponding weights.
[0058] Smoothness reward R smooth (K i ) is calculated as:
[0059] R smooth (K i )=-|θ i -θ i-1 | 2
[0060] Where θ i is the heading angle of the i-th point of the refined path, θi-1 is the heading angle of the i-1th point.
[0061] End distance reward R goal (K i ) is calculated as:
[0062] R goal (K i )=-|K i -s goal |
[0063] Where K i is the i-th path point of the refined path, s goal The target end point.
[0064] Obstacle avoidance safety reward R safe (K i ) is calculated as:
[0065]
[0066] Where, d obs,min (K i ) is the minimum distance between the i-th point of the refined path and the nearest obstacle, d safe The safety threshold is set.
[0067] Critic network is based on the current state S t 、Action a t and the next moment state S t+1 , so that the immediate reward r t The output state-action value function Q(s t ,a t ) Approaching the target expected maximum reward Q target , the Critic network scores the path quality according to the multi-objective evaluation function such as the smoothness of the current path, obstacle distance, safety, etc. and feeds it back to the Actor network, and calculates the mean square error loss function L critic , based on L critic Update the critic network parameters θ critic , mean square error loss function L critic The calculation formula is as follows:
[0068]
[0069] Where, is the target expected maximum return value, is the current state-action function of the i-th path point, is the current state of the i-th path point, is the action of the i-th path point at the current moment, The action probability distribution output by the policy network is expressed in state Next select action The probability of N is the number of path points.
[0070] Target expected maximum return value Q target The calculation formula is as follows:
[0071] Q target =r t +γQ(s t+1 ,a t+1 )
[0072] Where r t is the immediate reward of the Actor network, γ is the discount factor, Q(s t+1 ,a t+1 ) is the state-action function at the next moment.
[0073] Critic network parameter θ critic The update is as follows:
[0074]
[0075] Where θ critic is the Critic network parameter, is the mean square error loss function L critic For the Critic network parameter θ critic Find the partial derivative, α c is the learning rate of the Critic network.
[0076] The rough path in step 2 is used as the guide of the reinforcement learning module. The critic network scores the similarity between the rough path and the fine path through the path fusion strategy to obtain the similarity loss function L similarity , the Critic network will comprehensively evaluate the quality of the Actor-generated path and the reference quality of the hybrid A* path. In this way, the Critic network can capture the balance between the Actor network strategy and the hybrid A* guidance, and feed it back to the Actor network to critic and L similarity Weighted loss function L of the Actor network actor , the Actor network is based on the loss function L actor Adjust the Actor network parameters θ actor By weighting, the path generated by the Actor network is forced to be close to the search path to a certain extent, thereby improving the convergence speed and robustness of the policy network.
[0077] The similarity loss function L between the rough path planned by the graph search algorithm and the path generated by the Actor network similarityThe calculation formula is as follows:
[0078]
[0079] Where, L similarity is the similarity loss function between the coarse path and the fine path, are the path points generated by the graph search algorithm, is the waypoint generated by the Actor network, and N is the number of waypoints.
[0080] The loss function L of the Actor network actor The calculation formula is as follows:
[0081] L actor =aL critic +(1-a)L similarity
[0082] Where, L actor is the loss function of the Actor network, L critic is the mean square error loss function, L similarity is the similarity loss function between the coarse path and the fine path, and a is the weight coefficient.
[0083] Reference Figure 4 , adjust the Actor network parameters θ through the optimizer actor , thus gradually training and optimizing.
[0084]
[0085] Where θ actor is the Actor network parameter, is the loss function L actor Actor network parameters θ actor Find the partial derivative, α a is the learning rate of the Actor network.
[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A warehouse path planning algorithm for unmanned gripping vehicles based on graph search guided reinforcement learning, characterized by: The following steps are involved: Step 1: Use LiDAR to scan the warehouse environment, record the first point cloud data and create an environment map. At the same time, match and locate the unmanned gripping vehicle based on the real-time point cloud data to determine its current position in the environment map. Fuse the depth image generated by the depth camera image with the real-time point cloud data of LiDAR to obtain the relative position, size and dynamic information of the obstacle relative to the gripping vehicle. Step 2: Based on the environment map in step 1 and the given target point, a series of path points are planned through the graph search algorithm. The rough path of Step 3: The Actor network in the reinforcement learning optimization module based on the Actor-Critic framework takes the current position of the clamping vehicle, the relative position, size and dynamic information of the obstacle relative to the clamping vehicle as input, and generates a series of path points. The refined path and calculate the immediate reward r t , Critic network is based on the current state S t 、Action a t and the next moment state S t+1 , so that the immediate reward r t The output state-action value function Q(s t ,a t ) Approaching the target expected maximum reward Q target , and calculate the mean square error loss function L critic , based on L critic Update the critic network parameters θ critic , where the current state S t Including the position of the clamping vehicle, the position of the obstacle and the relative position of the obstacle, action a t Contains a series of path point coordinates The rough path in step 2 is used as the guide of the reinforcement learning module. The critic network scores the similarity between the rough path and the fine path through the path fusion strategy to obtain the similarity loss function L similarity , L critic and L similarity Weighted loss function L of the Actor network actor , based on the loss function L actor Adjust the Actor network parameters θ actor .
2. The path planning algorithm for the unmanned gripping vehicle storage area according to claim 1 is characterized in that: The loss function L of the Actor network in step 3 actor The calculation formula is as follows: L actor =aL critic +(1-a)L similarity Where, L actor is the loss function of the Actor network, L critic is the mean square error loss function, L similarity is the similarity loss function between the coarse path and the fine path, and a is the weight coefficient.
3. The path planning algorithm for the unmanned gripping vehicle storage area according to claim 2 is characterized in that: The mean square error loss function L critic The calculation formula is as follows: Where, For the target expected maximum return, is the current state-action function of the i-th path point, is the current state of the i-th path point, is the action of the i-th path point at the current moment, The action probability distribution output by the policy network is expressed in state Next select action The probability of N is the number of path points.
4. The path planning algorithm for the unmanned gripping vehicle storage area according to claim 2 is characterized in that: Similarity loss function L between coarse path and fine path similarity The calculation formula is as follows: Where, L similarity is the similarity loss function between the coarse path and the fine path, are the path points generated by the graph search algorithm, is the waypoint generated by the Actor network, and N is the number of waypoints.
5. The path planning algorithm for the unmanned gripping vehicle storage area according to claim 3 is characterized in that: The target expected maximum reward q target The calculation formula is as follows: q target =r t +γq(s t+1 ,a t+1 ) Where q target is the target expected maximum return, r t is the immediate reward of the Actor network, γ is the discount factor, Q(s t+1 ,a t+1 ) is the state-action function at the next moment.
6. The path planning algorithm for the unmanned gripping vehicle storage area according to claim 5 is characterized in that: Instant reward r of the Actor network t The calculation formula is: Where N is the number of path points, K i is the i-th path point on the refined path, R smooth (K i ) is the smoothness reward, R goal (K i ) is the end distance reward, R safe (K i ) is the obstacle avoidance safety reward, and w1, w2, and w3 are the corresponding weights.
7. The warehouse path planning algorithm for the unmanned gripping vehicle according to claim 6 is characterized in that: The smoothness reward R smooth (K i ) is calculated as: R smooth (K i )=-|θ i -θ i-1 | 2 Where θ i is the heading angle of the i-th path point of the refined path, θ i-1 is the heading angle of the i-1th path point; The end distance reward R goal (K i ) is calculated as: R goal (K i )=-|K i -s goal | Where K i is the i-th path point of the refined path, s goal is the target endpoint; The obstacle avoidance safety reward R safe (K i ) is calculated as: Where, d obs,min (K i ) is the minimum distance between the i-th path point and the nearest obstacle on the refined path, d safe The safety threshold is set.
8. The warehouse path planning algorithm for the unmanned gripping vehicle according to claim 1 is characterized in that: Critic network parameter θ in step 3 critic The update is as follows: Where θ critic is the Critic network parameter, is the mean square error loss function L critic For the Critic network parameter θ critic Find the partial derivative, α c is the learning rate of the Critic network.
9. The warehouse path planning algorithm for the unmanned gripping vehicle according to claim 1 is characterized in that: The Actor network parameters θ described in step 3 actor The update is as follows: Where θ actor is the Actor network parameter, is the loss function L actor Actor network parameters θ actor Find the partial derivative, α a is the learning rate of the Actor network.