Deep strategy network and unmanned clamping vehicle path planning method based on deep strategy network
Through the deep policy network fusion of visual and topological features, efficient and smooth paths of unmanned clamped vehicles are generated, which solves the problems of low path planning efficiency and poor obstacle avoidance adaptability in dynamic warehousing environments, and effectively deal with dynamic obstacles and multi-objective optimization.
Patent Information
- Application Number
- CN202510306442.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-14
- Publication Date
- 2025-08-05
AI Technical Summary
Unmanned clamped vehicles have low path planning efficiency and poor obstacle avoidance adaptability in dynamic storage environments. The existing reinforcement learning methods do not model the environmental geometric structure, and cannot efficiently extract the spatial relationship between dynamic obstacles and the driving area, and it is difficult to take into account multi-objective optimization such as path smoothness and obstacle distance.
Deep strategy network is adopted, combined with ResNet18 convolutional neural network to extract visual features of obstacles and graph neural network GNN extract global topological features. Through the fully connected feature extraction module, visual and topological features are fused to generate efficient and smooth path points, and trajectory is evaluated and optimized through the Critic network.
It realizes efficient and smooth path planning for unmanned clamped vehicles in a dynamic storage environment, and improves the adaptability to dynamic obstacles and multi-objective optimization capabilities for path planning.
Smart Images
Figure CN120426993A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned vehicle path planning, and in particular to a deep strategy network for unmanned gripping vehicle path planning and an unmanned gripping vehicle path planning method based on the deep strategy network. Background Art
[0002] In modern warehousing operations, unmanned gripper vehicles must efficiently and stably plan paths in complex, dynamic environments. However, path planning in warehouses presents significant challenges: first, the drivable area changes over time and may be blocked by dynamic obstacles; second, path planning must adapt to different task requirements and generate highly adaptable and safe paths in real time.
[0003] Traditional path planning methods, such as the A* algorithm and the Dijkstra algorithm, while performing well in static environments, suffer from poor adaptability to dynamic obstacles and computational efficiency constraints in dynamic and unstructured scenarios. This has led to numerous improvements. Shan Wei et al. first optimized the key points of the A* path and then introduced extreme polynomial curves to smooth the path. The results showed that the improved A* algorithm generated paths that satisfied the robot's kinematic characteristics and were computationally simple, meeting real-time requirements. B. Jin proposed a multi-objective optimization A* algorithm, defining multiple objective functions, considering various traffic modes and transition costs, and using Pareto optimality as a heuristic function. The algorithm successfully achieved multi-objective optimal path planning in a grid map, starting from a starting point, passing through a given number of key points, and ultimately reaching the destination. Yan Linwei et al. proposed an improved dynamic window method, using the analytic hierarchy process to enable weight factors to adaptively adjust as the environment changes, thereby achieving dynamic obstacle avoidance. Classic local path planning algorithms offer high real-time performance and simplicity, but they still rely on maps, cannot effectively handle dynamic obstacles that change over time, and cannot effectively extract and utilize environmental information.
[0004] To better address the problem of adapting to dynamic obstacles in path planning, reinforcement learning has become a popular research area. For example, to address the limitations of traditional Q-learning methods in unknown environments and the shortcomings of traditional Q-learning methods in multi-objective local path planning tasks, Ali et al. integrated vector field histograms into the reward function and used LiDAR sensors to obtain real-time environmental information, improving the robot's perception and decision-making efficiency in complex environments. Tai et al., based on the Deep Q-Network (DQN) framework, used multi-layer convolution to extract image features, estimate the Q value of each action, and select the optimal action to control the vehicle. Due to DQN's poor scalability to continuous action spaces, its performance in complex scenarios remains limited. Wang et al., based on a globally guided reinforcement learning method, combined deep learning and reinforcement learning to create the Double Deep Q-Learning (DDQN) model to avoid conflicts with dynamic obstacles and achieve efficient and robust path planning. However, value-based learning and policy learning can suffer from training instability or overfitting in complex tasks. Based on this, reinforcement learning methods based on the combination of value and strategy have emerged. For example, Liu et al. combined the D* algorithm and operational science, and introduced global path information into the deep deterministic policy gradient algorithm, enabling vehicles to perform better path planning performance in complex environments.
[0005] Despite this, existing reinforcement learning path planning algorithms still face the following challenges: insufficient modeling of the environment's geometric structure, making it impossible to efficiently extract the spatial relationship between dynamic obstacles and the drivable area; and difficulty balancing multiple optimization objectives, such as path smoothness and obstacle distance. To address this, this paper proposes a path planning method for an unmanned gripping vehicle based on a deep policy network. This method improves the path planning performance of unmanned gripping vehicles in dynamic warehousing environments by extracting path features and optimizing policies. Summary of the Invention
[0006] The purpose of this invention is to provide a deep strategy network for path planning of unmanned gripping vehicles in dynamic storage environments to address the problems of low path planning efficiency and poor obstacle avoidance adaptability in existing technologies.
[0007] Another object of the present invention is to provide a path planning method for an unmanned gripping vehicle based on a deep strategy network.
[0008] The technical solution adopted to achieve the purpose of the present invention is:
[0009] A deep strategy network for path planning of an unmanned gripping vehicle includes a visual feature extraction module, a topological feature extraction module, and a fully connected feature extraction module. The visual feature extraction module extracts visual features of obstacles from real-time depth images based on a ResNet18 convolutional neural network. The topological feature extraction module extracts global topological features from the coordinates and attribute information of all obstacles within a predetermined range of the vehicle based on a graph neural network (GNN). The fully connected feature extraction module maps a one-dimensional fused feature vector obtained by fusing visual features and global topological features into a new feature space to represent the generated path points.
[0010] A path planning method for an unmanned gripping vehicle based on a deep policy network includes the following steps:
[0011] Step 1: Input feature preprocessing:
[0012] Acquire real-time depth images and LiDAR point cloud data to extract the endpoint coordinates and the current position of the unmanned vehicle on the map, and collect the coordinates and attribute information of all obstacles within the vehicle's predetermined range;
[0013] Step 2: Extract visual features and global topological features:
[0014] The real-time depth image obtained in step 1 is input into the visual extraction module, and the visual features of obstacles are extracted through the ResNet18 convolutional neural network;
[0015] At the same time, the coordinates and attribute information of all obstacles within the predetermined range of the vehicle collected in step 1 are input into the topological feature extraction module, and the graph neural network (GNN) outputs the global topological features;
[0016] Step 3, feature fusion and trajectory generation:
[0017] The visual features and global topological features of the obstacle in step 2 are fused by splicing or weighting to obtain a one-dimensional fused feature vector. This one-dimensional fused feature vector is input into the fully connected feature extraction module in combination with the target position of the vehicle. The fully connected feature extraction module decodes the one-dimensional fused feature vector and outputs the next action, including the target trajectory point, generates a trajectory point sequence, interpolates the trajectory from the trajectory point sequence, and smoothes the trajectory to obtain the future trajectory. The critic network evaluates the smoothness of the future trajectory, obstacle distance, and task completion efficiency through a multi-objective reward function, and optimizes the weights of the deep policy network in combination with the backpropagation algorithm.
[0018] In the above technical solution, the real-time depth image in step 1 is obtained from the depth camera of the simulation or real vehicle, the current vehicle positioning is obtained from the inertial navigation, and the obstacle information within the forward visible range is obtained based on the perceived target detection or in the simulation environment.
[0019] In the above technical solution, the RestNet18 convolutional neural network described in step 2 includes a convolutional input layer, four residual blocks, a global average pooling layer and a fully connected layer.
[0020] In the above technical solution, the residual block consists of two 3×3 convolutional layers, each of which is followed by batch normalization and ReLU activation function. Finally, the input is directly added to the output of the second convolutional layer through a skip connection. After the input x passes through the first weight layer and the first activation function, it enters the second weight layer and the second activation function. The input x is then directly added to the output F(x) of the last activation function to form F(x)+x as the final output.
[0021] In the above technical solution, the visual features of the obstacle in step 2 include texture, edge and shape information.
[0022] In the above technical solution, in step 2, the graph neural network GNN uses vehicles and obstacles as nodes, and the node features include position, speed, direction and category. The relationship between nodes is the edge, and the edge features include distance, relative speed and pass weight. The graph structure is constructed through the adjacency matrix, and the graph's message passing mechanism aggregates the neighbor information of each node. The edge weight is calculated in combination with the attention mechanism, and the node state is iteratively updated through the update function to generate a new feature vector for each node that integrates the neighbor information. The global features of the entire graph are extracted through global pooling, and the global topological features are generated in combination with the target position of the vehicle to represent the dynamic topological relationship of the entire scene.
[0023] In the above technical solution, in step 2, the graph neural network (GNN) updates the adjacency matrix in real time. When a new obstacle is detected, new nodes are added and the edge weights are dynamically adjusted. When the obstacle disappears, the corresponding nodes and edges are removed and the sparse adjacency matrix is regenerated.
[0024] In the above technical solution, the fused feature vector in step 3 is combined with the target position of the vehicle and input into the fully connected feature extraction module, and then an activation function is used to make the fully connected feature extraction module nonlinear, and a batch normalization layer BN is used to make the data normalized and the variance distribution uniform.
[0025] In the above technical solution, in step 3, the trajectory is smoothed by a Bezier curve or a quintic polynomial.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] By combining convolutional neural networks and graph neural networks to extract environmental features, this paper proposes a path planning method for an unmanned gripping vehicle based on a deep policy network. This method integrates the visual features extracted by ResNet18 and the global topological features extracted by GNN, outputs path points through a fully connected feature extraction module, and outputs an efficient and smooth path in real time under the reinforcement learning framework. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Shown is a schematic diagram of a deep strategy network for path planning of an unmanned gripping vehicle according to the present invention.
[0029] Figure 2 Shown is a schematic diagram of the path planning method of the unmanned gripping vehicle based on the deep strategy network of the present invention.
[0030] Figure 3 The figure shows a specific network structure diagram of ResNet18 in the unmanned gripping vehicle path planning method based on deep strategy network of the present invention.
[0031] Figure 4 Shown is a schematic diagram of the residual block of ResNet18 in the unmanned gripping vehicle path planning method based on deep strategy network of the present invention. DETAILED DESCRIPTION
[0032] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0033] Example 1
[0034] Reference Figure 1 A deep strategy network for path planning of unmanned gripping vehicles includes a visual feature extraction module, a topological feature extraction module, and a fully connected feature extraction module. The visual feature extraction module extracts visual features of obstacles from real-time depth images based on the ResNet18 convolutional neural network. The topological feature extraction module extracts global topological features from the coordinates and attribute information of all obstacles within a predetermined range of the vehicle based on the graph neural network (GNN). The fully connected feature extraction module maps a one-dimensional fused feature vector obtained by fusing visual features and global topological features into a new feature space to represent the generated path points.
[0035] Example 2
[0036] Reference Figure 2 , a path planning method for an unmanned gripping vehicle based on a deep policy network, comprising the following steps:
[0037] Step 1: Input feature preprocessing:
[0038] Acquire real-time depth images as the main visual information input of the current environment. The real-time depth images are obtained from the depth camera of the simulation or real vehicle; the lidar point cloud data extracts the coordinates of the end point and the current position of the unmanned gripping vehicle in the map for global path planning target positioning. The current vehicle positioning is obtained from the inertial navigation, and the obstacle information within the visible range in front is obtained based on the perceived target detection or in the simulation environment; collect the coordinates and attribute information of all obstacles within the predetermined range of the vehicle.
[0039] Step 2: Extract visual features and global topological features:
[0040] The real-time depth image obtained in step 1 is input into the visual extraction module, and the visual features of the obstacle, including texture, edge and shape information, are extracted through the ResNet18 convolutional neural network.
[0041] Reference Figure 3 , the RestNet18 convolutional neural network includes a convolution input layer, four residual blocks, a global average pooling layer and a fully connected layer. Specifically, it includes an input layer with a 7×7 convolution kernel, 64 output channels, a stride of 2, and a padding of 3. After batch normalization and ReLU activation function, it is followed by a 3×3 maximum pooling layer with a stride of 2, a padding of 1, and an output tensor size of H / 4×W / 4×64; the maximum pooling layer is followed by four residual block groups, each residual block consists of two 3×3 convolution kernels, 64 output channels, a stride of 1, and a padding of 1. After batch normalization and ReLU activation function, the output tensor size of the first residual block group is H / 4×W / 4×64, and the first residual block of the second residual block group uses a 3×3 convolution kernel, 128 output channels. The first residual block of the third residual block group uses a 3×3 convolution kernel, 256 output channels, a stride of 2, and a padding of 1, and the output tensor size is H / 16×W / 16×256. The first residual block of the fourth residual block group uses a 3×3 convolution kernel, 512 output channels, a stride of 2, and a padding of 1, and the output tensor size is H / 32×W / 32×512. A global average pooling layer is used to reduce the dimension of the feature map to 1×1×512, and finally a fully connected layer is used to obtain the visual features of 512 obstacles, including texture, edge and shape information.
[0042] Reference Figure 4The above residual block consists of two 3×3 convolutional layers, each followed by batch normalization and ReLU activation function. Finally, the input is directly added to the output of the second convolutional layer through a skip connection. After the input x passes through the first weight layer and the first activation function, it enters the second weight layer and the second activation function. The input x is then directly added to the output F(x) of the last activation function to form F(x)+x as the final output.
[0043] At the same time, the coordinates and attribute information of all obstacles within the predetermined range of the vehicle collected in step 1 are input into the topological feature extraction module. The graph neural network GNN uses vehicles and obstacles as nodes. The node features include position, speed, direction and category. The node feature matrix uses the multi-layer perceptron MLP to perform high-dimensional encoding of the position, speed and category information and maps it to a high-dimensional space. The relationship between nodes is the edge. The edge features include distance, relative speed and pass weight. The graph structure is constructed through the adjacency matrix. The message passing mechanism of the graph aggregates the neighbor information of each node. The edge weight is calculated according to the distance and pass probability in combination with the attention mechanism, and the node state is iteratively updated through the update function to generate a new feature vector for each node that integrates the neighbor information. The global features of the entire graph are extracted through global pooling, and the global topological features are generated in combination with the target position of the vehicle to represent the dynamic topological relationship of the entire scene. The graph neural network GNN updates the adjacency matrix in real time. When a new obstacle is detected, a new node is added and the edge weight is dynamically adjusted; when the obstacle disappears, the corresponding node and edge are removed, and the sparse adjacency matrix is regenerated.
[0044] Step 3, feature fusion and trajectory generation:
[0045] The visual features and global topological features of the obstacle in step 2 are fused by splicing or weighting to obtain a one-dimensional fused feature vector. This one-dimensional fused feature vector is input into the fully connected feature extraction module in combination with the target position of the vehicle. An activation function is used to make the fully connected feature extraction module nonlinear, and a batch normalization layer (BN) is used to make the data normalized and the variance distribution uniform. The fully connected feature extraction module decodes the one-dimensional fused feature vector and outputs the next control instruction, including the target trajectory point or control action, to generate trajectory points. The future trajectory is extracted from the trajectory point sequence generated by the deep policy network, and the trajectory is smoothed using a Bezier curve or a quintic polynomial. The critic network evaluates the smoothness of the generated trajectory, the obstacle distance, and the task completion efficiency using a multi-objective reward function, and combines the backpropagation algorithm for network weight optimization learning.
[0046] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A deep strategy network for path planning of unmanned gripping vehicles, characterized by: The deep strategy network includes a visual feature extraction module, a topological feature extraction module and a fully connected feature extraction module. The visual feature extraction module extracts the visual features of obstacles from real-time depth images based on the RestNet18 convolutional neural network. The topological feature extraction module extracts global topological features from the coordinates and attribute information of all obstacles within the predetermined range of the vehicle based on the graph neural network GNN. The fully connected feature extraction module maps a one-dimensional fused feature vector obtained by fusing the visual features and the global topological features into a new feature space for representing the generated path points.
2. A path planning method for an unmanned gripping vehicle based on the deep policy network of claim 1, characterized in that: The following steps are involved: Step 1: Input feature preprocessing: Acquire real-time depth images and LiDAR point cloud data to extract the endpoint coordinates and the current position of the unmanned vehicle on the map, and collect the coordinates and attribute information of all obstacles within the vehicle's predetermined range; Step 2: Extract visual features and global topological features: The real-time depth image obtained in step 1 is input into the visual extraction module, and the visual features of obstacles are extracted through the RestNet18 convolutional neural network; At the same time, the coordinates and attribute information of all obstacles within the predetermined range of the vehicle collected in step 1 are input into the topological feature extraction module, and the graph neural network (GNN) outputs the global topological features; Step 3, feature fusion and trajectory generation: The visual features and global topological features of the obstacle in step 2 are fused by splicing or weighting to obtain a one-dimensional fused feature vector. This one-dimensional fused feature vector is input into the fully connected feature extraction module in combination with the target position of the vehicle. The fully connected feature extraction module decodes the one-dimensional fused feature vector and outputs the next action, including the target trajectory point, generates a trajectory point sequence, interpolates the trajectory from the trajectory point sequence, and smoothes the trajectory to obtain the future trajectory. The critic network evaluates the smoothness of the future trajectory, obstacle distance, and task completion efficiency through a multi-objective reward function, and optimizes the weights of the deep policy network in combination with the backpropagation algorithm.
3. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: In step 1, the real-time depth image is obtained through the depth camera of the simulation or real vehicle, the current vehicle positioning is obtained through inertial navigation, and the obstacle information in the forward visible range is obtained based on the perceived target detection or in the simulation environment.
4. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: The RestNet18 convolutional neural network described in step 2 includes a convolutional input layer, four residual blocks, a global average pooling layer, and a fully connected layer.
5. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: The residual block consists of two 3×3 convolutional layers, each followed by batch normalization and ReLU activation function. Finally, the input is directly added to the output of the second convolutional layer through a skip connection. After the input x passes through the first weight layer and the first activation function, it enters the second weight layer and the second activation function. The input x is then directly added to the output F(x) of the last activation function to form F(x)+x as the final output.
6. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: The visual features of the obstacle in step 2 include texture, edge and shape information.
7. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: In step 2, the graph neural network (GNN) uses vehicles and obstacles as nodes. Node features include position, speed, direction, and category. The relationship between nodes is the edge. The edge features include distance, relative speed, and pass weight. The graph structure is constructed through the adjacency matrix. The graph's message passing mechanism aggregates the neighbor information of each node, and the edge weight is calculated in combination with the attention mechanism. The node state is iteratively updated through the update function to generate a new feature vector for each node that integrates the neighbor information. The global features of the entire graph are extracted through global pooling, and the global topological features are generated in combination with the target position of the vehicle to represent the dynamic topological relationship of the entire scene.
8. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 7, characterized in that: In step 2, the graph neural network (GNN) updates the adjacency matrix in real time. When a new obstacle is detected, new nodes are added and edge weights are dynamically adjusted. When the obstacle disappears, the corresponding nodes and edges are removed and the sparse adjacency matrix is regenerated.
9. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: In step 3, the fused feature vector is combined with the target position of the vehicle and input into the fully connected feature extraction module. The activation function is used to make the fully connected feature extraction module nonlinear, and the batch normalization layer BN is used to make the data normalized and the variance distribution uniform.
10. A path planning method for an unmanned gripping vehicle based on a deep policy network as claimed in claim 2, characterized in that: In step 3, the trajectory is smoothed using a Bezier curve or a quintic polynomial.