Robot dynamic obstacle avoidance planning method and navigation system based on path key node and deep reinforcement learning

Abstract

The application relates to a robot dynamic obstacle avoidance planning method and a navigation system based on a path key node and deep reinforcement learning, and the method comprises a global path planning stage and a local path planning stage. The global path planning stage converts an actual scene map into a grid map, extracts representative obstacle coordinates, constructs and optimizes a Voronoi diagram, applies an A* algorithm to generate a preliminary path point set, and constructs a path core node set. The local path planning stage divides the global path planning task into a plurality of subtasks, designs a multi-dimensional reward function based on a trajectory landing point prediction and an environment potential field based on a deep Q network of reinforcement learning, adopts a priority experience playback strategy based on instant rewards and TD error dynamic weighting, executes local path planning in each subtask, generates a local planning path of each subtask, realizes accurate obstacle avoidance and path adjustment in a dynamic environment, and effectively improves the path planning accuracy, robustness and real-time performance of the robot in a complex dynamic environment.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent navigation technology, and in particular to a robot dynamic obstacle avoidance planning method and navigation system based on path key nodes and deep reinforcement learning. It is applicable to path planning scenarios in complex dynamic environments, especially addressing the need for safe and efficient robot passage in scenarios with dynamic obstacle interference and incomplete global environmental information. Background Technology

[0002] Robot path planning is a core technology in the field of intelligent navigation. Its core objective is to enable robots to safely and efficiently reach their destination from a starting point in complex environments. It is widely used in industrial automation, service robots, unmanned systems, and many other fields. With the continuous expansion of application scenarios, path planning in dynamic and unknown environments has become a key research focus and challenge. Existing path planning methods are mainly divided into two categories: traditional path planning algorithms and reinforcement learning-based path planning algorithms. Traditional path planning algorithms (such as Dijkstra's algorithm, A* algorithm, PSO algorithm, RRT algorithm, etc.) usually rely on detailed prior information about the global environment to generate a passable path through calculation. However, in actual navigation, robots often only have a local field of view and have poor adaptability when facing the uncertainty of dynamic obstacles. At the same time, these algorithms need to process a large amount of data and perform complex calculations, consuming huge computational resources and making it difficult to meet real-time requirements. Reinforcement learning-based path planning algorithms (such as DQN, DDPG, SAC, TD3, etc.) have made some progress in path optimization and dynamic obstacle avoidance due to their autonomous learning capabilities. However, these methods still have significant shortcomings: First, training efficiency is low, and the sparse reward problem can easily cause the robot to get lost during the learning process, making it difficult to converge quickly. Second, scene generalization is insufficient; existing algorithms are mostly validated in simple environmental settings, and when faced with complex scenes where multiple static and dynamic obstacles coexist, obstacle avoidance performance and navigation stability drop significantly. Third, the experience playback mechanism lacks specificity and cannot prioritize the sampling of key experiences, resulting in limited model training efficiency and decision accuracy. Furthermore, in existing technologies, global path planning and local path planning are mostly designed independently, failing to form effective synergy. Global path planning struggles to provide precise local optimization targets, and local path planning cannot fully utilize the safe zone information of the global environment, making it difficult to balance the safety, flexibility, and efficiency of overall path planning. Therefore, there is an urgent need for a path planning method that can integrate the advantages of global and local planning to improve adaptability to complex dynamic environments, training efficiency, and navigation robustness. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a robot dynamic obstacle avoidance planning method and navigation system based on path key nodes and deep reinforcement learning. It solves the problems of poor adaptability, low training efficiency, and insufficient robustness of existing robot path planning methods in complex dynamic environments. It provides a composite path planning method that integrates global path core node selection and deep reinforcement learning local optimization, enabling safe, efficient, and accurate navigation of robots in scenarios where static and dynamic obstacles coexist.

[0004] To achieve the above technical objectives, this invention provides the following technical solution: a robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning, comprising the following steps: S1. Perform binarization and rasterization preprocessing on the actual scene map, set a threshold to extract representative obstacle areas, determine raster values, generate a raster map, and extract the coordinates and center points of representative obstacles; the coordinates of the representative obstacles are the vertices of the representative obstacle grids in the raster map; the representative obstacles include dynamic obstacles and static obstacles. S2. Construct a set of obstacle center coordinates based on the raster map, and then construct the Voronoi diagram and its adjacency matrix; S3. Map the vertex coordinates of the Voronoi graph to their corresponding grid indices in the grid map, and optimize the Voronoi graph and its adjacency matrix by filtering obstacles vertices, filtering invalid edges, and removing redundant nodes. S4. Apply the A* algorithm to the optimized Voronoi diagram and its optimized adjacency matrix to generate a preliminary path point set. Calculate the minimum distance between each path segment and a representative obstacle, and extract the path core nodes from the preliminary path point set based on a preset safe distance threshold to construct a path core node set. The path segment is a line segment formed by connecting adjacent path points in the preliminary path point set. S5. Based on the core node set of the path, the global path planning task is divided into several sub-tasks. Based on the reinforcement learning deep Q network, local path planning is performed in each sub-task to generate the local planned path of each sub-task. S6. Integrate the local planning paths of each subtask to generate a global planning path.

[0005] Optionally, the preprocessing of the actual scene map, including binarization and rasterization, setting a threshold to extract representative obstacle areas, determining raster values, and generating a raster map, includes: The actual scene map is converted into an image matrix and binarized using a threshold function to obtain a binary image matrix; in the binary image matrix, elements with a value of 0 represent passable areas, and elements with a value of 1 represent obstacle areas. The binary image matrix is ​​rasterized according to a preset reduction ratio to obtain a rasterized matrix; Traverse the connected regions in the rasterized matrix. If the proportion of pixels with a value of 1 in the connected region exceeds a preset threshold and the area of ​​the connected region is greater than a preset size threshold, then mark the connected region as a representative obstacle region. The grid cells in the representative obstacle region of the rasterized matrix are marked as representative obstacle grid cells, with their grid values ​​set to 1, and the grid values ​​of the remaining grid cells are set to 0, thus generating a raster map.

[0006] Optionally, the step of constructing a set of obstacle center coordinates based on a raster map and then constructing a Voronoi diagram and its adjacency matrix includes: The grid map is traversed to extract a set of representative obstacle coordinates, and then a set of obstacle center coordinates is constructed, which is mathematically represented as follows: ; in Represents the set of coordinates of the obstacle's center; , A set of representative obstacle coordinates The horizontal and vertical coordinates of representative obstacles; , These represent the horizontal and vertical offsets from the geometric center coordinates of the raster map to the coordinates of the representative obstacle, respectively, with values ​​ranging from [value range missing]. ; Using the set of obstacle center coordinates as the vertex set, construct a Voronoi graph and generate an adjacency matrix representing the vertex connectivity of the Voronoi graph; The adjacency matrix has elements that satisfy: if the first element in the Voronoi diagram is... vertex , No. vertex If there is an edge connecting them, then ,otherwise ;in, Represents the adjacency matrix of the nth element. Line number The elements of the column represent vertices in the Voronoi diagram. , Connectivity between them; , These represent the vertices in the Voronoi diagram. The x and y coordinate values, , These represent the vertices in the Voronoi diagram. The horizontal and vertical coordinate values.

[0007] Optionally, the mapping of vertex coordinates of the Voronoi diagram to their corresponding raster indices in the raster map is mathematically represented as follows: ; ; in, , Let x and y represent the x and y coordinates of a vertex in the Voronoi diagram, respectively. , They represent , The corresponding raster index value in the raster map; , These are the horizontal and vertical coordinates of the bottom left corner of the raster map; Indicates the resolution of the raster map; The obstacle vertex filtering process involves removing the row and column corresponding to a vertex from the adjacency matrix of the Voronoi diagram if the corresponding grid cell in the grid map is a representative obstacle grid cell. The invalid edge filtering is based on the coordinates of representative obstacles to calculate the shortest vertical distance between the representative obstacle and each edge of the Voronoi graph. If the distance is less than or equal to a preset threshold, the corresponding element in the Voronoi graph adjacency matrix is ​​set to 0; otherwise, the corresponding element in the Voronoi graph adjacency matrix is ​​retained. The redundant node removal process involves traversing the vertices in the Voronoi graph. If three vertices are collinear, the middle vertex is considered a redundant node and removed from the Voronoi graph.

[0008] Optionally, the calculation of the minimum distance between each path segment and the representative obstacle is performed as follows: ; in, This represents the minimum distance between a path segment and a representative obstacle. This indicates taking the minimum value; This represents the Voronoi diagram vertex coordinates of the starting point of the current path segment in the preliminary path point set. The Voronoi diagram coordinates of the endpoints of the current path segment in the initial path point set; Indicates the length of the path segment; Indicates the origin of the point , and the The center point of each obstacle The determinant formed by the matrix is ​​defined as follows: ; in , These represent the x and y coordinates of the Voronoi diagram vertex at the starting point of the current path segment, respectively. , These represent the x and y coordinates of the Voronoi diagram vertex at the endpoint of the current path segment, respectively. , They represent the first The horizontal and vertical coordinates of the center point of each obstacle; The extraction of core path nodes from the initial path point set based on a preset safe distance threshold includes: If the minimum distance between a path segment and a representative obstacle is less than a preset safe distance threshold, then the path points at both ends of the path segment are set as path core nodes.

[0009] Optionally, the local path planning performed within each subtask using the reinforcement learning deep Q-network includes: S51. Define the state space and action space, and construct a multidimensional reward function; S52. The robot is positioned at the starting point of the current subtask, and the current state of the robot is obtained. S53. Input the robot's current state into the prediction network, generate and execute actions based on the defined action space, update the robot's position, obtain the robot's new state and calculate the instant reward based on the multidimensional reward function, and calculate the TD error based on the prediction network and the target network. S54. Design a priority experience replay strategy to train and update the parameters of the prediction network, and then softly update the parameters of the target network. The priority experience replay strategy receives the robot's current state, the current sub-task target point position, the immediate reward, the TD error, and experience samples from the experience replay pool as input. It calculates the experience sampling probability, samples the experience samples from the experience replay pool based on the experience sampling probability, and inputs the sampled experience samples into the prediction network to update the prediction network parameters. The experience samples include the robot's current state, the executed action, the immediate reward, the robot's new state, the network training termination flag, and the dynamically weighted priority generated by the experience sampling probability. The empirical sampling probability is calculated as follows: ; in Indicates the first The empirical sampling probability of each empirical sample; , These represent the weighting parameters for immediate reward and TD error, respectively; , They represent the first Instant reward and TD error for each experience sample; This represents the index of the experience samples in the experience replay pool; This represents the index of the experience sample in the experience replay pool used for traversal calculation and summation. This represents the total number of experience samples in the experience replay pool; This indicates the operation of taking the minimum value; This indicates the logarithmic operation; This indicates the modulus length operation, used in the above formula to take the absolute value; S55. Determine whether the robot has reached the target point of the current subtask; S56. If the robot has not reached the target point of the current subtask, update the robot state and re-execute steps S53-S55; if the robot has reached the target point of the current subtask, the current subtask ends, and the local planning path of the current subtask is generated using the prediction network parameters at this time, and the local path planning of the next subtask is performed.

[0010] Optionally, the state space includes target features, environmental perception features, and temporal features; The target features include the Euclidean distance and azimuth angle between the robot's position and the current subtask target point position; The environmental perception features include the distribution of static and dynamic obstacles within the robot's perception range and the predicted landing point (PLP) of dynamic obstacles; the predicted landing point (PLP) of dynamic obstacles is generated by a pre-trained recurrent neural network (RNN) based on the historical displacement data of dynamic obstacles. The temporal features include a sequence of several environmental snapshots within the robot's perception range of obstacles, stacked in time steps. The action space, using an 8-neighborhood discrete movement method, is defined as follows: ; in Represents the action space. , These represent the robot's lateral and longitudinal displacements, respectively. The action space, through -The "greedy" strategy selects the action; The multidimensional reward function includes distance reward. Orientation alignment reward Obstacle avoidance rewards Predicting the landing point is close to the penalty Step count penalty ; The distance reward The definition is as follows: ;in express The Euclidean distance between the robot at any given moment and the target point of the current subtask. This represents the positive reward coefficient when the robot approaches the target point of the current subtask, and its value range is [range missing]. , This represents the negative penalty coefficient when the robot moves away from the current subtask target point, with a value range of... ; The distance reward In each subtask, the Euclidean distance between the robot and the target point of the current subtask is... When the value is first less than the preset position tolerance threshold, an additional stage completion reward will be granted. Value range ; The direction alignment reward The definition is as follows: ; in, , These are the horizontal and vertical coordinates of the robot's current position; , The x and y coordinates of the current subtask target point location; For reward scaling factor; The obstacle avoidance reward The definition is as follows: ; in, This represents the Euclidean distance between the robot's current position and the nearest obstacle. This represents the penalty coefficient during a collision. This represents the penalty coefficient for near-field obstacle avoidance. This represents the mid-range obstacle avoidance bonus coefficient. This represents the long-distance obstacle avoidance bonus coefficient. This represents the long-distance obstacle avoidance bonus coefficient; , , , , The range of values ​​is ; The predicted landing point is close to the penalty The definition is as follows: ; in, This represents the relative distance between the robot and the predicted landing point PLP of the dynamic obstacle; This indicates that the predicted landing point is close to the penalty coefficient, and the range of values ​​is... ; This indicates that the preset predicted landing point is close to the judgment threshold; The step penalty Give a small negative reward at every moment value range ; The multidimensional reward function is defined as follows: ; in, This represents a multidimensional reward function.

[0011] Optionally, the prediction network includes a CNN+MLP module, an LSTM layer, and a dense layer; wherein CNN represents a convolutional neural network, MLP represents a multilayer perceptron, and LSTM represents a long short-term memory network; the dense layer uses a DuelingDQN structure to output action decisions. The expected cumulative reward of the prediction network output satisfies: ; in, Indicates based on robot state With action Expected cumulative reward; For robot state-based State value; This indicates the mold length taking operation, in the above formula Representing the action space The total number of actions in the process; Representing the action space Any candidate action in the list.

[0012] Optionally, training and updating the parameters of the prediction network and the target network includes: Update the expected cumulative reward based on the multi-step optimal Bellman equation: ; in, Indicates about robot state at any moment and Moment of action Expected cumulative reward; symbol This indicates an update to; This represents the predicted network learning rate, which is predefined manually and has a range of values. ; Indicates the summation step index for multi-step learning; This represents the total number of steps in a multi-step learning process. Indicates the first The discount factor for the step, with a value range of 100%. ; express Instant rewards for each moment; express robot state at any moment Next, take candidate actions Expected cumulative reward; Indicates the robot's state Next, traverse all candidate actions. The maximum expected cumulative reward obtained; Therefore, the strategy is determined as follows: ; in, Indicates based on robot state With action Strategies; Indicates taking The largest value , The round index of strategy iteration.

[0013] This invention also provides a robot navigation system for applying the aforementioned robot dynamic obstacle avoidance planning method based on path critical nodes and deep reinforcement learning, comprising: The map preprocessing module is used to perform binarization and rasterization preprocessing on the actual scene map, extract the coordinates of representative obstacles, and generate a raster map. The global path planning module is used to construct and optimize the Voronoi diagram and its adjacency matrix based on the raster map, extract the core nodes of the path, and construct the set of core nodes of the path. The local path planning module is used to divide the global path planning task into several sub-tasks based on the core node set of the path, and to perform local path planning in each sub-task based on the reinforcement learning Q network, generating the local planning path of each sub-task and forming the local planning path instruction output. The environmental perception module is used to collect information on static and dynamic obstacles within the robot's preset perception range, and predicts the trajectory landing point PLP of dynamic obstacles through the recurrent neural network (RNN) module. The motion execution module is used to receive the local planning path instructions output by the local path planning module and control the robot to move according to the global planning path formed by the local planning path. The model training module is used to set training parameters and update the parameters of the prediction network and the target network.

[0014] By employing the above technical solution, this invention provides a robot dynamic obstacle avoidance planning method and navigation system based on path key nodes and deep reinforcement learning, which has at least the following beneficial effects: (1) This invention constructs a safe area through Voronoi diagram, combines dynamic obstacle trajectory prediction with multimodal feature fusion, and achieves accurate avoidance of static and dynamic obstacles. It is suitable for various complex scenarios such as indoor, open, and maze, enhances the adaptability to dynamic environments, and improves the success rate of path planning. (2) This invention divides the global path planning task into several sub-tasks through a multi-stage path segmentation strategy. Each sub-task corresponds to an independent target point. A multi-dimensional reward function is designed for each sub-task, including distance reward, direction alignment reward, obstacle avoidance reward, predicted landing point proximity penalty, and step penalty. An additional stage completion reward is granted when the sub-task is completed, realizing the transformation from sparse reward to dense reward. (3) This invention assigns higher sampling weights to key experiences with high rewards and large TD errors by calculating the empirical sampling probability; whereby key experiences refer to samples that can significantly correct model prediction bias and provide high-value reward gradients (such as samples that are close to the target, successful / failed obstacle avoidance, and samples with large TD errors); this strategy can improve the utilization rate of key experiences, accelerate the model convergence speed, avoid interference from invalid samples to training, reduce sample noise and estimation bias, and significantly improve training efficiency and planning accuracy. (4) The network training and update algorithm designed in this invention adopts a CNN-LSTM-Dueling composite structure to accurately extract spatiotemporal features and separate value and advantage flows; combined with dynamic weighted priority experience replay, it improves the utilization rate of key experience, accelerates convergence, and reduces variance; multi-step Bellman update improves the accuracy of Q-value estimation. By combining CNN to extract spatial features, LSTM to model temporal features, and Dueling structure to separate value and advantage flows, the stability of the target network soft update is guaranteed; (5) This invention reduces computational complexity by filtering core nodes of the path from a global perspective and dynamically adjusts the path based on reinforcement learning deep Q network from a local perspective, thereby shortening the average execution time and total path length of the algorithm and achieving stable operation in a complex environment where static and dynamic obstacles coexist. (6) This invention provides safe target points for local path planning through global path planning, and local path planning dynamically adapts to environmental changes, achieving a balance of safety, flexibility and efficiency. Its overall performance is superior to traditional swarm intelligence algorithms and conventional reinforcement learning algorithms. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the overall algorithm of the present invention; Figure 2This is a flowchart of the global path planning stage of the present invention; Figure 3 This is a flowchart of the local path planning stage of the present invention; Figure 4 This is a structural diagram of the deep Q-network model constructed for local path planning in this invention; Figure 5 This is a comparison diagram of the global path planning of the algorithm of this invention and existing algorithms; Figure 6 This invention provides path planning curves for various complex scenarios. Figure 7 This invention demonstrates obstacle avoidance performance under different numbers of obstacles. Figure 8 The reward curves for different reinforcement learning algorithms in this invention; Figure 9 This is a flowchart of the algorithm for extracting key points from a path according to the present invention. Detailed Implementation

[0016] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0017] Those skilled in the art will understand that all or part of the steps in the implementation of the methods of the embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0018] Please refer to Figures 1-9This illustration shows a specific implementation of the present embodiment. This embodiment constructs a grid map based on a real-world scene map, then constructs and optimizes a Voronoi diagram and its adjacency matrix to generate a preliminary path point set, and then constructs a path core node set. The global path planning task is divided into several sub-tasks. Local path planning is performed within each sub-task based on a reinforcement learning deep Q-network. A multi-dimensional reward function is designed by combining trajectory landing point prediction and environmental potential field. A priority experience replay strategy based on dynamic weighting of immediate reward and TD error (immediate sequential difference error) is adopted. At the same time, CNN, MLP and LSTM networks are integrated to construct a multi-modal feature fusion model to achieve accurate obstacle avoidance and path adjustment in dynamic environments. This effectively improves the accuracy, robustness and real-time performance of the robot's path planning in complex dynamic environments, and is applicable to various navigation scenarios such as industrial automation, service robots, and intelligent warehousing.

[0019] This embodiment proposes a robot dynamic obstacle avoidance planning method based on path critical nodes and deep reinforcement learning. This method decomposes the robot's dynamic obstacle avoidance planning into two stages: global path planning and local path planning. The overall process conforms to... Figure 1 The algorithm framework is shown.

[0020] Figure 1 This is a flowchart of the overall algorithm framework of the robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning in this invention. It mainly shows the complete process of map preprocessing, global path planning, local path planning and network training and iterative optimization. The “metric map” shown in the figure corresponds to the raster map generated in step S1 of the paper, that is, the discrete raster matrix representing the distribution of environmental obstacles obtained after binarizing and rasterizing the actual scene map.

[0021] The robot dynamic obstacle avoidance planning method based on path critical nodes and deep reinforcement learning proposed in this embodiment includes the following steps: (1) Global path planning stage S1. Map Preprocessing (processing the captured map based on the actual map): Perform binarization and rasterization preprocessing on the actual scene map, set thresholds to extract representative obstacle areas, determine raster values, generate a raster map, and extract the coordinates and center points of representative obstacles.

[0022] As a preferred embodiment of step S1, the specific process includes: S11. Convert the actual scene map into an image matrix. ( It is the space of real numbers; , These represent the number of rows and columns in the actual map scenario, respectively. Their values ​​can be adjusted according to the actual scenario; in this embodiment, they are set as follows: , ) and through the threshold function Binarization is performed to obtain a binary image matrix. The threshold function satisfies: ; The preset threshold is set to [value] in this embodiment. ; For image matrix The element values ​​in the table. Passed through a threshold function. Image matrix Positions with a value greater than or equal to 128 in the binary image matrix The corresponding value is set to 1 (obstacle area), and the corresponding value for positions less than 128 is set to 0 (passable area). This step is... Figure 2 Preprocessing is a core component of global path planning.

[0023] S12, convert the binary image matrix According to the preset reduction ratio coefficient Rasterization processing is performed (set in this embodiment) ), will the original (In this embodiment, the grid is 80×68) divided into (In this embodiment, there are 64×54) new grid areas, where This indicates a floor operation, resulting in a rasterized matrix.

[0024] S13. Next, the function is judged based on the characteristics of representative obstacles. ( , (These represent the number of raster rows and columns of the binary image matrix corresponding to each raster block in the rasterized matrix, respectively.) The system identifies and extracts representative obstacle regions in the rasterized matrix that satisfy preset connectivity and size thresholds, determines raster values, and outputs a raster map. This process corresponds to Figure 2 The "rasterization" step in the process.

[0025] Specifically, the representative obstacle feature determination function traverses the connected regions in the rasterized matrix. If the proportion of pixels with a value of 1 in the connected region exceeds a preset threshold, the function will determine the obstacle. And the area of ​​the connected region is greater than a preset size threshold. The connected region is then marked as a representative obstacle region, and the grid cells within the representative obstacle region are designated as representative obstacle grid cells. A grid map is constructed using the coordinates of the representative obstacle grid cells as the representative obstacle coordinates. In this grid map, the grid cell value at the position corresponding to the representative obstacle coordinates is 1, and the grid cell value at all other positions is 0.

[0026] S14. Extract the coordinates of representative obstacles and the center point of representative obstacles from the generated grid map. The center point of representative obstacles is obtained by traversing the grid map to identify the grid occupied by the obstacles. The geometric center of the connected representative obstacle grid area is calculated, which is the mean value of all grid coordinates occupied.

[0027] The "representative obstacle" in this invention differs from a conventional obstacle in that: a conventional obstacle contains all pixels with a value of 1 in the binary image matrix, while a representative obstacle, through the above steps S12-S13, filters out invalid interference such as minor noise and isolated points, retaining only obstacles that actually hinder the robot's path planning, thereby reducing the computational complexity of subsequent Voronoi diagram construction and path search.

[0028] S2. Generate Voronoi diagram (based on raster map, raster map Voronoi diagram construction): Construct a set of obstacle center coordinates based on the raster map, and then construct the Voronoi diagram and its adjacency matrix.

[0029] As a preferred embodiment of step S2, the specific process includes: S21. Load and traverse the raster map. Obtain obstacle coordinates and extract the obstacle coordinate set. ,in Represents a raster map The Middle Line number Column elements.

[0030] Then, the set of obstacle center coordinates is constructed, mathematically represented as follows: ; in Represents the set of coordinates of the obstacle's center; , A set of representative obstacle coordinates The horizontal and vertical coordinates of representative obstacles; , These represent the horizontal and vertical offsets from the geometric center coordinates of a raster to the coordinates of a representative obstacle in the raster map. They are used to convert raster vertex coordinates to the geometric center coordinates of the corresponding raster. The values ​​are all within the range of... The value is determined by the grid coordinate origin definition and obstacle center positioning requirements. After replacement, it is necessary to ensure that the coordinates still fall within the current grid. In this embodiment, it is taken as 0.5.

[0031] S22, Set by the center coordinates of the obstacle For a given set of vertices, construct a Voronoi graph and generate an adjacency matrix representing the vertex connectivity of the Voronoi graph. , This represents the number of vertices in a Voronoi diagram; the elements of the adjacency matrix satisfy: ; Among them, vertex , These are the Voronoi diagrams for the 1st, 2nd, and 3rd. Vertex, First vertex, Representing the adjacency matrix The Middle Line number The elements of the column represent vertices in the Voronoi diagram. , Connectivity between them; , These represent the vertices in the Voronoi diagram. The x and y coordinate values, , These represent the vertices in the Voronoi diagram. The x and y coordinate values. This process corresponds to... Figure 2 The "Generate Voronoi Diagram" step in the process.

[0032] This invention constructs safe zones using Voronoi diagrams, combining dynamic obstacle trajectory prediction with multimodal feature fusion to achieve accurate obstacle avoidance for both static and dynamic obstacles. It is adaptable to various complex scenarios such as indoors, open spaces, and mazes, enhancing adaptability to dynamic environments and improving path planning success rates. Specifically, this invention constructs a Voronoi diagram based on obstacle distribution on a gridded map, using Voronoi edges as the initial safe path skeleton. Then, it performs preliminary path planning on the Voronoi diagram topology using the A* algorithm. Compared to planning directly on a gridded map, this design significantly reduces the path search space and computational complexity, while naturally ensuring a safe distance between the path and obstacles, improving the safety and accessibility of the global path. Dynamic obstacle trajectory prediction is achieved using a pre-trained recurrent neural network (RNN). This RNN takes the historical displacement sequence of dynamic obstacles as input, learns their motion patterns, and outputs the predicted dynamic obstacle trajectory landing point (PLP) for several future time steps. By sensing the movement trend of dynamic obstacles in advance, it avoids collision risks during the local path planning stage, improving obstacle avoidance robustness in dynamic environments.

[0033] S3. Skeleton Extraction and Optimization: Map the vertex coordinates of the Voronoi graph to their corresponding raster indices in the raster map, and optimize the Voronoi graph and its adjacency matrix by filtering obstacles vertices, filtering invalid edges, and removing redundant nodes.

[0034] As a preferred embodiment of step S3, the specific process includes: S31. Raster Coordinate Transformation: Mapping the vertex coordinates of the Voronoi diagram to their corresponding raster indices in the raster map to filter vertices located in obstacle regions and edges intersecting with obstacles; the mapping formula is as follows: ; ; in, , Let x and y represent the x and y coordinates of a vertex in the Voronoi diagram, respectively. , They represent , The corresponding raster index value in the raster map; , These are the horizontal and vertical coordinates of the bottom left corner of the raster map; This indicates the resolution of the raster map; in this embodiment, it is set to... ; S32. Obstacle Vertex Filtering: If the grid value of a vertex in the Voronoi diagram corresponding to its position in the grid map is 1, indicating that the grid is a representative obstacle grid, then the vertex is determined to be a representative obstacle vertex, and the obstacle is filtered from the adjacency matrix of the Voronoi diagram. Delete the row and column corresponding to the vertex to eliminate invalid vertices inside representative obstacles.

[0035] S33. Invalid Edge Filtering: Traverse all edges in the Voronoi diagram. Calculate the shortest perpendicular distance between each edge and the representative obstacle based on the coordinates of the representative obstacle. If this distance is less than or equal to a preset threshold, the edge is determined to intersect with the representative obstacle and is considered an invalid edge. Set the Voronoi diagram adjacency matrix. The corresponding element is 0; otherwise, the Voronoi diagram adjacency matrix is ​​retained. The corresponding elements are used to eliminate invalid edges that pass through representative obstacles.

[0036] S34. Redundant Node Removal: Traverse the vertices in the Voronoi graph. If three vertices are collinear, the middle vertex is considered a redundant node and removed from the Voronoi graph.

[0037] Collinearity of three points is achieved by calculating the cross product of vectors: taking three vertices in the Voronoi diagram. , , The cross product is calculated as follows: ; in and Representing the vertices respectively , The vector composed of vertices , The vector formed; , Vertices in the Voronoi diagram The x and y coordinate values, , Vertices in the Voronoi diagram The x and y coordinate values, , Vertices in the Voronoi diagram The x and y coordinate values. When the cross product calculation result satisfies: At that time, make a judgment Redundant nodes are removed from the Voronoi diagram.

[0038] Steps S2-S3 implement Voronoi graph construction (including Voronoi graph vertex set and edge set extraction) and adjacency matrix establishment.

[0039] S4. Path Core Node Selection (Based on the metric map, planning the Voronoi diagram) Figure 1 The “metric map” in this context is the “rasterized matrix” obtained in step S12; then the planned map is converted into a raster map: the A* algorithm is applied to the optimized Voronoi graph and its optimized adjacency matrix to generate a preliminary path point set, the minimum distance between each path segment and representative obstacles is calculated, and the path core nodes are extracted from the preliminary path point set based on a preset safe distance threshold to construct a path core node set; the path segment is the line segment formed by connecting adjacent path points in the preliminary path point set.

[0040] As a preferred embodiment of step S4, the specific process includes: S41. Apply the A* algorithm to the vertices of the optimized Voronoi graph to generate a preliminary set of path points. .

[0041] Specifically, based on the adjacency matrix optimized in step S3, the connectivity between vertices is determined and the path cost is calculated. In this process, the adjacency matrix is ​​used to represent the connection relationship between vertices in the Voronoi graph and the weight of the corresponding path, so as to improve the efficiency of path search.

[0042] S42, Based on a preset safety distance threshold The path segment (preliminary path point set) is calculated using the following formula. Minimum distance between the line segment formed by connecting adjacent path points and the obstacle: ; in, This represents the minimum distance between a path segment and a representative obstacle. This indicates taking the minimum value; This represents the Voronoi diagram vertex coordinates of the starting point of the current path segment in the preliminary path point set. The Voronoi diagram coordinates of the endpoints of the current path segment in the initial path point set; Indicates the length of the path segment; Indicates the origin of the point , and the The center point of each obstacle The determinant formed by the matrix is ​​defined as follows: ; in , These represent the x and y coordinates of the Voronoi diagram vertex at the starting point of the current path segment, respectively. , These represent the x and y coordinates of the Voronoi diagram vertex at the endpoint of the current path segment, respectively. , They represent the first The horizontal and vertical coordinates of the center point of each obstacle.

[0043] S43, when The path points at both ends of the path segment are designated as path core nodes, forming a path core node set. In this embodiment, the following settings are provided. .

[0044] The specific screening process is as follows: Figure 9 As shown, the process is as follows: Initialize the core node set of the path If empty, set the index for the outer loop traversal. ; judge If established, it will Add to the core node set of the path Get the Voronoi diagram vertex coordinates (key points) of the starting point of the current path segment. ,make ;in This indicates the search for the initial path set. The total number of elements in the array. Represents the initial path point set Inner Path points, This indicates that the inner loop iterates over the index; judge If true, then take the path point. ; Calculate the minimum distance between the path segment and the obstacle. And determine whether it satisfies ; If true, join in ,renew And return to the outer loop; If not successful, update. And continue to the next inner loop; Finally, the path endpoint join in Output path core node set .

[0045] Steps S3-S4 correspond to Figure 2 The section on "Extracting the skeleton from the Voronoi diagram to generate the critical path" is as follows: Step S31 converts the map type; Step S32 divides the vertices of the Voronoi diagram into valid points and filter points to filter vertices within obstacles; Step S33 divides the edges of the Voronoi diagram into valid edges and filter edges (i.e., invalid edges; an edge is valid if its shortest perpendicular distance (Distance) to an obstacle is greater than a preset threshold, otherwise it is invalid / filtered); Step S34 identifies redundant points. If any three vertices in the Voronoi diagram are not collinear, they are considered to have no redundant points; if they are collinear, the middle point is identified as a redundant point and removed. Step S4 corresponds to... Figure 2 The process involves generating critical path points; ultimately, the generated set of core path nodes is formatted as a TXT map and stored.

[0046] Step S4 uses A* to initially screen the point set in the image and extracts path points using the algorithm of this invention to remove redundant points and edges; at the same time, it records the key point paths and stores the raster map as a .txt file.

[0047] (2) Local path planning stage S5. Based on the core node set of the path, the global path planning task is divided into several sub-tasks. Based on the reinforcement learning deep Q network, local path planning is performed in each sub-task to generate the local planned path of each sub-task.

[0048] As a preferred embodiment of step S5, the specific process includes: S51. Define the state space and action space, and construct a multidimensional reward function; The state space includes target features, environmental perception features, and temporal features; Target features include the Euclidean distance between the robot's position and the current subtask target point. and azimuth ,in , These are the horizontal and vertical coordinates of the robot's position, respectively. , The x and y coordinates of the current subtask target point location; Environmental perception features include the robot's perception range (encoding radius) of obstacles. , representing the effective radius range for perceiving surrounding obstacles centered on the current robot (which can be predefined manually) within which the static and dynamic obstacles are distributed and the predicted landing point PLP of the dynamic obstacles is generated by a pre-trained recurrent neural network (RNN) based on the historical displacement data of the dynamic obstacles (the RNN can be used to predict the position of the dynamic obstacles in the future several time steps). Temporal features include a sequence of several environmental snapshots within the robot's perception range of obstacles, stacked in time steps. This embodiment collects the most recent snapshots. The scene frames are stacked to construct a sequence. The scene frames contain information such as obstacle distribution and target location, and are generated according to the preset number of time frames to capture the patterns of environmental changes.

[0049] The composition of the state space and Figure 4 The multimodal feature fusion requirements of the input layer of the network model shown are consistent. Figure 4 This is a structural diagram of the deep Q-network model constructed for local path planning in this invention. In the input layer, based on the scene around the robot grid, the grid environment spatial features are extracted through two layers of CNN (the extracted feature scale is transformed from 7*7 to 7*7*16 and then further transformed to 4*4*32). At the same time, a recurrent neural network (RNN) is used to extract the predicted landing point (PLP) of the dynamic obstacle based on the last frame number (4 frames in this embodiment) and perform indexing (the index is the action index of the action space, used to map the position of the predicted landing point (PLP) of the dynamic obstacle to the index consistent with the action space, so as to achieve feature alignment). A multilayer perceptron (MLP) is used to process the temporal information of the target and dynamic obstacles based on the distance and direction between the robot and the target point, and it is connected with the output of the second layer of CNN (the feature scale after connection is 32*4*4+64). After the temporal dependency is modeled by the long short-term memory (LSTM) layer, it is input into the variant part of the dense layer Dueling DQN to separate the state value stream. With the advantage of action flow (The feature scale is 256*128, which is then mapped to 128*1 and 128*9 respectively, state value stream) A normalization operation with a coefficient of 1.0 is used to achieve a uniform numerical scale for the average value of the network predictions, ensuring computational stability. This is applied to the action-dominant flow. Normalization is performed, and then a convolution operation with a kernel size of 7*7, a stride of 3, and padding of 1 is applied to the LSTM temporal feature mapping with an input sequence length of 7 frames, an adaptive dimension index of -1, and a feature flattening index of -2, to unify the scale of the advantage values ​​of each action and fuse them to obtain the action value function. Finally, the dimension with the highest value is selected by taking the maximum value, and the corresponding optimal action index is output. This enables robots to perform local path planning and obstacle avoidance in dynamic environments.

[0050] The action space uses an 8-neighborhood discrete movement method, defined as follows: ; in Represents the action space. , These represent the robot's lateral and longitudinal displacements, respectively. Action space through The -greedy strategy selects actions, and the action selection satisfies the following: ; in, For a moment The action, for -Exploration rate of the greedy strategy This indicates the generation of random numbers in the range of 0-1. Indicates selection The action with the highest value , Indicates based on robot status at all times With action Expected cumulative reward. Initial exploration rate. It decays exponentially with the number of training rounds, with a decay coefficient of 1. This ensures that the training is fully explored in the early stages and stably utilized in the later stages.

[0051] Figure 3This is a flowchart of the local path planning stage of the present invention, fully presenting the entire process of local path planning based on the SW-RDQN algorithm (a stage-key & weight-balanced replay Rainbow deep Q network algorithm with dynamic weighting of immediate reward and TD error) designed in this invention: ① The input network extracts the image and data information of the current state and outputs the fused features to the prediction network; ② The prediction network predicts the expected value of the Q value of each action (up, down, ..., stop) (corresponding to 0.12, 0.36, ..., 0.05), providing a basis for action selection; ③ The action direction is selected based on the ε-greedy strategy to balance exploration and utilization; ④ The action is input into the environment to perform interaction and obtain environmental feedback; ⑤ The environment output is executed, and it is determined whether the round has ended. If it has ended, a round reward is given (i.e., an additional positive reward is given when the robot successfully reaches the target point, and an additional negative reward is given if a collision or timeout occurs, used to evaluate the overall completion quality of this path planning task and guide the model to learn the globally optimal path); if it has not ended, a temporary reward is given (i.e., the designed multi-dimensional reward function). The system comprehensively considers dimensions such as the robot's safe distance from obstacles, changes in distance to the target point, and motion smoothness to guide the robot in real time to execute safe and efficient single-step actions. It then continues executing the action; ⑥ transmits new states and rewards, observes the current state, and updates the input network data by combining image information (grid map) and data information (the robot's relative pose to the current sub-task target point); ⑦ calculates action rewards based on rewards and the memory bank (as an experience replay pool); ⑧ uses six-tuples... The experience is stored in a memory bank in the following format: S represents the robot state, A represents the action, R represents the temporary reward, N_S represents the new robot state, P represents the experience sampling probability, and D represents the round termination flag (1 indicates the round is over, 0 indicates the round is not over); ⑨ Perform weighted priority experience revisit, combining high reward and high TD error, and take multiple steps to weighted sample high-value experience; ⑩ Input the sampled experience into the prediction network for training; ⑪ Calculate the loss gradient of the prediction network to prepare for parameter updates; ⑫ Train the neural network and optimize the prediction network parameters according to the gradient; ⑬ Synchronously update the parameters of the target network and the prediction network to stabilize the training process. The prediction network and the target network have the same structure, consisting of two CNN layers, one MLP layer (CNN+MLP module), an LSTM layer, and a dense layer, used to extract spatiotemporal features and evaluate the value of actions. The RNN module predicts the landing point of dynamic obstacles based on historical trajectories, providing a basis for obstacle avoidance, and integrates other scene information to update the environmental state, providing a forward-looking basis for obstacle avoidance. The memory bank stores interaction experience to support the playback of priority experience. The reward module corresponds to a multi-dimensional reward function, guiding the robot to plan paths safely and efficiently.

[0052] The multidimensional reward function includes distance reward. Orientation alignment reward Obstacle avoidance rewards Predicting the landing point is close to the penalty Step count penalty ; The distance reward The definition is as follows: ;in express The Euclidean distance between the robot at any given moment and the target point of the current subtask. This represents the positive reward coefficient when the robot approaches the target point of the current subtask, and its value range is [range missing]. In this embodiment, the following settings are provided. ; This represents the negative penalty coefficient when the robot moves away from the current subtask target point, with a value range of... In this embodiment, the following settings are provided. ; The distance reward In each subtask, the Euclidean distance between the robot and the target point of the current subtask is... When the value is first less than the preset position tolerance threshold, an additional stage completion reward will be granted. Value range .

[0053] The direction alignment reward The definition is as follows: ; Among them, among them, , These are the horizontal and vertical coordinates of the robot's current position; , The x and y coordinates of the current subtask target point location; To reward the scaling factor, this embodiment sets ; The obstacle avoidance reward The definition is as follows: ; in, This represents the Euclidean distance between the robot's current position and the nearest obstacle. This represents the penalty coefficient during a collision, which is set in this embodiment. ; This represents the near-distance obstacle avoidance penalty coefficient, which is set in this embodiment. ; This represents the mid-range obstacle avoidance bonus coefficient, which is set in this embodiment. ; This embodiment indicates that the following settings are provided. ; This represents the long-distance obstacle avoidance reward coefficient, which is set in this embodiment. ; The predicted landing point is close to the penalty The definition is as follows: ; in, This represents the relative distance between the robot and the predicted landing point PLP of the dynamic obstacle; This indicates that the predicted landing point is close to the penalty coefficient, and the range of values ​​is... In this embodiment, it is set to -1.25; This indicates that the preset predicted landing point is close to the judgment threshold; The step penalty Give a small negative reward at every moment value range ; The multidimensional reward function is defined as follows: ; in, This represents a multidimensional reward function.

[0054] Please refer to Table 1 below for the settings of various reward parameters: Table 1. Reward Function Parameter Settings

[0055] This invention employs a multi-stage path segmentation strategy to divide the global path planning task into several sub-tasks, each corresponding to an independent target point. For each sub-task, a multi-dimensional reward function is designed, incorporating distance reward, orientation alignment reward, obstacle avoidance reward, predicted landing point proximity penalty, and step count penalty. An additional stage completion reward is granted upon sub-task completion, transforming sparse rewards into dense rewards. Compared to traditional single-target rewards, this design provides more frequent feedback after each action, guiding the robot to quickly learn sub-task goal-oriented behavior and avoiding the slow learning problem caused by sparse rewards.

[0056] S52. The robot is positioned at the starting point of the current subtask, and the current state of the robot is obtained.

[0057] S53. Input the robot's current state into the prediction network, generate and execute actions based on the defined action space, update the robot's position, obtain the robot's new state and calculate the immediate reward based on the multidimensional reward function, and calculate the TD error based on the prediction network and the target network.

[0058] The prediction network consists of a CNN+MLP module, an LSTM layer, and a dense layer; where CNN stands for Convolutional Neural Network, MLP stands for Multilayer Perceptron, and LSTM stands for Long Short-Term Memory Network. The CNN+MLP module extracts visual and numerical features, the LSTM layer captures temporal correlations, and the dense layer uses a DuelingDQN structure to output action decisions.

[0059] The expected cumulative reward for predicting the network output satisfies: ; in, Indicates based on robot state With action Expected cumulative reward; For robot state-based State value; This indicates the mold length taking operation, in the above formula Representing the action space The total number of actions in the process; Represents the action space used for summation calculations. Any candidate action in the list.

[0060] TD error calculation satisfies ,in , To predict the network output, For the target network output, , Indicates the first TD error of empirical samples Indicates the target Q value. Indicates the first Instant rewards for each experience sample Indicates the discount factor. This indicates that the prediction network has parameters Based on robot state ,action Q-value estimation, This indicates the current parameters of the predicted network. , They represent the first Robot states and actions from a set of empirical samples. Indicates based on Select the optimal action for the next obtained state. This represents the target network parameters at the current time (before the soft update). Indicates selection The action that reaches its maximum value.

[0061] S54. Design a priority experience replay strategy to train and update the parameters of the prediction network, and then softly update the parameters of the target network. The priority experience replay strategy receives the robot's current state, the current sub-task target point position, the immediate reward, the TD error, and experience samples (i.e., historical experience samples) in the experience replay pool as input. It calculates the experience sampling probability, samples the samples in the experience replay pool based on the experience sampling probability, and inputs the sampled samples into the prediction network to update the prediction network parameters. The experience samples include the robot's current state, the executed action, the immediate reward, the robot's new state, the network training termination flag, and the dynamically weighted priority generated by the experience sampling probability. The empirical sampling probability is calculated as follows: ; in Indicates the first The empirical sampling probability of a single empirical sample; , These represent the weighting parameters for immediate reward and TD error, respectively; , They represent the first Instant reward and TD error for each experience sample; This represents the index of the experience samples in the experience replay pool; This represents the index of the experience sample in the experience replay pool used for traversal calculation and summation. This represents the total number of experience samples in the experience replay pool; This indicates the operation of taking the minimum value; This indicates the logarithmic operation; This indicates the modulus length operation, used in the above formula to take the absolute value.

[0062] This invention assigns higher sampling weights to key experiences with high rewards and large TD errors by calculating the probability of empirical sampling. Key experiences refer to samples that can significantly correct model prediction bias and provide high-value reward gradients (such as samples close to the target, successful / failed obstacle avoidance, and samples with large TD errors). This strategy can improve the utilization rate of key experiences, accelerate model convergence, avoid interference from invalid samples in training, reduce sample noise and estimation bias, and significantly improve training efficiency and planning accuracy.

[0063] The prediction network updates the expected cumulative reward based on the multi-step optimal Bellman equation: ; in, Indicates about robot state at any moment and Moment of action Expected cumulative reward ( (It is itself the action value function of a deep Q-network); symbol This indicates an update to; This represents the predicted network learning rate, which is predefined manually and has a range of values. ; Indicates the summation step index for multi-step learning; This represents the total number of steps in a multi-step learning process. Indicates the first The discount factor for the step, with a value range of 100%. ; express Instant rewards for each moment; express robot status at all times Next, take candidate actions Expected cumulative reward; Indicates the robot's state Next, traverse all candidate actions. The maximum expected cumulative reward obtained; Therefore, the strategy is determined as follows: ; in, Indicates based on robot state With action Strategies; Indicates taking The largest value , This represents the round index of the strategy iteration, and its value is a non-negative integer.

[0064] Set the parameters as shown in Table 2 below, including the discount factor. Learning rate Reward weighting parameters TD error weighting parameters Number of training rounds Maximum number of moves per round The parameters are configured as follows Figure 3 The basic input to the local path planning algorithm framework shown is as follows.

[0065] Table 2. Experimental Parameter Settings

[0066] Model training: The SW-RDQN algorithm designed in this invention is used for training. This algorithm covers steps S53 to S54. The core is to use the CNN-LSTM-Dueling DQN composite structure to extract spatial and temporal features, separate the state value stream and the action advantage stream to accurately evaluate the action value, and combine the priority experience replay strategy based on the dynamic weighting of the TD error of the immediate reward to prioritize the sampling of high-value experience.

[0067] In step S53, a memory bank mechanism is introduced as an experience replay pool, with the memory bank consisting of six tuples. Store experience samples ( The current state of the robot For the action to be performed, For instant rewards, For the robot's new state, For network training termination flag, (This is a dynamically weighted priority generated from the probability of empirical sampling). Empirical samples are generated and encapsulated into a memory bank after the robot interacts with the environment.

[0068] Each time, 32 empirical samples are drawn from the memory, and the prediction network parameters are updated using gradient descent. The target network parameters are then adjusted accordingly. ( Weighted update, where This represents the target network parameters after the soft update. Represents the target network soft update coefficient; TD error calculation satisfies ( The training process strictly follows Figure 3 The closed-loop logic is shown.

[0069] The SW-RDQN algorithm designed in this invention for network training and updating employs a CNN-LSTM-Dueling composite structure to accurately extract spatiotemporal features and separate value and dominance flows. It combines dynamic weighted priority experience replay to improve the utilization rate of key experiences, accelerate convergence, and reduce variance. Multi-step Bellman updates enhance the accuracy of Q-value estimation. By jointly using CNN to extract spatial features, LSTM to model temporal features, and the Dueling structure to separate value and dominance flows, the stability of the target network's soft updates is ensured.

[0070] S55. Determine whether the robot has reached the target point of the current subtask.

[0071] S56. If the robot has not reached the target point of the current subtask, update the robot state and re-execute steps S53-S55; if the robot has reached the target point of the current subtask, the current subtask ends, and the local planning path of the current subtask is generated using the prediction network parameters at this time, and the local path planning of the next subtask is performed.

[0072] This invention reduces computational complexity by selecting core path nodes from a global perspective and dynamically adjusts the path based on a deep Q-network of reinforcement learning from a local perspective, thus shortening the average execution time and total path length of the algorithm and achieving stable operation in complex environments where static and dynamic obstacles coexist. At the same time, this invention provides safe target points for local path planning through global path planning, and the local path planning dynamically adapts to environmental changes, achieving a balance between safety, flexibility, and efficiency. Its overall performance is superior to traditional swarm intelligence algorithms and conventional reinforcement learning algorithms.

[0073] Step S5 implements training based on a raster map, reading in path point coordinate pairs generated by global path planning, and a raster environment map, converting the raster map into a metric map for processing, and using Stage-Key EnhancedRainbow DQN for training and deployment testing.

[0074] S6. Integrate the local planning paths of each subtask to generate a global planning path.

[0075] Deployment testing was conducted based on the above global planning path method: the trained model was deployed to the robot, environmental status information was collected in real time, and action commands were output through the model to realize path planning and obstacle avoidance navigation in dynamic environments. The judgment criteria for environmental output and action feedback strictly refer to the provisions in Table 3 below.

[0076] Table 3. Environmental Output

[0077] In Table 3, "Allocation Step Reward" refers to the basic reward for each step the robot takes to move within the passable area (i.e., step penalty). "Proceed to the next stage, allocate intermediate rewards" means allocate positive distance rewards. "Distributing the final episode's rewards" includes obstacle avoidance rewards. Distance Rewards Phased completion rewards and step count penalties These are used to constrain three scenarios: robot collision with obstacles, robot reaching the endpoint, and exceeding the step limit.

[0078] The following experiment will verify this.

[0079] The algorithm of this invention was tested in a complex environment containing 30 static obstacles and 10 dynamic obstacles. The global path planning from the starting point to the target using this algorithm, compared with existing algorithms (Dijkstra, A*, RRT*, and bi-A*), can be found in [reference needed]. Figure 5 On the left, you can refer to the line chart comparison of various indicators for global path planning. Figure 5 Right side, Figure 5The algorithm in this invention outperforms traditional Dijkstra, A*, RRT*, and bi-A* algorithms in terms of path length (grids), number of turns K (times / 100), number of inflection points (units), and navigation time T (seconds).

[0080] Robots in Figure 6 It can effectively avoid obstacles and reach the target point in three complex scenarios: indoor, open, and maze. Figure 6 a) is an indoor scene, with a size of 80*86; Figure 6 b) is an open scene, with a size of 120*100; Figure 6 c) is a maze scene with multiple turning points, with a size of 100*80; Figure 6 The dimensions of each scene are in pixels.

[0081] Comparison of obstacle avoidance performance under different numbers of obstacles (reward curve and success rate) Figure 7 As shown, Figure 7 In Chinese: S0D0 means no obstacles are set, S30D0 means only 30 static obstacles are set, S0D10 means only 10 dynamic obstacles are set, and S30D10 means both 30 static obstacles and 10 dynamic obstacles are set.

[0082] from Figure 7 It can be seen that the method of the present invention maintains a stable cumulative reward even in scenarios where static and dynamic obstacles coexist.

[0083] Figure 8 The reward curves of three traditional reinforcement learning algorithms, DQN, SAC, and TD3, are shown, combined with... Figure 7 As can be seen from the graph on the left, compared with traditional reinforcement learning algorithms, the algorithm in this invention has a faster convergence speed and a higher final reward value. Figure 8 In the diagram, a) is the reward curve for the DQN algorithm, b) is the reward curve for the SAC algorithm, and c) is the reward curve for the TD3 algorithm.

[0084] The overall performance comparison (including success rate, average time, and average path length) of the algorithm of this invention with existing navigation algorithms (IPSO, IGA) is shown in Table 4 below: Table 4. Comparison of different navigation algorithms

[0085] The algorithm of this invention significantly outperforms the IPSO and IGA algorithms in terms of average time (291.0ms) and average path length (349.15 grid units).

[0086] The following section conducts an ablation study on the method of this invention. In practical applications, this invention can adopt a transfer learning strategy of multi-scenario generalized pre-training + target scenario fine-tuning. First, the model is trained based on multi-scenario datasets to obtain general path planning capabilities, and then a small amount of target scenario data is used to achieve rapid adaptation.

[0087] The performance comparison of the modules in the ablation experiment is shown in Table 5 below: Table 5. Comparison of experimental performance of module ablation

[0088] In Table 5: ALL indicates the complete scheme of this invention; No_predict indicates the removal of the process of predicting the landing point of dynamic obstacles using a pre-trained recurrent neural network; No_TDR indicates the removal of the priority experience replay strategy; No_Due indicates the removal of the Dueling DQN structure; S_scene indicates single-scene training, that is, the model is trained using only single-environment scene data without multi-scene transfer learning; One&S indicates single-step reward update, that is, the network parameters are updated using only single-step reward, without using the expected cumulative reward update mechanism based on the multi-step optimal Bellman equation, used to verify the effectiveness of the multi-dimensional reward and multi-step update of this invention; No_all indicates the basic scheme without core modules, that is, the basic DQN scheme that removes all core improved modules such as dynamic obstacle landing point prediction PLP, dynamic weighted priority experience replay, and Dueling DQN, used to verify the overall gain of each core module of this invention.

[0089] As can be seen from Table 5, the success rate decreased significantly after removing the landing point prediction module, TDR weight, DuelingDQN and other core modules, which verifies the necessity and synergistic effect of each module in this invention.

[0090] This application also provides a robot navigation system for applying the robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning, including: The map preprocessing module is used to perform binarization and rasterization preprocessing on the actual scene map, extract the coordinates of representative obstacles, and generate a raster map. The global path planning module is used to construct and optimize the Voronoi diagram and its adjacency matrix based on the raster map, extract the core nodes of the path, and construct the set of core nodes of the path. The local path planning module is used to divide the global path planning task into several sub-tasks based on the core node set of the path, and to perform local path planning in each sub-task based on the reinforcement learning Q network, generating the local planning path of each sub-task and forming the local planning path instruction output. The environmental perception module is used to collect information on static and dynamic obstacles within the robot's preset perception range, and predicts the trajectory landing point PLP of dynamic obstacles through the recurrent neural network (RNN) module. The motion execution module is used to receive the local planning path instructions output by the local path planning module and control the robot to move according to the global planning path formed by the local planning path. The model training module is used to set training parameters and update the parameters of the prediction network and the target network.

[0091] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0092] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0093] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A dynamic obstacle avoidance planning method for robots based on path key nodes and deep reinforcement learning, characterized in that, include: S1. Perform binarization and rasterization preprocessing on the actual scene map, set a threshold to extract representative obstacle areas, determine raster values, generate a raster map, and extract the coordinates and center points of representative obstacles; the coordinates of the representative obstacles are the vertices of the representative obstacle grids in the raster map; the representative obstacles include dynamic obstacles and static obstacles. S2. Construct a set of obstacle center coordinates based on the raster map, and then construct the Voronoi diagram and its adjacency matrix; S3. Map the vertex coordinates of the Voronoi graph to their corresponding grid indices in the grid map, and optimize the Voronoi graph and its adjacency matrix by filtering obstacles vertices, filtering invalid edges, and removing redundant nodes. S4. Apply the A* algorithm to the optimized Voronoi diagram and its optimized adjacency matrix to generate a preliminary path point set. Calculate the minimum distance between each path segment and a representative obstacle, and extract the path core nodes from the preliminary path point set based on a preset safe distance threshold to construct a path core node set. The path segment is a line segment formed by connecting adjacent path points in the preliminary path point set. S5. Based on the core node set of the path, the global path planning task is divided into several sub-tasks. Based on the reinforcement learning deep Q network, local path planning is performed in each sub-task to generate the local planned path of each sub-task. S6. Integrate the local planning paths of each subtask to generate a global planning path.

2. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 1, characterized in that: The preprocessing of the actual scene map, including binarization and rasterization, setting thresholds to extract representative obstacle areas, determining raster values, and generating a raster map, includes: The actual scene map is converted into an image matrix and binarized using a threshold function to obtain a binary image matrix; in the binary image matrix, elements with a value of 0 represent passable areas, and elements with a value of 1 represent obstacle areas. The binary image matrix is ​​rasterized according to a preset reduction ratio to obtain a rasterized matrix; Traverse the connected regions in the rasterized matrix. If the proportion of pixels with a value of 1 in the connected region exceeds a preset threshold and the area of ​​the connected region is greater than a preset size threshold, then mark the connected region as a representative obstacle region. The grid cells in the representative obstacle region of the rasterized matrix are marked as representative obstacle grid cells, with their grid values ​​set to 1, and the grid values ​​of the remaining grid cells are set to 0, thus generating a raster map.

3. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 1, characterized in that: The process of constructing a set of obstacle center coordinates based on a raster map, and then constructing a Voronoi diagram and its adjacency matrix, includes: The grid map is traversed to extract a set of representative obstacle coordinates, and then a set of obstacle center coordinates is constructed, which is mathematically represented as follows: ； in Represents the set of coordinates of the obstacle's center; , A set of representative obstacle coordinates The horizontal and vertical coordinates of representative obstacles; , These represent the horizontal and vertical offsets from the geometric center coordinates of the raster map to the coordinates of the representative obstacle, respectively, with values ​​ranging from [value range missing]. ; Using the set of obstacle center coordinates as the vertex set, construct a Voronoi graph and generate an adjacency matrix representing the vertex connectivity of the Voronoi graph; The adjacency matrix has elements that satisfy: if the first element in the Voronoi diagram is... vertex , No. vertex If there is an edge connecting them, then ,otherwise ;in, Represents the adjacency matrix of the nth element. Line number The elements of the column represent vertices in the Voronoi diagram. , Connectivity between them; , These represent the vertices in the Voronoi diagram. The x and y coordinate values, , These represent the vertices in the Voronoi diagram. The horizontal and vertical coordinate values.

4. The robot dynamic obstacle avoidance planning method based on path critical nodes and deep reinforcement learning according to claim 1, characterized in that: The mapping of vertex coordinates of the Voronoi diagram to their corresponding raster indices in the raster map is mathematically represented as follows: ； ； in, , Let x and y represent the x and y coordinates of a vertex in the Voronoi diagram, respectively. , They represent , The corresponding raster index value in the raster map; , These are the horizontal and vertical coordinates of the bottom left corner of the raster map; Indicates the resolution of the raster map; The obstacle vertex filtering process involves removing the row and column corresponding to a vertex from the adjacency matrix of the Voronoi diagram if the corresponding grid cell in the grid map is a representative obstacle grid cell. The invalid edge filtering is based on the coordinates of representative obstacles to calculate the shortest vertical distance between the representative obstacle and each edge of the Voronoi graph. If the distance is less than or equal to a preset threshold, the corresponding element in the Voronoi graph adjacency matrix is ​​set to 0; otherwise, the corresponding element in the Voronoi graph adjacency matrix is ​​retained. The redundant node removal process involves traversing the vertices in the Voronoi graph. If three vertices are collinear, the middle vertex is considered a redundant node and removed from the Voronoi graph.

5. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 1, characterized in that: The minimum distance between each path segment and a representative obstacle is calculated as follows: ； in, This represents the minimum distance between a path segment and a representative obstacle. This indicates taking the minimum value; This represents the Voronoi diagram vertex coordinates of the starting point of the current path segment in the preliminary path point set. The Voronoi diagram coordinates of the endpoints of the current path segment in the initial path point set; Indicates the length of the path segment; Indicates the origin of the point , and the The center point of each obstacle The determinant formed by the matrix is ​​defined as follows: ； in , These represent the x and y coordinates of the Voronoi diagram vertex at the starting point of the current path segment, respectively. , These represent the x and y coordinates of the Voronoi diagram vertex at the endpoint of the current path segment, respectively. , They represent the first The horizontal and vertical coordinates of the center point of each obstacle; The extraction of core path nodes from the initial path point set based on a preset safe distance threshold includes: If the minimum distance between a path segment and a representative obstacle is less than a preset safe distance threshold, then the path points at both ends of the path segment are set as path core nodes.

6. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 1, characterized in that: The reinforcement learning-based deep Q-network performs local path planning within each subtask, including: S51. Define the state space and action space, and construct a multidimensional reward function; S52. The robot is positioned at the starting point of the current subtask, and the current state of the robot is obtained. S53. Input the robot's current state into the prediction network, generate and execute actions based on the defined action space, update the robot's position, obtain the robot's new state and calculate the instant reward based on the multidimensional reward function, and calculate the TD error based on the prediction network and the target network. S54. Design a priority experience replay strategy to train and update the parameters of the prediction network, and then softly update the parameters of the target network. The priority experience replay strategy receives the robot's current state, the current sub-task target point position, the immediate reward, the TD error, and experience samples from the experience replay pool as input. It calculates the experience sampling probability, samples the experience samples from the experience replay pool based on the experience sampling probability, and inputs the sampled experience samples into the prediction network to update the prediction network parameters. The experience samples include the robot's current state, the executed action, the immediate reward, the robot's new state, the network training termination flag, and the dynamically weighted priority generated by the experience sampling probability. The empirical sampling probability is calculated as follows: ； in Indicates the first The empirical sampling probability of each empirical sample; , These represent the weighting parameters for immediate reward and TD error, respectively; , They represent the first Instant reward and TD error for each experience sample; This represents the index of the experience samples in the experience replay pool; This represents the index of the experience sample in the experience replay pool used for traversal calculation and summation. This represents the total number of experience samples in the experience replay pool; This indicates the operation of taking the minimum value; This indicates the logarithmic operation; This indicates the modulus length operation, used in the above formula to take the absolute value; S55. Determine whether the robot has reached the target point of the current subtask; S56. If the robot has not reached the target point of the current subtask, update the robot state and re-execute steps S53-S55; if the robot has reached the target point of the current subtask, the current subtask ends, and the local planning path of the current subtask is generated using the prediction network parameters at this time, and the local path planning of the next subtask is performed.

7. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 6, characterized in that: The state space includes target features, environmental perception features, and temporal features; The target features include the Euclidean distance and azimuth angle between the robot's position and the current subtask target point position; The environmental perception features include the distribution of static and dynamic obstacles within the robot's perception range and the predicted landing point (PLP) of dynamic obstacles; the predicted landing point (PLP) of dynamic obstacles is generated by a pre-trained recurrent neural network (RNN) based on the historical displacement data of dynamic obstacles. The temporal features include a sequence of several environmental snapshots within the robot's perception range of obstacles, stacked in time steps. The action space, using an 8-neighborhood discrete movement method, is defined as follows: ； in Represents the action space. , These represent the robot's lateral and longitudinal displacements, respectively. The action space, through -The "greedy" strategy selects the action; The multidimensional reward function includes distance reward. Orientation alignment reward Obstacle avoidance rewards Predicting the landing point is close to the penalty Step count penalty ; The distance reward The definition is as follows: ;in express The Euclidean distance between the robot at any given moment and the target point of the current subtask. This represents the positive reward coefficient when the robot approaches the target point of the current subtask, and its value range is [range missing]. , This represents the negative penalty coefficient when the robot moves away from the current subtask target point, with a value range of... ; The distance reward In each subtask, the Euclidean distance between the robot and the target point of the current subtask is... When the value is first less than the preset position tolerance threshold, an additional stage completion reward will be granted. Value range ; The direction alignment reward The definition is as follows: ； in, , These are the horizontal and vertical coordinates of the robot's current position; , The x and y coordinates of the current subtask target point location; For reward scaling factor; The obstacle avoidance reward The definition is as follows: ； in, This represents the Euclidean distance between the robot's current position and the nearest obstacle. This represents the penalty coefficient during a collision. This represents the penalty coefficient for near-field obstacle avoidance. This represents the mid-range obstacle avoidance bonus coefficient. This represents the long-distance obstacle avoidance bonus coefficient. This represents the long-distance obstacle avoidance bonus coefficient; , , , , The range of values ​​is ; The predicted landing point is close to the penalty The definition is as follows: ； in, This represents the relative distance between the robot and the predicted landing point PLP of the dynamic obstacle; This indicates that the predicted landing point is close to the penalty coefficient, and the range of values ​​is... ; This indicates that the preset predicted landing point is close to the judgment threshold; The step penalty Give a small negative reward at every moment value range ; The multidimensional reward function is defined as follows: ； in, This represents a multidimensional reward function.

8. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 7, characterized in that: The prediction network includes a CNN+MLP module, an LSTM layer, and a dense layer; where CNN stands for Convolutional Neural Network, MLP stands for Multilayer Perceptron, and LSTM stands for Long Short-Term Memory Network; the dense layer uses a DuelingDQN structure to output action decisions. The expected cumulative reward of the prediction network output satisfies: ； in, Indicates based on robot state With action Expected cumulative reward; For robot state-based State value; This indicates the mold length taking operation, in the above formula Representing the action space The total number of actions in the process; Representing the action space Any candidate action in the list.

9. The robot dynamic obstacle avoidance planning method based on path key nodes and deep reinforcement learning according to claim 8, characterized in that: The training and updating of the parameters of the prediction network and the target network includes: Update the expected cumulative reward based on the multi-step optimal Bellman equation: ； in, Indicates about robot state at any moment and Moment of action Expected cumulative reward; symbol This indicates an update to; This represents the predicted network learning rate, which is predefined manually and has a range of values. ; Indicates the summation step index for multi-step learning; This represents the total number of steps in a multi-step learning process. Indicates the first The discount factor for the step, with a value range of 100%. ; express Instant rewards for each moment; express robot state at any moment Next, take candidate actions Expected cumulative reward; Indicates the robot's state Next, traverse all candidate actions. The maximum expected cumulative reward obtained; Therefore, the strategy is determined as follows: ； in, Indicates based on robot state With action Strategies; Indicates taking The largest value , The round index of strategy iteration.

10. A robot navigation system for applying the robot dynamic obstacle avoidance planning method based on path critical nodes and deep reinforcement learning as described in any one of claims 1-9, characterized in that, include: The map preprocessing module is used to perform binarization and rasterization preprocessing on the actual scene map, extract the coordinates of representative obstacles, and generate a raster map. The global path planning module is used to construct and optimize the Voronoi diagram and its adjacency matrix based on the raster map, extract the core nodes of the path, and construct the set of core nodes of the path. The local path planning module is used to divide the global path planning task into several sub-tasks based on the core node set of the path, and to perform local path planning in each sub-task based on the reinforcement learning Q network, generating the local planning path of each sub-task and forming the local planning path instruction output. The environmental perception module is used to collect information on static and dynamic obstacles within the robot's preset perception range, and predicts the trajectory landing point PLP of dynamic obstacles through the recurrent neural network (RNN) module. The motion execution module is used to receive the local planning path instructions output by the local path planning module and control the robot to move according to the global planning path formed by the local planning path. The model training module is used to set training parameters and update the parameters of the prediction network and the target network.

Images