Robotic autonomous exploration method based on deep reinforcement learning
By using a deep reinforcement learning approach, combined with Voronoi diagrams and convolutional neural networks, an autonomous exploration model for robots is constructed. This solves the problem of low efficiency in complex environments in existing technologies, and enables efficient and reliable autonomous exploration and map building.
Patent Information
- Application Number
- CN202211706791.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2042-12-29
AI Technical Summary
Existing methods for autonomous robot exploration are inefficient in complex environments and require a large amount of prior data, making them difficult to adapt to the complex environments of the real world.
A deep reinforcement learning-based approach, combining Voronoi diagrams and convolutional neural networks, is used to construct an autonomous exploration model for the robot. An initial environment map is built using real-time environmental data, and the map is incrementally constructed. Boundaries and information gain are detected, and an Actor-Critic structure is used to learn exploration strategies. An ineffective action masking mechanism is employed to optimize the exploration process.
It enables efficient, reliable, and accurate autonomous exploration in complex environments, reducing learning costs and improving exploration efficiency and map quality.
Smart Images
Figure CN115963832B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a robot autonomous exploration method based on deep reinforcement learning. BACKGROUND
[0002] With the development of economic technology and the improvement of living standards, artificial intelligence technology has been widely applied in people's production and life, bringing endless convenience to people's production and life.
[0003] In recent years, robot autonomous exploration technology has received extensive attention in academic research and has been applied in many fields, such as search and rescue tasks, intelligence, surveillance and reconnaissance tasks, and planetary exploration tasks. When robots navigate and work in complex and unknown environments, they often need to have the ability of autonomous environmental exploration. Robots with autonomous exploration capabilities can be deployed flexibly because they can use previously observed environmental knowledge to quickly collect useful information in new environments without relying on humans. This ability enables such robots to prepare for downstream tasks to be performed in new environments, such as reconstruction, navigation, and interaction. With the development of economic technology, autonomous exploration capability has become an essential skill for intelligent robots.
[0004] The application of intelligent robots in complex environments depends on robust and efficient exploration strategies. In the past few decades, researchers have proposed different autonomous exploration methods, such as the famous boundary-based exploration method and artificial potential field method. However, in complex real-world environments, it is difficult to define appropriate and flexible exploration strategies artificially; the reason is that the artificially designed exploration method usually makes too idealized assumptions about the environment and the task, which may limit the adaptability of the robot to complex environments, thereby limiting the application of the robot in the real world. In order to solve the above problems, researchers have proposed a deep reinforcement learning (DRL) method, the core idea of which is to let the agent (such as a robot) learn skills in the trial-and-error process of interacting with the environment, rather than receiving explicit instructions. The deep reinforcement learning method avoids the huge dataset labeling requirement, thereby greatly reducing the learning cost.
[0005] The classical robot autonomous exploration methods mainly include a boundary-based exploration method and a sampling-based exploration method: the boundary-based exploration method firstly detects the boundary in the map and extracts frontier points (FPs) as candidate exploration targets; the sampling-based exploration method does not need to traverse all parts of the map, but randomly samples in the space, then constructs an abstract graph or tree structure, and connects the sampling points to form an exploration path. However, both of the two exploration methods have high exploration time and exploration path cost, and need a large amount of prior data, and the efficiency of the algorithm is low. SUMMARY
[0006] The robot autonomous exploration method based on deep reinforcement learning provided by the present application has high reliability, good accuracy and high efficiency.
[0007] The robot autonomous exploration method based on deep reinforcement learning provided by the present application has high reliability, good accuracy and high efficiency.
[0008] S1. The robot acquires data information of the surrounding environment in real time;
[0009] S2. According to the data information obtained in step S1, an initial environment map of the working space of the robot is constructed based on the Voronoi diagram idea;
[0010] S3. According to the data information obtained in step S1, the map is incrementally constructed, and the boundary and information gain of the map are detected;
[0011] S4. According to the data information obtained in steps S2 and S3, a final environment topological map is obtained;
[0012] S5. Based on a convolutional neural network, a robot autonomous exploration learning model is constructed;
[0013] S6. The robot uses the autonomous exploration learning model constructed in step S5 to explore the surrounding environment and calculates the execution strategy of the next step;
[0014] S7. Repeat the above steps to complete the robot autonomous exploration based on deep reinforcement learning.
[0015] According to the data information obtained in step S1, the initial environment map of the working space of the robot is constructed based on the Voronoi diagram idea in step S2, which specifically includes the following steps:
[0016] The robot works in a working space W; the working space W is an m-dimensional space R m , C2,..., C mFilling, non-convex obstacles in the workspace are modeled by convex obstacles; the boundary of the workspace W is the set of convex sets belonging to the obstacle set;
[0017] The distance d between a point x and an obstacle C is represented by the following equation i i (x) :
[0018]
[0019]
[0020] where c0 is the closest point of the obstacle C to the point x; x-c0 is the Euclidean distance between the point x and the point c0; i is the unit vector in the direction from the point x to c0; x-c0 represents the direction vector between the point x and the point c0;
[0021] There exists an equidistant point x1 in the obstacle, and the shortest distance D(x) between the equidistant point and the obstacle is represented by the following equation The definition of the equidistant point is that there exists a point x between the obstacle i and the obstacle j, and the distance between the point x and the obstacle i and the obstacle j is the same, then the point x is called the equidistant point between the obstacle i and the obstacle j;
[0022] The set between a group of collinear equidistant points is defined as a generalized Voronoi edge, and is represented by the following equation:
[0023]
[0024] where F ij is the generalized Voronoi edge between the obstacle i and the obstacle j, and more intuitively, F ij represents the angle bisector between the unit vector of the point x to the obstacle i and the obstacle j; and ;
[0025] The intersection point of the generalized Voronoi edge is defined as a generalized Voronoi vertex, and is represented by the following equation:
[0026]
[0027] where F ijk is the generalized Voronoi vertex; more intuitively, F ijk is the equidistant point between the obstacles with the same distance to the obstacles i, j and k;
[0028] A group of generalized Voronoi edges and vertices are used to construct a generalized Voronoi diagram as the initial environment map of the workspace of the robot.
[0029] According to the data information obtained in step S1, the map is incrementally constructed, and the boundary and information gain of the map are detected, specifically including the following steps:
[0030] The incrementally constructed map The boundary is detected as a candidate target;
[0031] The information gain at the boundary measures the amount of remaining free space outside the boundary, and the remaining free space is also a navigable unit that has not been explored in the current partial map;
[0032] The undetected free space unit is grouped into connected components Ω = {ω1,..., ω n} using the CCL method in OpenCV, and each connected component ω i is associated with the map boundary; only when at least one pixel in the connected component ω i is an 8-connected neighbor of a pixel in the boundary, the connected component ω i is associated with the boundary;
[0033] For each boundary, the corresponding information gain is calculated as the sum of the areas of the connected components associated with the boundary, and is normalized by the total free space on the complete map.
[0034] According to the data information obtained in steps S2 and S3, the final environment topology map is obtained, specifically including the following steps:
[0035] According to the shortest distance principle, the boundary, the position of the robot, and the generalized Voronoi diagram are connected to obtain the final environment topology map; the connection between all nodes is conflict-free, and the domain conversion of the semantic map is completed
[0036] According to the data information obtained in steps S2 and S3, the final environment topology map is obtained, specifically including the following steps:
[0037] Based on the graph convolutional neural network and the residual connection, a spatial feature encoder is constructed;
[0038] Based on the time series information and the temporal convolutional neural network, a time feature encoder is constructed;
[0039] The outputs of the spatial feature encoder and the time feature encoder are fused using an attention mechanism, and the fusion result is learned through an Actor-Critic structure;
[0040] An invalid action masking mechanism is used to limit the actions of the robot.
[0041] The space feature encoder is constructed based on the graph convolutional neural network and the residual connection, and specifically includes the following steps:
[0042] The space feature encoder is used for encoding the spatial structure information of the topological graph.
[0043] The node feature of the first l layer is represented as Then, the node aggregation calculation of the first l layer is performed and taken as the input G of the first l+1 layer (l+1) :
[0044]
[0045] In the formula, σ S () is a ReLU function; D is a diagonal matrix; is an intermediate matrix and A is an adjacency matrix of the exploration graph, and I is a unit matrix; W s (l) is a weight matrix to be learned of the first l layer.
[0046] The time feature encoder is constructed based on the time sequence information and the time convolutional neural network, and specifically includes the following steps:
[0047] The history context storage is realized by establishing a buffer area.
[0048] The robot reads and takes out the map spatial information obtained by the space feature encoder in each time period tau to the cache;
[0049] The buffer area M in the memory stores the high-dimensional spatial features and the frontier features In the implementation, the time period tau is realized in the form of a sampling sliding window.
[0050] At time tau, the input of the time convolutional neural network is composed of the history spatial features stored in the buffer area M, represented as M={M0, M1,..., M t-1}; M t represents the segment of the tth memory in the buffer area M.
[0051] The tth buffer memory segment M t (l) in the first l layer of the time convolutional neural network is updated by using the following formula:
[0052]
[0053] In the formula, K is the size of the one-dimensional convolution kernel; η1~η k are to-be-learned parameters of the convolution kernel; d t is an expansion factor for controlling the t-hop connection of the layer.
[0054] The output of the spatial feature encoder and the output of the temporal feature encoder are fused by using an attention mechanism, and the fusion result is learned by an Actor-Critic structure, and specifically includes the following steps:
[0055] The output r of the spatial feature encoder is fused by using an attention mechanism t G And the output of the temporal feature encoder Retrieve information related to the current map state from the long-term memory;
[0056] The output of the attention mechanism Is expressed as:
[0057]
[0058] In the formula, α1 is a spatial weight and α2 is a time weight and W and b are parameters of the attention mechanism; exp() is an acceleration function;
[0059] The current spatial feature and the corresponding query result z on the exploratory memory are integrated by using an aggregator t , expressed as:
[0060]
[0061] In the formula, FFL() is a feedforward layer processing function; cat() is a matrix splicing operation;
[0062] The robot exploration strategy is modeled as a diagonal Gaussian distribution, wherein the exploration strategy and the state value function are learned by an Actor-Critic structure: the mean μ and the logarithmic standard deviation logσ of the distribution are learned by an Actor-head structure; a Critic-head structure is used to estimate the state value function of the current strategy Wherein is the expected reward, T exp is the maximum time step of the robot interacting with the environment, γ t is the discount factor at time step t, r t is the environment reward at time t;
[0063] Both the Actor-head structure and the Critic-head structure are implemented by a multi-layer perceptron.
[0064] The invalid action shielding mechanism is used to limit the action of the robot, and specifically includes the following steps:
[0065] The following policy gradient estimate is proposed for the target J by using a gradient algorithm:
[0066]
[0067] In the formula g policy E represents the gradient of the reward function in reinforcement learning based on policy optimization. τ () is the function for calculating the expected value; For differential operators; T is the maximum time step for interaction with the environment; π θ () represents the reinforcement learning policy function; a t Let s be the action performed by the robot at time t; t Let π be the state of the robot at time t; θ (a t |s t ) represents the robot in state s t Based on the exploration strategy π θ Execute action a t The probability of R; t The discounted return after a time step τ, and T exp This represents the maximum number of steps the robot can take to interact with its environment.
[0068] Nodes far from the boundary are considered invalid operations with no information gain; then invalid action masking is used to process the non-standardized scores corresponding to invalid actions, thereby avoiding sampling invalid actions.
[0069] The invalid action masking process is considered as a differentiable function inv s Applied to strategy π θ Given a state s∈S, the output non-standardized score l(s) is determined by the policy π. θ (·|s t By differentiable functions inv s The softmax calculation, with invalid operation masking represented by the following formula:
[0070] π θ (·|s t = softmax(inv) s (l(s)))
[0071]
[0072] In the formula l i =logπ θ (a i |s) is based on strategy π θ Execute action a in state s. i The probability; N is a set constant; inv s (l(s)) represents the state s, where action a i If it is valid, then it is action a. iExecution probability of action a i Invalid, taking a value of a very large negative number set, through the softmax function to reduce the execution probability of action a i 0.
[0073] The robot described in step S6 explores the surrounding environment using the autonomous exploration learning model constructed in step S5, and calculates the execution strategy for the next step, which includes the following steps:
[0074] Path planning and map construction are realized based on move_base and Gmapping;
[0075] In order to improve the exploration efficiency and balance the map quality, the following reward function R is set for the robot to encourage the robot to exchange for greater exploration coverage with less path and time consumption:
[0076]
[0077] Where β1-β4 are weighting factors; is the total information gain before time step t; is the total path length before time step t; N T is the total time step before time step t; is a parameter to encourage or punish the influence of the currently selected action on the accuracy of the map, and Where is the predicted environment map at time step t, M is the real environment map, and ω is the width of the spatial feature matrix M, and h is the height of the spatial feature matrix M, is an indicator function, if then then
[0078] The autonomous exploration method of the robot based on deep reinforcement learning provided by the present application proposes a workspace fine-grained topology representation method based on generalized Voronoi diagram to realize domain conversion, breaks the curse of dimensionality of action space and observation space, and does not lose the structural properties of the workspace; a neural network framework with spatial perception and scenario memory is proposed as an autonomous exploration learning model, and a model-free reinforcement learning technique is used to learn the exploration strategy in the process of trial and error interaction with the physical or simulation system without human intervention; a method of masking and reshaping the autonomous exploration action space and state space using invalid actions is proposed to ensure efficient and complete exploration; therefore, the method of the present application not only enables autonomous exploration of the robot, but also has high reliability, good accuracy and high efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0079] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0080] like Figure 1 The diagram shown illustrates the method flow of this invention: This method for autonomous robot exploration based on deep reinforcement learning, provided by this invention, includes the following steps:
[0081] S1. The robot acquires real-time data about its surrounding environment;
[0082] S2. Based on the data obtained in step S1, construct the initial environment map of the robot's workspace using the Voronoi diagram concept; specifically, this includes the following steps:
[0083] The robot is configured to work in a workspace W; the workspace W is an m-dimensional space R. m , consisting of convex obstacles C1, C2, ..., C m Filling: Non-convex obstacles in the workspace are modeled as convex obstacles; the boundary of the workspace W is a set of convex sets belonging to the obstacle set;
[0084] The following formula represents the distance from point x to obstacle C. i The distance d between i (x):
[0085]
[0086]
[0087] In the formula, c0 represents the obstacle C. i The point closest to the median point x; x-c0 is the Euclidean distance between point x and point c0. x is the unit vector from point x to c0; x-c0 represents the direction vector between point x and point c0.
[0088] There exists an equidistant point x1 within the obstacle, and the shortest distance D(x) between the equidistant point and the obstacle is denoted as... The definition of an equidistant point is: if there is a point x between obstacle i and obstacle j, and the distance between point x and obstacle i and obstacle j is the same, then point x is called an equidistant point between obstacle i and obstacle j.
[0089] The set of collinear equidistant points is defined as a generalized Voronoi edge, and is expressed by the following formula:
[0090]
[0091] In the formula F ijF is the generalized Voronoi edge between obstacle i and obstacle j, or intuitively F ij is the bisector of the angle between the unit vectors from point x to obstacle i and obstacle j and ;
[0092] The intersection of generalized Voronoi edges is defined as a generalized Voronoi vertex, and is denoted as F
[0093]
[0094] where F ijk is the generalized Voronoi vertex; intuitively F ijk is the equidistant point between obstacles i, j and k with equal distance to them;
[0095] The generalized Voronoi diagram is constructed from a set of generalized Voronoi edges and vertices, and serves as the initial environment map of the robot's workspace;
[0096] S3. Based on the data information obtained in step S1, incrementally construct the map and detect the boundary of the map and the information gain; specifically including the following steps:
[0097] For the incrementally constructed map detect the boundary as a candidate target;
[0098] The information gain at the boundary measures the amount of remaining free space outside the boundary, which is also the amount of unexplored navigable cells in the current partial map;
[0099] The unexplored free space cells are grouped into connected components Ω = {ω1,..., ω n} using the CCL method in OpenCV, and then each connected component ω i is associated with the map boundary; only when at least one pixel in the connected component ω i is an 8-connected neighbor of some pixel in the boundary, the connected component ω i is associated with the boundary;
[0100] For each boundary, the corresponding information gain is calculated as the sum of the areas of the connected components associated with the boundary, and is normalized by the total free space on the complete map;
[0101] S4. Based on the data information obtained in step S2 and step S3, obtain the final environment topology map; specifically including the following steps:
[0102] According to the shortest distance principle, the boundary, the position of the robot and the generalized Voronoi diagram are connected to obtain the final environment topology map; the connection between all nodes is collision-free, and the domain conversion of the semantic map is completed
[0103] S5. Based on the convolutional neural network, a robot autonomous detection learning model is constructed; specifically including the following steps:
[0104] Based on the graph convolutional neural network and the residual connection, a spatial feature encoder is constructed; specifically including the following steps:
[0105] The spatial feature encoder is used to encode the spatial structure information of the topology graph;
[0106] The node feature of the first l layer is represented as Then the node aggregation calculation of the lth layer is calculated and taken as the input G of the l+1th layer (l+1) :
[0107]
[0108] In the formula, σ S () is the ReLU function; D is a diagonal matrix; is an intermediate matrix and A is the adjacency matrix of the exploration graph, and I is the unit matrix; W s (l) is the weight matrix to be learned of the lth layer;
[0109] Based on the time sequence information and the time convolutional neural network, a time feature encoder is constructed; specifically including the following steps:
[0110] The history context storage is realized by establishing a buffer area;
[0111] The robot reads and takes out the map spatial information obtained by the spatial feature encoder in each time period τ to the cache;
[0112] The buffer area M in the memory stores the high-dimensional spatial features and the front feature In specific implementation, the time period τ is realized in the form of a sampling sliding window;
[0113] At time τ, the input of the time convolutional neural network is composed of the historical spatial features stored in the buffer area M, represented as M={M0,M1,...,M t-1}; M t represents the tth memory segment in the buffer area M;
[0114] The tth buffer memory segment M in the lth layer of the time convolutional neural network is updated by using the following formulat (l):
[0115]
[0116] In the formula, K is the size of the one-dimensional convolution kernel; η1~η k The parameters to be learned for the convolution kernel; d t The expansion factor for the jump connection of the control layer t;
[0117] The outputs of the spatial feature encoder and the temporal feature encoder are fused using an attention mechanism, and the fusion result is then learned through an Actor-Critic structure; specifically, the following steps are included:
[0118] The output r of the spatial feature encoder is fused using an attention mechanism. t G and the output of the time feature encoder Retrieve information related to the current map state from long-term memory;
[0119] Output of attention mechanism Represented as:
[0120]
[0121] In the formula, α1 is the spatial weight and α2 is the time weight and W and b are parameters of the attention mechanism; exp() is the acceleration function;
[0122] An aggregator is used to integrate current spatial features and corresponding query results from exploratory in-memory searches. t , is represented as:
[0123]
[0124] In the formula, FFL() is the feedforward layer processing function; cat() is the matrix concatenation operation;
[0125] The robot's exploration strategy is modeled as a diagonal Gaussian distribution, where the exploration strategy and state-value function are learned by an Actor-Critic structure: the mean μ and log-standard deviation logσ of the distribution are learned by an Actor-head structure; the Critic-head structure is used to estimate the state-value function of the current strategy. in For expected reward, T exp γ represents the maximum time step for the robot to interact with its environment. t r is the discount factor at time step t. t The environmental reward at time t;
[0126] The actor-head structure and the critic-head structure are both implemented by a multi-layer perceptron;
[0127] An invalid action masking mechanism is adopted to limit the action of the robot, and specifically includes the following steps:
[0128] The following policy gradient estimation is proposed for the target J using a gradient algorithm:
[0129]
[0130] where g policy is the gradient of the reinforcement learning reward function based on policy optimization; E τ () is an expected value calculation function; is a differential operator; T is the maximum number of time steps for interaction with the environment; π θ () is a reinforcement learning policy function; a t is the action performed by the robot at time t; s t is the state of the robot at time t; π θ (a t |s t ) is the probability that the robot performs action a t according to the exploration policy π θ at state s t ; R t is the discounted return after time step τ, and T exp is the maximum number of steps for the robot to interact with the environment;
[0131] Nodes far from the boundary are considered as invalid operations without information gain, and then the non-standardized scores corresponding to the invalid actions are processed using invalid action masking to avoid sampling invalid actions;
[0132] The invalid action masking process is regarded as a differentiable function inv s applied to the policy π θ ; according to a given state s∈S, the output non-standardized score value l(s), the policy π θ (·|s t ) is calculated by performing softmax on the differentiable function inv s , and the invalid operation masking is represented by the following formula:
[0133] π θ (·|s t )=softmax(inv s (l(s)))
[0134] where l i =logπ θ(a i is the probability of performing action a θ in state s according to policy π i ; N is a set constant; inv s (l(s)) is the execution probability of action a i if action a i is valid for state s, and is a set large negative number if action a i is invalid, by which the execution probability of action a i is reduced to 0 by the softmax function;
[0135] S6. The robot explores the surrounding environment using the autonomous exploration learning model constructed in step S5, and calculates the execution strategy of the next step; specifically comprising the following steps:
[0136] Path planning and map construction are realized based on move_base and Gmapping;
[0137] In order to improve the exploration efficiency and balance the map quality, the following reward function R is set for the robot, which is used to encourage the robot to exchange for greater exploration coverage with less path and time consumption:
[0138]
[0139] where β1-β4 are weighting factors; is the total information gain before time step t; is the total path length before time step t; N T is the total time step before time step t; is a parameter for encouraging or punishing the influence of the currently selected action on the map accuracy, and where is the predicted environment map at time step t, M is the real environment map, and ω is the width of the spatial feature matrix M, and h is the height of the spatial feature matrix M, is an indicator function, and if then then
[0140] S7. Repeat the above steps to complete the autonomous exploration of the robot based on deep reinforcement learning.
[0141] The robot autonomous exploration method based on deep reinforcement learning can be used for environment exploration and reconstruction in special environments, such as environment exploration and rescue in dangerous houses after fire and earthquake; and in daily life, the robot autonomous exploration algorithm based on deep learning can be deployed on a household robot (for example, a sweeping robot) to help the robot clean the room efficiently.
[0142] As a fast cleaning method applied to a sweeping robot, the method comprises the following steps:
[0143] placing the sweeping robot in a target room;
[0144] The robot autonomous exploration method based on deep reinforcement learning is used to explore the target room and acquire an environment space feature map;
[0145] The robot plans a path, avoids obstacles and quickly sweeps garbage in the room by using the acquired environment map.
Claims
1. A robot autonomous exploration method based on deep reinforcement learning, comprising the following steps: S1. The robot acquires real-time data about its surrounding environment; S2. Based on the data obtained in step S1, construct the initial environment map of the robot's workspace using the Voronoi diagram concept; S3. Based on the data obtained in step S1, incrementally construct a map and detect the map's boundaries and information gain; S4. Based on the data obtained in steps S2 and S3, obtain the final environmental topology map; S5. Construct a robot autonomous exploration and learning model based on convolutional neural networks; S6. The robot uses the autonomous exploration learning model constructed in step S5 to explore the surrounding environment and calculates the next execution strategy; specifically, it includes the following steps: Implement path planning and map building based on move_base and Gmapping; To improve exploration efficiency and balance map quality, the following reward function R is set for the robot to encourage it to achieve greater exploration coverage with less path and time consumption: In the formula, β1 to β4 are weighting factors; The total information gain before time step t; N represents the total path length before time step t; T This represents the total time step before time step t; A parameter that incentivizes or penalizes the impact of the currently selected action on map accuracy, and in M is the predicted environment map at time step t, and M is the actual environment map. ω is the width of the spatial characteristic matrix M, and h is the height of the spatial characteristic matrix M. For indicator functions, if but but S7. Repeat the above steps to complete the robot's autonomous exploration based on deep reinforcement learning.
2. The robot autonomous exploration method based on deep reinforcement learning according to claim 1, characterized in that... Step S2, which involves constructing an initial environment map of the robot's workspace based on the data information obtained in step S1 and the Voronoi diagram concept, specifically includes the following steps: The robot is configured to work in a workspace W; the workspace W is an m-dimensional space R. m , consisting of convex obstacles C1, C2, ..., C m Filling: Non-convex obstacles in the workspace are modeled as convex obstacles; the boundary of the workspace W is a set of convex sets belonging to the obstacle set; The following formula represents the distance from point x to obstacle C. i The distance d between i (x): In the formula, c0 represents the obstacle C. i The point closest to the median point x; ||x-c0|| is the Euclidean distance between point x and point c0. x is the unit vector from point x to c0; x-c0 represents the direction vector between point x and point c0. There exists an equidistant point x1 within the obstacle, and the shortest distance D(x) between the equidistant point and the obstacle is denoted as... The definition of an equidistant point is: if there is a point x between obstacle i and obstacle j, and the distance between point x and obstacle i and obstacle j is the same, then point x is called an equidistant point between obstacle i and obstacle j. The set of collinear equidistant points is defined as a generalized Voronoi edge, and is expressed by the following formula: In the formula F ij Let be the generalized Voronoi edge between obstacle i and obstacle j; The intersection of generalized Voronoi edges is defined as a generalized Voronoi vertex, and is expressed by the following formula: In the formula F ijk Vertices of the generalized Voronoi; A generalized Voronoi graph, consisting of a set of generalized Voronoi edges and vertices, serves as the initial environment graph for the robot's workspace.
3. The robot autonomous exploration method based on deep reinforcement learning according to claim 2, characterized in that... Step S3, which involves incrementally constructing a map based on the data obtained in step S1 and detecting the map's boundaries and information gain, specifically includes the following steps: Maps constructed incrementally Detect the boundary as a candidate target; Information gain at the boundary measures the amount of remaining free space outside the boundary, which is also the unexplored navigable unit in the current part of the map; The undetected free space units are grouped into connected components Ω = {ω1, ..., ω2} using the CCL method in OpenCV. n }, and then for each connected component ω i Associated with map boundaries; only when the connected component ω i When at least one pixel in a given component is an 8-connected neighbor of some pixel in the boundary, the connected component ω i Only then is it associated with the boundary; For each boundary, the corresponding information gain is calculated as the sum of the areas of the connected components associated with the boundary, and normalized by the total free space on the complete mapping.
4. The deep reinforcement learning-based autonomous exploration method for robots according to claim 3, characterized in that... Step S4, which involves obtaining the final environmental topology map based on the data obtained in steps S2 and S3, specifically includes the following steps: Based on the shortest distance principle, the boundaries, robot positions, and generalized Voronoi diagrams are connected to obtain the final environmental topology map; all connections between nodes are conflict-free, completing the domain transformation of the semantic map.
5. The deep reinforcement learning-based autonomous exploration method for robots according to claim 4, characterized in that... Step S5, which describes building a robot autonomous exploration and learning model based on a convolutional neural network, specifically includes the following steps: A spatial feature encoder is constructed based on graph convolutional neural networks and residual connections; A temporal feature encoder is constructed based on temporal information and a temporal convolutional neural network. The outputs of the spatial feature encoder and the temporal feature encoder are fused using an attention mechanism, and the fusion result is learned through an Actor-Critic structure. An invalid action shielding mechanism is used to restrict the robot's actions.
6. The robot autonomous exploration method based on deep reinforcement learning according to claim 5, characterized in that... The construction of a spatial feature encoder based on graph convolutional neural networks and residual connections specifically includes the following steps: Spatial feature encoders are used to encode the spatial structure information of topological maps; The node features of the first l layers are represented as follows: Then, the nodes of layer l are aggregated and computed, and used as the input G of layer l+1. (l+1) : In the formula σ S () represents the ReLU function; D is a diagonal matrix; It is an intermediate matrix and A is the adjacency matrix of the exploration graph, I is the identity matrix; W s (l) Let be the weight matrix to be learned in the l-th layer.
7. The robot autonomous exploration method based on deep reinforcement learning according to claim 6, characterized in that... The construction of a temporal feature encoder based on temporal information and a temporal convolutional neural network specifically includes the following steps: Historical context storage is achieved by establishing a buffer. Within each time period τ, the robot reads and retrieves the map spatial information obtained by the spatial feature encoder from the cache; The buffer M in the memory stores paired high-dimensional spatial features from the previous time step τ. and cutting-edge features In practice, it is implemented using a time period τ sampling sliding window method; At time τ, the input to the temporal convolutional neural network consists of historical spatial features stored in a buffer M, denoted as M = {M0, M1, ..., M}. t-1 };M t This represents the t-th memory segment in buffer M; The following formula is used to update the t-th buffer memory segment M in the l-th layer of the temporal convolutional neural network. t (l): In the formula, K is the size of the one-dimensional convolution kernel; η1~η k The parameters to be learned for the convolution kernel; d t This is the expansion factor for the jump connection of the control layer t.
8. The robot autonomous exploration method based on deep reinforcement learning according to claim 7, characterized in that... The process of fusing the outputs of the spatial feature encoder and the temporal feature encoder using an attention mechanism, and then learning the fusion result through an Actor-Critic structure, specifically includes the following steps: The output r of the spatial feature encoder is fused using an attention mechanism. t G and the output of the time feature encoder Retrieve information related to the current map state from long-term memory; Output of attention mechanism Represented as: In the formula, α1 is the spatial weight and α2 is the time weight and W and b are parameters of the attention mechanism; exp() is the acceleration function; An aggregator is used to integrate current spatial features and corresponding query results from exploratory in-memory searches. t , is represented as: In the formula, FFL() is the feedforward layer processing function; cat() is the matrix concatenation operation; The robot's exploration strategy is modeled as a diagonal Gaussian distribution, where the exploration strategy and state-value function are learned by an Actor-Critic structure: the mean μ and log-standard deviation logσ of the distribution are learned by an Actor-head structure; the Critic-head structure is used to estimate the state-value function of the current strategy. in For expected reward, T exp γ represents the maximum time step for the robot to interact with its environment. t r is the discount factor at time step t. t The environmental reward at time t; Both the Actor-head and Critic-head structures are implemented using multilayer perceptrons.
9. The robot autonomous exploration method based on deep reinforcement learning according to claim 8, characterized in that... The aforementioned invalid action shielding mechanism, which restricts the robot's actions, specifically includes the following steps: The following policy gradient estimation is proposed for target J using a gradient algorithm: In the formula g policy E represents the gradient of the reward function in reinforcement learning based on policy optimization. τ () is the function for calculating the expected value; For differential operators; T is the maximum time step for interaction with the environment; π θ () represents the reinforcement learning policy function; a t Let s be the action performed by the robot at time t; t Let π be the state of the robot at time t; θ (a t |s t ) represents the robot in state s t Based on the exploration strategy π θ Execute action a t The probability of R; t This represents the discounted return after a time step τ, and T exp This represents the maximum number of steps the robot can take to interact with its environment. Nodes far from the boundary are considered invalid operations with no information gain; then invalid action masking is used to process the non-standardized scores corresponding to invalid actions, thereby avoiding sampling invalid actions. The invalid action masking process is considered as a differentiable function inv s Applied to strategy π θ Given a state s∈S, the output non-standardized score l(s) is determined by the policy π. θ (·|s t By differentiable functions inv s The softmax calculation, with invalid operation masking represented by the following formula: π θ (·|s t )=softmax(inv s (l(s))) In the formula l i =logπ θ (a i |s) is based on strategy π θ Execute action a in state s. i The probability; N is a set constant; inv s (l(s)) represents the state s, where action a i If it is valid, then it is action a. i Execution probability, if action a i If invalid, the value is set to a very large negative number, and the action 'a' is processed by the softmax function. i The execution probability drops to 0.