Robot autonomous exploration method based on expert demonstration guidance

Abstract

The invention discloses a robot autonomous exploration method based on expert demonstration guidance, and the method comprises the steps: modeling robot exploration into a sequential decision problem under a Markov decision framework, discretizing an action space into motion elements, representing a to-be-explored environment as a grid map, and carrying out the modeling of the grid map; constructing a state space through historical behavior information and maps of different scales; solving a shortest path planning problem in an exploration task with a complete environment structure to obtain expert demonstration; using an objective function of near-end strategy optimization, taking expert demonstration as actions adopted by a robot in a real environment, and performing strategy optimization by utilizing expert demonstration operation and intelligent agent experience; and exploring the robot in an unknown environment by using the updated strategy. According to the method, privileged structure information of the environment is utilized, expert demonstration and experience of the robot are combined, cost is reduced, efficiency is improved, and the convergence speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and in particular to a robot autonomous exploration method based on expert demonstration guidance. Background Art

[0002] Autonomous robot exploration is a fundamental yet challenging task in robotics applications. Given only partial knowledge of the environment, robots must plan paths in real time to efficiently map all unknown areas while minimizing the exploration path. Over the past few decades, a variety of classic autonomous robot exploration methods have been extensively studied, including frontier-based and sampling-based approaches. Building on this foundation, some newer studies have adopted a hierarchical decision framework to reduce computational complexity and utilized the Traveling Salesman Problem (TSP) algorithm as a local path planner to compute a near-optimal path within a local area.

[0003] However, these methods usually lack the ability to predict the structure of unseen environments, often leading to globally suboptimal solutions. Due to its emphasis on long-term rewards, deep reinforcement learning (DRL) has become a promising method for exploration tasks. Despite this, most reinforcement learning-based methods are difficult to deploy in large-scale environments, mainly due to problems such as scene diversity, sparse rewards, and low simulation efficiency. To alleviate the training difficulty in reinforcement learning, learning from demonstrations (LfD) was proposed and has received attention in decision-making and planning tasks. However, generating enough high-quality demonstrations in diverse environments remains a major challenge in robotic exploration tasks and a problem that has not yet been fully solved.

[0004] Frontier-based methods build maps by identifying the boundary between known and unknown spaces (i.e., the frontier) and guiding the robot to explore these areas sequentially. Although there have been improvements in frontier utility evaluation and frontier point detection, the inherent greediness of frontier-based methods limits their exploration performance in large-scale scenarios.

[0005] Sampling-based methods use sampling algorithms such as Rapidly-exploring Random Tree (RRT) to generate candidate observation points and evaluate the potential exploration benefits of these points. However, in large scenes, the exploration paths generated by these methods often still have a lot of redundancy.

[0006] In recent years, some methods have combined sampling with frontier point methods to generate candidate observation points. These points are then traversed by solving path planning problems (such as the traveling salesman problem (TSP)), effectively improving exploration efficiency. However, these methods lack the ability to predict unknown scenarios, resulting in short-sighted exploration paths. Furthermore, as scene complexity increases, the computational overhead of observation point detection and TSP solving increases significantly, limiting the scalability of these methods.

[0007] Combining deep reinforcement learning (DRL) with heuristic exploration methods can simplify policy training and facilitate the migration from simulation environments to real-world applications. For example, some studies have used sector-based methods to assist the Next-Best-View Planner (NBVP) in observation point selection, while others have employed neural networks to infer exploration benefits. Although these methods reduce the complexity of policy training to some extent, they still suffer from the same poor real-time performance issues as traditional heuristic methods.

[0008] End-to-end deep reinforcement learning approaches treat the robot's exploration task as a "black box" process, directly taking raw sensor data or an occupancy map as input to predict control actions. While this process is more concise, the "curse of dimensionality" and reward shaping issues limit its scalability in large-scale environments and lead to slow convergence.

[0009] ARIADNE and its subsequent work alleviated the above challenges by constructing a collision-free graph through equidistant sampling in free space and restricting actions to adjacent nodes. Although this strategy reduces the complexity of exploration and achieves significant results in large-scale scenarios, the construction and updating of the graph still imposes additional computational burden.

[0010] In addition to reducing the dimensionality of state and action spaces, learning from demonstrations offers another viable approach for accelerating convergence and improving the performance of intelligent agents in domains with abundant expert demonstrations. This approach has been widely used in fields such as autonomous driving. However, obtaining high-quality expert demonstrations remains challenging for large-scale exploration tasks. Even the most advanced exploration algorithms currently available typically take tens of minutes to complete the exploration of a single scene and require fine-tuning of hyperparameters to optimize the exploration path, making it difficult to collect sufficient demonstrations. Summary of the Invention

[0011] The purpose of this invention is to address the deficiencies of the existing technology and propose a robot autonomous exploration method based on expert demonstration guidance.

[0012] The object of the present invention is achieved through the following technical solutions: a robot autonomous exploration method based on expert demonstration guidance, comprising:

[0013] S1. Robot exploration is modeled as a sequential decision problem in a Markov decision framework, where the action space is discretized into motion primitives, the environment to be explored is represented as a grid map, and the state space is constructed using historical behavior information and maps of different scales.

[0014] S2. Solve the shortest path planning problem in an exploration task with a complete environment structure and obtain expert demonstration;

[0015] S3. Using the objective function of proximal policy optimization, the expert demonstration is used as the action taken by the robot in the real environment, and the strategy is optimized using the expert demonstration operation and the agent's experience;

[0016] S4. Use the updated strategy to enable the robot to explore the unknown environment.

[0017] Furthermore, the robot exploration is modeled as a sequential decision problem under the Markov decision framework, specifically including:

[0018] In the Markov decision process framework, it is defined as a tuple At each time t, the robot observes a state from the explored area Follow its exploration strategy To perform an action To explore new areas, the robot will receive a reward r after the action is executed t =R(s t ,a t ,s t+1 ), and obtain a new observation s t+1 ~T(s|s t ,a t ), the optimal strategy is obtained by solving the problem of obtaining the maximum cumulative discount reward under the premise of the existence of a discount factor.

[0019] Furthermore, the state space is specifically:

[0020] The environment to be explored W is represented as a 2D grid map, and different grayscales represent the free space W f , obstacle W o , and the unknown area W u , the state space is defined as S = {V, I}, and the historical behavior information V = {v0,v1,…,v n} is designed to help robots try new exploration behaviors, where v i =(μ t-i ,a t-i ) contains the robot's historical position and the actions it performs at that position. The multi-level observation I = {I o ,I l ,I g}Contains maps of different scales: I o It is an obstacle avoidance map. l It is a 64×64 local map. g It is a global map scaled to the size of the local map.

[0021] Furthermore, the reward function is specifically:

[0022] Explore Rewards Evaluate the proportion of newly explored areas, step penalty Used to prevent the agent from traveling along invalid paths, collision penalty Used to punish collisions. It means the reward for completing the task.

[0023] Furthermore, the method for solving the shortest path planning problem in an exploration task with a complete environment structure specifically includes: using a two-stage decoupling method to solve the WRP problem in the exploration task. In the first stage, the environment to be explored is first modeled as a polygon with obstacles, and a set of convex polygons, denoted as C, is calculated using the maximum area convex set solving algorithm to cover the polygon. At the same time, minimize the number of convex polygons in the group;

[0024] The second stage: sampling from the boundaries of each convex polygon, and these boundaries do not intersect with obstacles, according to the discretized action space, using the A* algorithm to calculate the shortest collision-free path between the sampling points and their corresponding movement costs, where these sampling points do not belong to the same convex polygon; finally, the polygon is obtained by solving the generalized traveling salesman problem The optimal path Ψ in WRP is applied to the environment to establish the robot's action set.

[0025] Furthermore, the maximum area convex set solving algorithm is specifically as follows:

[0026] Initialization: Initialize the convex polygon set to an empty set and initialize the unknown area to the entire environment

[0027] Calculation steps:

[0028] Calculate the initial calculation point in the unknown area.

[0029] Calculate the polygon visible from the initial calculation point based on the environment and sensor range;

[0030] Calculate the largest convex polygon c0 in the visible polygon and store it in the first convex polygon list

[0031] Perform iterative optimization, select a calculation point in the obtained maximum convex polygon, repeat the visible polygon calculation and the maximum convex polygon according to the calculation point and store them in the first convex polygon list;

[0032] Select the convex polygon c with the largest intersection area with the uncovered area from the first convex polygon list, store it in the convex polygon set, set the convex polygon c as the known area, and update the unknown area;

[0033] Repeat the above calculation steps until the ratio of the area of ​​the unknown region to the total area is less than the set value, and output the convex polygon set.

[0034] Furthermore, in the first stage, for some known environments, the explored areas are masked and the remaining areas are regarded as a set of polygons with holes. The nearest sampling points between these polygons are then used as optimization items and incorporated into the GTSP solution algorithm.

[0035] Furthermore, the S3 is specifically:

[0036] The objective function of PPO is expressed as:

[0037]

[0038] Where ε is a hyperparameter, and π θ They are sampling strategy and training strategy, is an advantage function, Denotes the expert demonstration set, E denotes the agent experience, then the sampling strategy can be expressed as:

[0039]

[0040] in represents sampling from the expert demonstration set, π E represents sampling from the agent’s experience, φ∈(0,1) is a random number, and c is a hyperparameter.

[0041] On the other hand, the specification of the present invention also provides a robot autonomous exploration device based on expert demonstration guidance, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the robot autonomous exploration method based on expert demonstration guidance.

[0042] On the other hand, the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the robot autonomous exploration method based on expert demonstration guidance is implemented.

[0043] Beneficial effects of the present invention:

[0044] This paper exploits the privileged structural information of the environment and models online robot exploration as an offline watchman route problem. By adopting a decoupling strategy, we efficiently compute the globally optimal exploration path at a low computational cost.

[0045] This paper presents an end-to-end exploration framework based on reinforcement learning, EDGE, which leverages expert exploration demonstrations during policy training. EDGE combines expert demonstrations with the robot's own experience for policy updates, accelerating neural network convergence and improving exploration performance (reducing exploration distance by up to 19%).

[0046] Building on the aforementioned end-to-end exploration framework, we also propose a multi-level grid map state representation method that captures environmental information at different scales and improves computational efficiency by over 10 times. Thanks to expert demonstrations, EDGE is the first exploration method that can be trained and applied in large-scale environments using only raw grid maps as training input. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The process of autonomous robot exploration in a large-scale indoor environment using EDGE according to an embodiment of the present invention;

[0048] Figure 2 EDGE framework provided by the embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a map rendering experiment provided by an embodiment of the present invention;

[0050] Figure 4 This figure compares the exploration performance of EDGE and PPO in 100 different scenarios provided by the embodiments of the present invention.

[0051] Figure 5 This is a diagram showing the evaluation results of exploration rewards and training rewards when different maps are removed from the original multi-level map in the ablation experiment provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of a simulation experiment provided by an embodiment of the present invention;

[0053] Figure 7A schematic diagram of a physical experiment provided by an embodiment of the present invention;

[0054] Figure 8 Schematic diagram of a robot autonomous exploration device based on expert demonstration guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0056] like Figure 2 As shown in the figure, the present invention provides a robot autonomous exploration method guided by expert demonstrations. This method consists of two phases: demonstration generation and policy training. In the first phase, expert exploration demonstrations are generated by solving the WRP problem with complete structural information about the environment. Subsequently, in the second phase, this demonstration data and the robot's experience in online exploration are used to jointly train the policy.

[0057] In practice, expert demonstrations are first generated and stored in an experience replay buffer, which allows these demonstrations to be repeatedly sampled and used in different policy training runs.

[0058] Exploratory Problem Modeling

[0059] Robot exploration can be modeled as a sequential decision problem, which can be defined as a tuple in the Markov decision process (MDP) framework. At each time t, the robot observes a state from the explored area Follow its exploration strategy To perform an action To explore new areas, the robot will receive a reward r after the action is executed t =R(s t ,a t ,s t+1 ), and obtain a new observation s t+1 ~T(s|s t ,a t ). The goal of reinforcement learning training is to find an optimal strategy π * , so that the maximum cumulative discounted reward is obtained under the premise of a discount factor γ∈(0,1], as shown in formula (1).

[0060]

[0061] State space. The environment to be explored W can be represented as a 2D grid map, with different grayscales representing the free space W f , obstacle W o , and the unknown area W uOur state space is defined as S = {V, I}. The historical behavior information V = {v0, v1, ..., v n} is designed to help robots try new exploration behaviors, where v i =(μ t-i ,a t-i ) contains the robot's historical position and the actions it performs at that position. Multi-level observation I = {I o ,I l ,I g}Contains maps of different scales: I o Is a 4×4 obstacle avoidance map (unit is pixel, the same below), I l It is a 64×64 local map. g A global map, scaled to 64×64. Local map I l Focusing on the immediate similarities between different maps rather than differences (such as rooms and corridors) improves model robustness. In contrast, the global map focuses on the entire environment, thereby improving the model's long-term planning ability.

[0062] Action Space. We discretize the action space into eight motion primitives, each corresponding to a fixed number of pixel movements (actually 3 pixels) in eight directions on a two-dimensional plane. These eight directions are {forward, backward, right, left, right front, left front, right back, left back}, i.e., directions with different multiples of 45°.

[0063] Reward function. Since the agent can learn from expert demonstrations, a simple reward mechanism can be used. The proportion of the explored area at time t is ρ t After executing an action, the reward obtained by the agent is As shown in Formula 2. Exploration reward Evaluate the proportion of newly explored areas, step penalty Used to prevent the agent from traveling along invalid paths, collision penalty Used to punish collisions. It means the reward for completing the task.

[0064]

[0065] Expert Demonstration Generation

[0066] Although generating globally optimal paths in online exploration tasks is challenging, determining the shortest traversal path in a known environment is a well-established problem. This problem is equivalent to the Shortest Path Planning (WRP) problem in computational geometry. By solving WRP in exploration tasks with prior information (i.e., the complete environment structure), we can compute the optimal traversal path and convert it into expert demonstrations for policy training.

[0067] We adopt a two-stage decoupling method to solve the WRP problem in the exploration task. We model the environment to be explored as a polygon Assume that the robot has a fixed field of view (FOV) and obstacles in the environment block its view, so its perception area is limited to the neighborhood polygon of its position ψ Given any two coordinates ψ in the environment i and ψ j ,τ i,j represents the shortest collision-free distance connecting these two coordinates. Our goal is to determine a set of observation points Ψ={ψ1,ψ2,…,ψ n} so that all edges of polygon P can be covered and observed, while minimizing the traversal distance of the observation point set, as shown in Formula 3:

[0068]

[0069] The first stage of our method is to compute a set of convex polygons, denoted as C, that cover the polygon At the same time, the number of convex polygons in the set is minimized. This process is described in detail in Algorithm 1. Because the convexity of a polygon ensures that a line segment between any two points within it does not exceed its boundary, the entire polygon can be fully observed from any point within it. Based on this property of convex polygons, we model the watcher's path problem as: planning a path that passes through at least one point within each convex polygon in the set C.

[0070] Specifically, Algorithm 1 includes:

[0071] Algorithm description: Maximum area convex set solution algorithm

[0072] enter

[0073] P: environment to be explored

[0074] d: sensor range

[0075] ∈: target coverage

[0076] M: maximum number of iterations (max iterations)

[0077] Output

[0078] C: convex polygon set

[0079] initialization

[0080] 1. Initialize the convex polygon set C to an empty set:

[0081] 2. Initialize the uncovered area U to the entire environment P: U←P

[0082] Algorithm steps

[0083] 1. Select a point p0 in the uncovered area U

[0084] 2. Calculate the polygon V visible from point p0 based on the environment P, point p0, and sensor range d

[0085] 3. Calculate the largest convex polygon c0 among the visible polygons and store it in cList

[0086] 4. Iterate optimization, select point p in c0, repeat steps 2-3, and store the newly obtained convex polygon in cList

[0087] 5. Select the convex polygon with the largest intersecting area with U in cList and set it as c

[0088] 6. Store c in the convex polygon set, set c as the known area, and update the position area U

[0089] Repeat algorithm steps 1-6 until the ratio of the unknown area to the total area is less than the set value

[0090] In the second stage, we first sample from the boundaries of each convex polygon, and these boundaries do not intersect with obstacles, because traversing a polygon must pass through its edges. Then, based on our discretized action space, we use the A* algorithm to calculate the shortest collision-free path between the sample points and their corresponding movement costs, where these sample points do not belong to the same convex polygon. Finally, we obtain the polygon by solving the generalized traveling salesman problem (GTSP) The optimal path Ψ in the WRP is . For partially known environments, we mask the explored area and treat the remaining area as a set of polygons with holes. We then incorporate the closest sampling points between these polygons into the GTSP solution as an optimization term.

[0091] Finally, the path is applied to the environment to establish the robot's action set A = {a1, a2, ..., an}. Subsequently, we reset the environment to an unexplored state, allowing the robot to perform optimal exploration according to the action set A and collect state transition data {s t ,a t ,r t ,s t+1 Repeat this process until a sufficient number of expert demonstrations are obtained.

[0092] Learning from demonstration and experience

[0093] After obtaining a large number of high-quality expert demonstrations, we need to design a reinforcement learning algorithm that enables the robot to learn exploratory behaviors from them. We use Proximal Policy Optimization (PPO) as the basis for this algorithm because of its advantages such as simple implementation and good performance. Drawing on ideas from the literature, we improve PPO's sampling method so that it can learn from both expert demonstrations and the agent's own experience. The objective function of PPO can be expressed as:

[0094]

[0095] Where ε is a hyperparameter, and π θ They are sampling strategy and training strategy, is an advantage function. Let Denotes the expert demonstration set, E denotes the agent experience, then the sampling strategy can be expressed as:

[0096]

[0097] in Representatives from sampling, φ∈(0,1) is a random number, and c is a hyperparameter initialized to 0.5.

[0098] This approach enables the agent to adjust its learning policy based on the balance between guided exploration and autonomous experience. With this approach, we treat expert demonstrations as actions the agent would take in the real environment. We can do this because, in PPO, the policy only specifies a distribution over the action space. By forcing the policy's actions to be equal to the demonstration actions, rather than sampling from the policy distribution, we accelerate the exploration phase.

[0099] like Figure 1 Figure 1 illustrates autonomous robotic exploration in a large-scale indoor environment using EDGE. Left: A 3D point cloud map of the environment used for localization. Right: The voxel map, which is projected onto a 2D plane and used as input for the neural network.

[0100] experiment

[0101] Our method was trained in simulation and tested both in simulation and in the real world. At the beginning of each exploration mission, the robot was randomly placed in a collision-free location. The robot moved at a speed of 1.5 m / s and was equipped with an omnidirectional ranging sensor with a maximum detection range of 12 meters. All simulation experiments were conducted on an Ubuntu 18.04 system equipped with an Intel(R) Xeon(R) Gold 6326 CPU and an NVIDIA GeForce RTX 3090 GPU.

[0102] Training settings

[0103] Our training used two types of rendered environments: dungeons and rooms, with 80% of the scenes used for training and 20% for testing. Each map pixel in the rendered environment represents 1 meter in the real world. Successful exploration was determined by exploring more than 97% of the total area; failure occurred if a collision occurred or if a single exploration exceeded 500 steps. Before training, we pre-collected 10,000 expert demonstrations from a variety of scenarios. We used the Gymnasium framework to implement interaction between the agent and the environment, and the reinforcement learning implementation was based on StableBaselines3.

[0104] Rendering map experiments

[0105] To comprehensively evaluate EDGE's performance across various scenarios, we categorized dungeon maps into three levels of complexity: simple, medium, and complex. We compared EDGE with three heuristic-based exploration methods (Frontiers, TARE, and FAEL) and two state-of-the-art deep reinforcement learning (DRL)-based methods (ARIADNE and Cao et al.), using the exploration distance as the evaluation metric. We then further compared EDGE with ARIADNE and [the results are missing from the original text] to assess computational efficiency. Finally, we conducted ablation experiments to verify the impact of expert demonstrations on policy convergence speed and exploration effectiveness.

[0106] Explore performance.

[0107] For each scene type, we randomly selected 20 maps for testing. Each method was run five times for each scene type, and the average exploration distance was recorded as the result. FAEL and TARE were run using their default parameters; TARE also used a configuration file for indoor scenes, which is similar in size and structure to our experimental environment. ARIADNE's training is limited to inputting 640×480 grid maps, so we needed to resize the input maps to a suitable size for training and testing.

[0108] As shown in Table 1, EDGE outperforms all baseline methods in exploration distance across all scene types. Specifically, EDGE improves the exploration distance by 9% in simple scenes, 11% in medium scenes, 13% in complex scenes, and 19% in room scenes. This is primarily due to EDGE's ability to predict the long-term visit sequence to unexplored areas, significantly improving exploration performance across all scenes, especially in complex ones.

[0109] like Figure 3 This example shows a rendering map experiment. A comparison of exploration trajectories and efficiency in a sample scene. Trajectories start at the green square and end at the blue tilted cross.

[0110]

[0111] Table 1: Comparison of exploration performance

[0112] Computational efficiency.

[0113] Thanks to the guidance of expert demonstrations, EDGE can achieve fast training convergence even when using the original grid map as input (with only simple cropping and scaling). In addition, in the decision-making stage, EDGE does not rely on the utility evaluation module or the local planner. Therefore, as shown in Table 2, on our experimental equipment, the average processing time for generating state representations from the grid map is only 0.0002 seconds, while the neural network inference takes about 0.012 seconds. This shows that compared with the current state-of-the-art reinforcement learning-based exploration methods ARIADNE and , EDGE's decision time is reduced by more than 10 times. This efficient computing capability significantly improves the real-time performance of mobile robots with limited computing resources in exploration tasks and enables them to be effectively deployed on actual robot platforms.

[0114]

[0115] Table 2: Computational efficiency comparison

[0116] Ablation experiment.

[0117] We conducted an ablation experiment to evaluate the impact of expert demonstration on the exploration task. In this experiment, the expert demonstration part was removed and the robot learned only through its own experience, that is, the sampling ratio was set to c = 0, and the policy learning degenerated into PPO. Figure 4 As shown in the figure, PPO struggles to complete the task due to its low exploration efficiency and quickly reaches the maximum number of steps set. It significantly lags behind EDGE in both convergence speed and overall reward performance. The experimental results demonstrate the important role of expert demonstrations in improving training speed and enhancing exploration performance.

[0118] like Figure 4 Figure 2 shows ablation experiments. (a) (b) Comparison of exploration performance between EDGE and PPO in 100 different scenarios. (c) Comparison of rewards during training under five different random seeds. The curves in the figure represent the mean, minimum, and maximum values ​​of each metric. Comparison of computational efficiency.

[0119] In this application, we conducted additional ablation experiments in the training phase and exploration tasks to further evaluate the role of multi-level maps. Figure 5 As shown in Figure 2, in each ablation experiment, we removed the obstacle map, local map, or global map from the original multi-level map and evaluated it in terms of exploration reward and training reward.

[0120] EDGE (without an obstacle map): EDGE without an obstacle map cannot effectively identify obstacles in the early stages of training, resulting in significantly lower performance than standard EDGE.

[0121] EDGE (without a local map): EDGE without a local map lacks the ability to be used for local planning and identify common areas, and is inferior to standard EDGE in terms of exploration performance.

[0122] EDGE (without a global map): EDGE without a global map lacks the ability to plan long-term and is unable to complete the exploration task within the specified number of steps.

[0123] Gazebo simulation experiment

[0124] We further tested the proposed EDGE method in a large-scale Gazebo environment to evaluate its scalability in unseen environments. The robot platform used in the experiment was a Clearpath Jackal equipped with a Velodyne VLP-16 lidar providing a 360° field of view. The environment synthesized a 110m x 130m indoor scene.

[0125] like Figure 6 This figure shows a simulation experiment comparing the exploration paths of the four algorithms in a large scene. The color intensity gradually changes from red to purple as the robot moves.

[0126] In actual testing, we first constructed a 3D grid voxel map (UFOMap) and converted it into a 2D occupancy grid map as observation input for reinforcement learning. These observations were scaled and cropped before being fed into the neural network, which generated the robot's next action. Whenever the robot reached a target point, EDGE would plan the next exploration target for it. This process continued iteratively until the entire area was explored.

[0127] We evaluate the exploration performance of EDGE by comparing it with several state-of-the-art methods (TARE, FAEL, Cao et al.) on the exploration distance metric. Figure 6 As shown in Figure 2, the path length required by EDGE to complete the exploration task is approximately 14% shorter than that of the baseline planning method. Although existing methods take long-term exploration benefits into account to some extent, the paths planned by EDGE have fewer redundant moves and are closer to the global optimal solution.

[0128] Physical experiment

[0129] To verify the potential of EDGE in real-world applications, we conducted tests in a real indoor exhibition hall measuring 60 meters by 14 meters. This environment contains furniture, stairs, and fitness equipment, and no additional model training was performed. The robot platform used in the experiment was the Clearpath Jackal, equipped with a mini computer (equipped with an Intel Core i7-13620H CPU and an NVIDIA GeForce RTX 4060 GPU) and a Velodyne VLP-16 lidar. Figure 7 shown.

[0130] like Figure 7 Shown is a real-life experiment. Exploration took place in a 60m x 14m exhibition hall containing furniture, stairs, and fitness equipment.

[0131] We set the UFOMap resolution to 0.2 meters, and all other parameters were consistent with the simulation settings. To further ensure robot safety and enhance the regularity of the map structure, we inflated the occupied areas within the grid map. In the experiment, the robot explored the entire environment in 223.4 seconds, covering a distance of 273.3 meters at an average speed of 1.2 meters per second. This result demonstrates that EDGE is compatible with actual robotic systems and enables efficient exploration in real-world environments.

[0132] Corresponding to the aforementioned embodiment of a robot autonomous exploration method based on expert demonstration guidance, the present invention also provides an embodiment of a robot autonomous exploration device based on expert demonstration guidance.

[0133] See also Figure 8 An embodiment of the present invention provides a robot autonomous exploration device based on expert demonstration guidance, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a robot autonomous exploration method based on expert demonstration guidance in the above embodiment.

[0134] The embodiment of the robot autonomous exploration device based on expert demonstration guidance provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 8 As shown in the figure, a hardware structure diagram of a robot autonomous exploration device based on expert demonstration guidance provided by the present invention is located in any device with data processing capability, except Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0135] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0136] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0137] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the robot autonomous exploration method based on expert demonstration guidance in the above embodiment is implemented.

[0138] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0139] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the robot autonomous exploration method based on expert demonstration guidance.

[0140] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0141] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A robot autonomous exploration method based on expert demonstration guidance, characterized in that: The method includes: S1. Robot exploration is modeled as a sequential decision problem in a Markov decision framework, where the action space is discretized into motion primitives, the environment to be explored is represented as a grid map, and the state space is constructed using historical behavior information and maps of different scales. S2. Solve the shortest path planning problem in an exploration task with a complete environment structure and obtain expert demonstration; S3. Using the objective function of proximal policy optimization, the expert demonstration is used as the action taken by the robot in the real environment, and the strategy is optimized using the expert demonstration operation and the agent's experience; S4. Use the updated strategy to enable the robot to explore the unknown environment.

2. The robot autonomous exploration method based on expert demonstration guidance according to claim 1, characterized in that: The robot exploration is modeled as a sequential decision problem under the Markov decision framework, specifically including: In the Markov decision process framework, it is defined as a tuple At each time t, the robot observes a state from the explored area Follow its exploration strategy To perform an action To explore new areas, the robot will receive a reward r after the action is executed t =R(s t ,a t ,s t+1 ), and obtain a new observation s t+1 ~T(s|s t ,a t ), the optimal strategy is obtained by solving the problem of obtaining the maximum cumulative discount reward under the premise of the existence of a discount factor.

3. The robot autonomous exploration method based on expert demonstration guidance according to claim 1, characterized in that: The state space is specifically: The environment to be explored W is represented as a 2D grid map, and different grayscales represent the free space W f , obstacle W o , and the unknown area W u , the state space is defined as S = {V, I}, and the historical behavior information V = {v0,v1,…,v n } is designed to help robots try new exploration behaviors, where v i =(μ t-i ,a t-i ) contains the robot's historical position and the actions it performs at that position. The multi-level observation I = {I o ,I l ,I g }Contains maps of different scales: I o It is an obstacle avoidance map. l It is a 64×64 local map. g It is a global map scaled to the size of the local map.

4. The robot autonomous exploration method based on expert demonstration guidance according to claim 1, characterized in that: The reward function is specifically: Explore Rewards Evaluate the proportion of newly explored areas, step penalty Used to prevent the agent from traveling along invalid paths, collision penalty Used to punish collisions. It means the reward for completing the task.

5. The robot autonomous exploration method based on expert demonstration guidance according to claim 1, characterized in that: The method for solving the shortest path planning problem in an exploration task with a complete environment structure specifically includes: using a two-stage decoupling method to solve the WRP problem in the exploration task. In the first stage, the environment to be explored is first modeled as a polygon with obstacles, and a set of convex polygons, denoted as C, is calculated using the maximum area convex set solving algorithm to cover the polygon. At the same time, minimize the number of convex polygons in the group; The second stage: sampling from the boundaries of each convex polygon, and these boundaries do not intersect with obstacles, according to the discretized action space, using the A* algorithm to calculate the shortest collision-free path between the sampling points and their corresponding movement costs, where these sampling points do not belong to the same convex polygon; finally, the polygon is obtained by solving the generalized traveling salesman problem The optimal path Ψ in WRP is applied to the environment to establish the robot's action set.

6. The robot autonomous exploration method based on expert demonstration guidance according to claim 5, characterized in that: The maximum area convex set solution algorithm is specifically as follows: Initialization: Initialize the convex polygon set to an empty set and initialize the unknown area to the entire environment Calculation steps: Calculate the initial calculation point in the unknown area. Calculate the polygon visible from the initial calculation point based on the environment and sensor range; Calculate the largest convex polygon c0 in the visible polygon and store it in the first convex polygon list Perform iterative optimization, select a calculation point in the obtained maximum convex polygon, repeat the visible polygon calculation and the maximum convex polygon according to the calculation point and store them in the first convex polygon list; Select the convex polygon c with the largest intersection area with the uncovered area from the first convex polygon list, store it in the convex polygon set, set the convex polygon c as the known area, and update the unknown area; Repeat the above calculation steps until the ratio of the area of ​​the unknown region to the total area is less than the set value, and output the convex polygon set.

7. The robot autonomous exploration method based on expert demonstration guidance according to claim 6, characterized in that: In the first stage, for some known environments, the explored areas are masked and the remaining areas are treated as a set of polygons with holes. The nearest sampling points between these polygons are then used as optimization items and incorporated into the GTSP solution algorithm.

8. The robot autonomous exploration method based on expert demonstration guidance according to claim 1, characterized in that: The S3 is specifically: The objective function of PPO is expressed as: Where ε is a hyperparameter, and π θ They are sampling strategy and training strategy, is an advantage function, Denotes the expert demonstration set, E denotes the agent experience, then the sampling strategy can be expressed as: in represents sampling from the expert demonstration set, v E represents sampling from the agent’s experience, φ∈(0,1) is a random number, and c is a hyperparameter.

9. A robot autonomous exploration device based on expert demonstration guidance, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the robot autonomous exploration method based on expert demonstration guidance as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a robot autonomous exploration method based on expert demonstration guidance as described in any one of claims 1 to 8 is implemented.

Images