Autonomous environment exploration method and system for mobile robot in unknown environment

Abstract

The invention relates to an autonomous environment exploration method and system for a mobile robot in an unknown environment, and the method comprises the steps: obtaining the current position of the robot, and constructing a needed environment map; training a local environment exploration network through robot and environment interaction; performing local environment exploration by adopting the trained local environment exploration network, and recording track points until the local environment exploration is finished; using a randomness algorithm to screen out junction points of the explored area and the non-explored area, calculating to obtain center points, and screening out candidate points from the center points; calculating a track point with the minimum distance from each candidate point to form a candidate point-track point pair; calculating the gravitational force of the candidate points and the repulsive force of the trajectory points, and selecting a candidate point-trajectory point pair with the maximum resultant force; and the robot moves to the track point with the maximum resultant force along the historical track, and returns to the local environment exploration process until the environment exploration is completed. According to the method disclosed by the invention, rapid and efficient autonomous exploration of the robot in an unknown complex environment can be realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent robot environmental exploration, and particularly to a method and system for autonomous environmental exploration of a mobile robot in an unknown environment. Background Art

[0002] With the rapid development of technology, intelligent mobile robots have become an indispensable key technology in multiple industries and are widely used in industrial production, logistics distribution, medical health, military defense and other fields. In tasks such as disaster relief, urban inspection, intelligent agriculture, and resource exploration, intelligent mobile robots have demonstrated great potential and value. In these applications, the robot's autonomous navigation and environmental perception technology play a crucial role. It can not only effectively reduce the human burden, improve work efficiency and safety, but more importantly, it can replace humans to perform complex and high-risk tasks in extreme or dangerous environments, such as working under harsh conditions like high temperature, high radiation, and high pressure.

[0003] The autonomous exploration technology for unknown environments refers to the situation where a robot or an intelligent agent gradually obtains information about unknown areas through self-perception and decision-making in the absence of complete environmental information. This technology is widely used in indoor and outdoor robot navigation, unmanned driving, disaster rescue and other fields. Traditional path planning methods usually rely on known maps, but in actual scenarios, robots often need to deal with dynamically changing, complex obstacles, and environments that cannot be known in advance. Therefore, autonomous exploration technology must combine environmental perception, real-time map updating, and planning strategies to achieve the goal of comprehensive exploration from the initial state. Common autonomous exploration algorithms include those based on artificial potential field method, boundary method, sampling method, and graph search method, etc. In recent years, with the development of neural networks, deep reinforcement learning technology has also been gradually applied to the autonomous exploration of unknown environments. However, in the face of large-scale and complex unknown environments, these methods still face problems such as repeated exploration, incomplete environmental coverage, and being easily trapped in local optima, and there is still room for improvement in terms of environmental exploration efficiency and coverage rate. Summary of the Invention

[0004] To solve the problems of repeated exploration, incomplete environmental coverage, and being easily trapped in local optima that occur when traditional robot exploration methods face dynamically changing, complex obstacles, and environments that cannot be known in advance. The present disclosure proposes a method for autonomous environmental exploration of a mobile robot in an unknown environment to solve the above problems.

[0005] According to one aspect of the present disclosure, there is provided a method for autonomous environmental exploration of a mobile robot in an unknown environment, including:

[0006] S10. Obtain the current position of the robot and the sensing range of the sensor, and construct the required environmental map, where the environmental map includes a global true map, a global exploration map, and a local exploration map;

[0007] S20. Based on the local exploration map, construct a local environmental exploration network;

[0008] S30. Store the multiple five-tuple data obtained by the interaction between the robot and the environment into the experience pool, and use the multiple five-tuple data in the experience pool to train the local environmental exploration network, where the multiple five-tuple data include the executed action, the local maps before and after the executed action, the action reward, and the end flag;

[0009] S40. Use the trained local environmental exploration network to conduct local environmental exploration and record the trajectory points until the local environmental exploration ends;

[0010] S50. Use a randomness algorithm to screen out the nodes at the junction of the explored area and the unexplored area from the global exploration map, cluster the nodes through a clustering algorithm, and divide them into several different clusters, calculate and screen out the center points of each cluster;

[0011] S60. Screen out candidate points from the center points and determine the number of candidate points;

[0012] S70. Calculate the distances between each candidate point and all the trajectory points, find the trajectory point with the minimum distance to each candidate point, and form a candidate point - trajectory point pair;

[0013] S80. Calculate the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point - trajectory point pair, and select the candidate point - trajectory point pair with the largest resultant force after the interaction of the gravitational force and the repulsive force;

[0014] S90. Make the robot move along the recorded trajectory points to the trajectory point in the candidate point - trajectory point pair selected in step S80, and return to step S40 until the environmental exploration is completed.

[0015] Preferably, the construction of the local environmental exploration network includes constructing a reward function based on the path length, collision, and exploration area increment, and the reward function is expressed as:

[0016] r t =r explore +r dis +r collision ,

[0017] In the formula, r explore is the exploration reward, r dis is the path length reward, and r collision is the collision reward;

[0018] Among them,

[0019] r explore = c 1 ·ΔA explore ,

[0020] r dis = -c 2 ·||P t+1 -P t ||,

[0021]

[0022] In the formula, ΔA explore is the increase in the area of the explored region from time t to time t + 1, c 1 is the weight coefficient of the exploration reward, P t+1 and P t respectively represent the positions of the robot at time t + 1 and time t, c 2 is the weight coefficient of the path length reward, used to control the influence of the path length on the total reward, c 3 is a constant representing the collision reward (c 3 > 0).

[0023] Preferably, a plurality of five-tuple data obtained by the robot interacting with the environment are stored in an experience pool, and the plurality of five-tuple data are:

[0024] exp t = (s t , a t , s t+1 , r t , done),

[0025] In the formula, s t is the local map observed by the agent at time t, a t is the action output by the neural network and executed by the agent in the observed state s t , s t+1 is the local map observed by the agent after executing the action a t , r t is the action reward of the evaluation function for the action output by the neural network, and done is a flag indicating whether the task is completed.

[0026] Preferably, the center point of each cluster is obtained by calculating the average coordinates of all points within each cluster, and is expressed as:

[0027]

[0028] In the formula, x i and y i are the points p within the cluster C k ​i The coordinates, where m is the number of points within the cluster.

[0029] Preferably, candidate points are selected from the central points, specifically as follows:

[0030] Taking the central point as the center, a square area of a certain size is intercepted from the global exploration map as the local map, and the ratio Ue of the number of unexplored area grids in the local map to the total number of grids is calculated, and the central points with Ue > δ are selected as candidate points.

[0031] Preferably, the number of candidate points is determined, specifically as follows: Determine whether the number of candidate points is 0. If it is not 0, continue to execute step S70. If it is 0, there are no nodes at the junction of the explored area and the unexplored area, and the environment exploration is completed.

[0032] Preferably, the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point - trajectory point pair are calculated. The gravitational force calculation formula of the candidate point is:

[0033]

[0034] In the formula, p represents the candidate point position, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, and ξ represents the gravitational field coefficient, which determines the size of the gravitational field;

[0035] The repulsive force calculation formula of the trajectory point is:

[0036]

[0037] In the formula, p 1 represents the trajectory point position, p c represents the current position of the robot, and d(p 1 , p c ) represents the distance that needs to be traveled from the trajectory point p 1 along the historical trajectory to the current position p of the robot c , and η represents the repulsive field coefficient, which determines the size of the repulsive field.

[0038] According to one aspect of the present disclosure, there is provided a mobile robot autonomous environment exploration system in an unknown environment, including:

[0039] A robot position and environment map acquisition module, which acquires the current position of the robot and the sensing range of the sensor, and constructs the required environment map. Among them, the environment map includes a global real map, a global exploration map, and a local exploration map;

[0040] A local environment exploration network construction module, which constructs a local environment exploration network based on the local exploration map;

[0041] The local environment exploration network training module stores multiple five-tuple data obtained by the interaction between the robot and the environment into the experience pool, and uses the multiple five-tuple data in the experience pool to train the local environment exploration network. Among them, the multiple five-tuple data include the executed action, the local maps before and after the executed action, the action reward, and the end flag;

[0042] The local environment exploration module uses the trained local environment exploration network to conduct local environment exploration and records the trajectory points until the local environment exploration ends;

[0043] The center point calculation module uses a randomness algorithm to screen out the nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes through a clustering algorithm, divides them into several different clusters, and calculates and screens out the center points of each cluster;

[0044] The candidate point screening module screens out candidate points from the center points and determines the number of candidate points;

[0045] The candidate point-trajectory point pair acquisition module calculates the distances between each candidate point and all the trajectory points, and finds the trajectory point with the minimum distance from each candidate point to form a candidate point-trajectory point pair;

[0046] The candidate point-trajectory point pair with the maximum resultant force acquisition module calculates the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the maximum resultant force after the interaction between the gravitational force and the repulsive force;

[0047] The environment exploration module makes the robot move along the recorded trajectory points to the trajectory point with the maximum resultant force, returns to the local environment exploration module to conduct local environment exploration until the environment exploration is completed.

[0048] According to one aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to: execute the above-mentioned autonomous environment exploration method for a mobile robot in an unknown environment.

[0049] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned autonomous environment exploration method for a mobile robot in an unknown environment is implemented.

[0050] Compared with the prior art, the beneficial effects of the present disclosure are:

[0051] 1) The environmental global exploration strategy designed by the present disclosure determines boundary points through clustering and uses the artificial potential field method for target point selection. The combination of the two constitutes a complete global point selection strategy, which avoids the robot being trapped in the local optimum by relying only on local information and ensures the integrity of environmental exploration. At the same time, clustering reduces the number of target points, reduces the computational amount, and reduces the demand of the algorithm for device computing power.

[0052] 2) The method of the present disclosure uses reinforcement learning for obstacle avoidance and exploration in local exploration and adopts an artificial strategy for point selection at the global level. Reinforcement learning optimizes the efficiency of obstacle avoidance and exploration in unknown environments, while the artificial strategy ensures the reasonable selection of global exploration points. This design not only ensures the flexibility and environmental adaptability of the strategy, but also effectively avoids omission and repeated exploration, improving the comprehensiveness and stability of exploration.

[0053] 3) During the process of local environmental exploration, the method of the present disclosure effectively reduces the logical misjudgment caused by unknowable factors due to obstacle occlusion by marking the area that cannot be accurately sensed due to being occluded by obstacles within the sensing range of the robot sensor as an obstacle shadow area. At the same time, a convolutional layer and a pooling layer are introduced into the neural network structure to facilitate automatic extraction of environmental features, thereby improving the accuracy of environmental exploration and enhancing the adaptability to complex unknown environments.

[0054] 4) The method of the present disclosure records the environmental exploration process through trajectory points, matches the nearest trajectory point for each candidate point to form a candidate point - trajectory point pair. After selecting the candidate point, the robot only needs to move along the historical trajectory to the target point without an additional path planning module. This method simplifies path planning, reduces computational overhead, realizes fast and accurate navigation using the existing trajectory, improves the exploration efficiency and system response speed, and enhances stability and reliability.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present disclosure.

[0056] Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0057] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0058] Figure 1 Shows a flowchart of a method for a mobile robot to autonomously explore the environment in an unknown environment;

[0059] Figure 2Shows the flowchart of the autonomous exploration method of a mobile robot in an unknown environment in the present disclosure;

[0060] Figure 3 Shows the schematic diagrams of the global real map, the global exploration map, and the local exploration map in an embodiment of the present disclosure;

[0061] Figure 4 Shows the schematic diagram of the network structure of reinforcement learning in the local exploration map in an embodiment of the present disclosure;

[0062] Figure 5 Shows the schematic diagram of the system architecture of the local environment exploration algorithm based on reinforcement learning in an embodiment of the present disclosure;

[0063] Figure 6 Shows the schematic diagram of the exploration result of a robot in an unknown environment in an embodiment of the present disclosure;

[0064] Figure 7 Shows the block diagram of the autonomous environment exploration system structure of a mobile robot in an unknown environment. Detailed implementation manners

[0065] The following will describe various exemplary embodiments, features, and aspects of the present disclosure in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0066] The special word "exemplary" herein means "serving as an example, an embodiment, or illustrative". Any embodiment described as "exemplary" herein does not have to be construed as being superior to or better than other embodiments.

[0067] The term "and / or" herein merely describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set composed of A, B, and C.

[0068] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0070] Embodiment 1

[0071] Based on the above idea, the present invention proposes a method for autonomous environmental exploration of a mobile robot in an unknown environment. Figure 1 The flowchart of a method for autonomous environmental exploration of a mobile robot in an unknown environment is shown. The method includes:

[0072] S10. Obtain the current position of the robot and the sensing range of the sensor, and construct the required environmental map, where the environmental map includes a global real map, a global exploration map, and a local exploration map;

[0073] S20. Based on the local exploration map, construct a local environmental exploration network;

[0074] S30. Store multiple five-tuple data obtained by the interaction between the robot and the environment into an experience pool, and use the multiple five-tuple data in the experience pool to train the local environmental exploration network, where the multiple five-tuple data includes an executed action, local maps before and after the executed action, an action reward, and an end flag;

[0075] S40. Perform local environmental exploration using the trained local environmental exploration network and record trajectory points until the local environmental exploration ends;

[0076] S50. Use a randomness algorithm to screen out nodes at the junction of the explored area and the unexplored area from the global exploration map, cluster the nodes through a clustering algorithm, and divide them into several different clusters, calculate and screen the center points of each cluster;

[0077] S60. Screen out candidate points from the center points and determine the number of candidate points;

[0078] S70. Calculate the distance between each candidate point and all trajectory points, find the trajectory point with the minimum distance from each candidate point, and form a candidate point-trajectory point pair;

[0079] S80. Calculate the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and select the candidate point-trajectory point pair with the largest resultant force after the interaction of the gravitational force and the repulsive force;

[0080] S90. Cause the robot to move along the recorded trajectory points to the trajectory point in the candidate point-trajectory point pair selected in step S80, and return to step S40 until the environment exploration is completed.

[0081] An embodiment of the present disclosure provides a method for a mobile robot to autonomously explore the environment in an unknown environment. The flowchart of the specific method for the mobile robot to autonomously explore the unknown environment is as Figure 2 shown, including the following steps:

[0082] S10. Obtain the current position of the robot and the sensing range of the sensor, and construct the required environmental map, where the environmental map includes a global real map, a global exploration map, and a local exploration map.

[0083] In this embodiment, a simulation environment of the space to be explored and a robot model are built. The current position of the robot and radar data in the space to be explored are obtained, and based on this, the required environmental map is constructed, including a global real map, a global exploration map, and a local exploration map. Figure 3 Used to describe the update process of the 3 environmental maps, where Figure 3 (a), (b), and (c) respectively correspond to the construction processes of the global real map, the global exploration map, and the local exploration map.

[0084] The global exploration map refers to the environmental map of the entire scene for exploration. As the core tool when the robot explores the entire scene, the global exploration map is continuously updated during the process of environmental exploration and guides the robot to carry out subsequent exploration actions. The method for updating the global exploration map is as follows: Assume that the actual sensing range of the lidar carried by the robot is L, and now a value l smaller than L is set 0 as the current exploration range of the robot. During the exploration process, if the robot does not detect an obstacle, the space within the range from the robot's own position to L - l 0 is marked as the explored area. If an obstacle is detected, the position of the obstacle is updated as the obstacle area, and the space between the robot's own position and the obstacle is updated as the explored area. Specifically, as shown in Figure 3 (b), the white part represents the explored area, the gray part represents the unexplored area, and the black part represents the edge of the obstacle.

[0085] The global real map is a map drawn by the robot during the exploration process to record the real landform of the environment. The main difference between it and the global exploration map lies in the different lidar sensing ranges used when drawing the map. Specifically, as shown in Figure 3As shown in (a). In order to ensure the accuracy and completeness of the exploration map and effectively reduce the interference of noise information in the environment, the lidar detection distance used in the update of the global exploration map is deliberately set to be slightly smaller than the real perception range. Therefore, when the robot completes the exploration task using the global exploration map, the global real map drawn using real lidar data often shows a higher exploration coverage rate. In particular, the global real environment map mentioned in this embodiment is strictly drawn based on the actual perception capability of the lidar carried by the robot. It is mainly used for performance analysis, thereby providing a more accurate reference for the robot's exploration actions.

[0086] The local exploration map is a map representation used by robots to focus on surrounding areas and support immediate decision-making during their autonomous exploration of unknown environments. It is mainly used for detailed exploration of local environments. From a spatial perspective, the local exploration map delineates a limited area around the robot's current position, but it is sufficient to deal with close-range emergencies and support detailed perception and action planning. The size of this area is flexibly adjusted according to the effective range of the robot's laser radar, its flexibility of movement, and the importance of the task, and usually covers a surrounding range from several meters to tens of meters. The local exploration map is obtained as follows: With the robot's current position as the center, a square area of ​​a specific range is extracted from the global exploration map, and it is updated through real-time sensor data. The specific update method is: the obstacles perceived by the robot and part of the area behind them are marked as obstacle occlusion areas, while the attributes of other areas remain unchanged, as follows Figure 3 This is because the uncertainty of the area behind the obstacle may affect the accuracy of the decision during the robot's exploration of the environment. In particular, due to the different relative positions of the robot and the obstacle, the occlusion relationship of the obstacle will also change, so the local exploration map at each position is unique. It is important to note that the side length of the local map should be set larger than the detection range of the robot's lidar to ensure that it can fully perceive the surrounding environment.

[0087] S20. Based on the local exploration map, a local environment exploration network is constructed.

[0088] In the local map exploration problem of this embodiment, it is necessary to obtain the local exploration map of the robot's current location. Based on this local exploration map, the local exploration problem can be simplified into an optimization problem: select a target point P for the robot so that when it moves from its current position to P, the area of ​​the newly added unexplored area is maximized while satisfying the obstacle avoidance constraints and path feasibility.

[0089] The target point P in the local environment exploration is expressed as:

[0090]

[0091] In the formula, represents the set of all possible target points in the local map. T is the time used by the robot during the local map exploration process. p represents the current position of the robot. g(p) is a function for calculating the newly explored area when the robot moves from the current position to the target point P.

[0092] To solve the problem of robot local map exploration using reinforcement learning, it is necessary to model its exploration process and construct a corresponding Markov decision process (MDP) model. It should be noted that due to the limitation of the lidar detection range, the robot can usually only observe partial environmental information around itself and cannot comprehensively perceive the state of the entire environment. Therefore, the autonomous exploration process of the robot's local map is more suitable to be described by a partially observable Markov decision process (POMDP).

[0093] The model of the partially observable Markov decision process constructed for the problem of robot local map exploration can be represented by a seven-tuple (S, A, T, R, O, Z, γ). Among them: S is the set of all possible states of the environment; A is the set of all actions that the robot can execute; T is the state transition equation, and T(s t+1 |s t , a) represents the probability that the environment transfers from state s t to state s t+1 after the robot executes action a; R: S×A→R represents the immediate reward given by the environment after the robot takes action a in the current state s; O represents the set of observation results, that is, the finite set of all observation results that the robot's sensors can observe; Z represents the observation probability distribution, and Z(o|s t+1 , a t ) represents the probability that the robot can observe o after executing action a t and transferring to state s t+1 ; γ is the discount factor, which is used to balance the importance of the current reward and future rewards, and its value range is [0, 1].

[0094] Based on the design of the above seven-tuple, the basic process of the agent in local environment exploration can be described as follows: At time t, the state of the agent is s t . According to the policy function π θ (a t |s t ), the agent selects action a t , and then according to the state transition function T(s t+1 |s t , a t ), the environmental state transfers to the next state s t+1 . At the same time, the agent observes o according to the observation probability distribution Z(o|s t+1, a t ) Obtain the environmental observation value \(o\in Z\) and get the immediate reward \(r\). t (s t , a t ). This interaction process loops until the local exploration task ends.

[0095] Furthermore, use the SAC algorithm to construct the network architecture, including the design of the state space, action space, reward function, and network structure. First, use the local exploration map to construct the state space of the robot's local environment exploration problem. Since the adopted reinforcement learning algorithm can generate continuous actions based on probability distributions, the action space is set as a two-dimensional continuous action space. The target point output by the network is represented in polar coordinates relative to the robot's current position, with the robot's current position as the origin and the robot's orientation as the reference direction of the coordinate axis. The first dimension of the action space represents the angle of the target point relative to the robot's orientation, and its range can be set as \((\theta min , \theta max ); the second dimension represents the distance between the target point and the robot, and its range can be set as \((0, l 0 ). Among them, the specific value of l 0 can be adjusted according to the actual problem requirements, usually not exceeding the maximum detection range of the lidar carried by the robot to ensure the safety of the robot. Therefore, the robot's action vector can be expressed as \((\theta, l)\), and this polar coordinate representation method clearly describes the azimuth and distance of the target point relative to the robot, which helps to improve the efficiency and accuracy of the algorithm in target point planning.

[0096] To improve the numerical stability of the model, accelerate the convergence speed, and improve the training effect, and further enhance the versatility and portability of the method, each dimension of the action space is normalized, and the normalization formula is:[[]]

[0097]

[0098] In the formula, \(\theta norm is the normalized angle value, with a range between [-1, 1], and l norm is the normalized distance value, with a range of [0, 1].[[]]

[0099] The reward function is a key component of reinforcement learning. As the feedback mechanism of the agent, it defines the quality of the agent's behavior in different states, thereby driving the agent to continuously optimize its decision-making strategy and accelerating the training process. In the local map exploration task, the reward function needs to provide environmental feedback to the robot, guiding it to explore as much of the environmental area as possible in each decision while minimizing the length of the exploration path. In addition, considering the safety of the robot during movement, the action selection should avoid known obstacle areas as much as possible. Therefore, this embodiment introduces three basic rewards for the robot: exploration reward r explore , path length reward r dis , and collision reward r collision .

[0100] Specifically, for time t, the action reward r t obtained by the mobile robot is:

[0101] r t = r explore + r dis + r collision ,

[0102] The exploration reward r explore is defined as the reward given according to the increase in the area of the explored region in the global exploration map after the robot executes an action. Make corresponding transformations to the global exploration map, and represent the explored region and the unexplored region with binary matrices E t (i, j) respectively. The size of the matrix is N×M, where the explored region is represented by 1 and the unexplored region is represented by 0. Specifically, at time t, the exploration state of the grid (i, j) can be represented by the variable E t (i, j). If it has been explored, then E t (i, j) = 1; otherwise, E t (i, j) = 0. Then at time t, the number of grids in the explored region A explore，t can be expressed as:

[0103]

[0104] The increase in the area of the explored region ΔA explore from time t to time t + 1 is expressed as:

[0105]

[0106] The exploration reward r explore is expressed as:

[0107] r explore = c 1 ·ΔA explore ,

[0108] where c1 It is the weight coefficient for exploring rewards.

[0109] Path length reward r dis It is defined as the reward given according to the path length of the robot's movement. To encourage the robot to explore with the shortest path as much as possible, a negative reward can be set for the path length, that is, a small penalty is generated for each action step. This penalty should not be too large to avoid overly suppressing the robot's exploration behavior. The calculation formula for the path length reward is expressed as:

[0110] r dis = -c 2 ·||P t+1 -P t ||,

[0111] In the formula, P t+1 and P t respectively represent the positions of the robot at time t + 1 and time t, and c 2 is the weight coefficient of the path length reward, which is used to control the influence of the path length on the total reward.

[0112] Collision reward r collision It is defined as the reward given according to whether the robot collides. Specifically, when the robot collides, the reward is negative; if the robot does not collide, the reward is zero, which is expressed as:

[0113]

[0114] In the formula, c 3 is a constant used to represent the collision reward (c 3 > 0).

[0115] As shown in combination Figure 4 , since the input of the neural network is the local exploration map, in order to extract the planar structure information of the environment, a convolutional layer is introduced into the network, and the Relu function is used to activate after each convolutional layer to introduce non-linear transformation. In addition, after the activation function, a max pooling layer is added to reduce the size of the feature map, reduce the computational overhead and alleviate overfitting, while retaining the key information. Finally, the features are further processed through the fully connected layer, and the final result is given at the output layer.

[0116] S30. Store the multiple five-tuple data obtained by the interaction between the robot and the environment in the experience pool, and use the multiple five-tuple data in the experience pool to train the local environment exploration network, where the multiple five-tuple data include the executed action, the local maps before and after the executed action, the action reward, and the end flag.

[0117] In this embodiment, multiple five-tuple data obtained by the interaction between the robot and the environment are stored in the experience pool for the training of the agent. To ensure the diversity of experiences and prevent the model from falling into local optima, the robot randomly takes actions to obtain experience data in the early stage, and accumulates experience data while learning in the later stage. The multiple five-tuple data are as follows:

[0118] exp t =(s t , a t , s t+1 , r t , done),

[0119] where s t is the local map observed by the agent at time t, a t is the action output by the neural network and executed by the agent in the observed state s t , s t+1 is the local map observed by the agent after executing the action a t , r t is the action reward of the evaluation function for the action output by the neural network, and done is a flag indicating whether the task is completed. These data are the basis for reinforcement learning training. Through multiple interactions and experience accumulation, the agent learns how to select an action strategy that maximizes the cumulative reward. The interaction between the agent and the environment and the training process of the neural network are as Figure 5 shown.

[0120] Furthermore, multiple five-tuple data in the experience pool are used to train the local environment exploration network. The optimization objective of the local environment exploration network is:

[0121]

[0122] H(π(·|s t )) = E[-logπ(·|s t )],

[0123] where (s t , a t ) ~ π(·|s t ) represents the state-action sequence at time t generated under the policy function π, r(s t , a t ) is the immediate reward corresponding to the state s t and the action a t at time t, and H(π(·|s t )) represents the entropy of the policy function π in the state s tThe entropy value under this condition. The higher the entropy value, the greater the randomness of the strategy. α is the temperature coefficient, which is used to control the importance of the entropy term, so as to achieve a balance between exploration and reward. γ is the discount factor, which is used to balance the importance of the current reward and the future reward, and its value range is [0,1].

[0124] S40. Use the trained local environment exploration network to perform local environment exploration and record the trajectory points until the local environment exploration ends.

[0125] In this embodiment, the trained local environment exploration network is transplanted onto the robot for local environment exploration, the environmental map is gradually updated, and each trajectory point output by the neural network is recorded until the local environment exploration ends.

[0126] S50. Use a randomness algorithm to screen out the nodes at the junction of the explored area and the unexplored area from the global exploration map, cluster the nodes through a clustering algorithm, and divide them into several different clusters, and calculate and screen out the center points of each cluster.

[0127] In this embodiment, after the local environment exploration ends, the RRT algorithm is used to screen out the nodes at the junction of the explored area and the unexplored area from the global exploration map, that is, the boundary points. First, several points in the trajectory points recorded during the local exploration process are selected as the initial growth points, and these points are the root nodes of the RRT tree. All parameters such as the expansion step length and direction of the tree are shared. The step length controls the distance of each expansion and is usually set to a fixed value Δx. Each time an expansion is performed, a target point q rand is randomly selected from the current tree for expansion. This target point is randomly selected from the space and can be located within the explored area or the unexplored area. After determining q rand , the algorithm finds the node q near closest to the target point from the existing tree nodes, and expands a new node through this node. This is achieved by calculating the distance between each node and the target point:

[0128]

[0129] In the formula, (x rand , y rand ) and (x current , y current ) represent the coordinates of the target point and the current node respectively.

[0130] Next, the tree expands from the node q near towards the target point q rand to generate a new node q new :

[0131]

[0132] In addition, the new node q new must pass through collision detection to ensure that it is not located in the obstacle area. Whether the node is located in the spatial area can be determined by querying the grid map.

[0133] During the expansion process, the key lies in how to screen out the boundary points between the explored area and the unexplored area from the generated RRT tree. Specifically, the boundary points refer to those nodes located at the junction of the explored area and the unexplored area. Whenever the RRT tree expands to a new node, the algorithm checks the relationship between this node and its neighboring area. Specifically: If the node q new is located in the unexplored area and some of its adjacent nodes are located in the explored area, then the node q new is a boundary point. As the RRT tree continues to expand, the algorithm will repeat the above steps until enough boundary points are generated or a certain termination condition is reached. Finally, through the expansion of the RRT tree and boundary determination, we can obtain a set of points representing the boundary between the explored area and the unexplored area.

[0134] Use the DBSCAN algorithm to cluster the boundary points, divide the boundary points into several different clusters, and calculate the center point of each cluster. The center point of each cluster is obtained by calculating the average coordinates of all points within each cluster, expressed as:

[0135]

[0136] In the formula, x i and y i are the coordinates of the point p k within the cluster C i , and m is the number of points within the cluster.

[0137] Next, find the point within the cluster C k that is closest to the geometric center point , and use this point as the center point of the cluster.

[0138] S60. Screen out candidate points from the center points and determine the number of candidate points.

[0139] In this embodiment, screening out candidate points from the center points is specifically: taking the center point as the center, intercepting a square area of a certain size from the global exploration map as the local map, calculating the proportion Ue of the number of unexplored area grids in the local map to the total number of grids, and screening out the center points with Ue > δ as candidate points, where δ is a threshold and is adjusted accordingly according to the environmental complexity.

[0140] Further, determine the number of candidate points, specifically: Determine whether the number of candidate points is 0. If it is not 0, continue to execute step S70. If it is 0, there are no nodes at the junction of the explored area and the unexplored area, and the environment exploration is completed.

[0141] S70. Calculate the distance between each candidate point and all trajectory points, and find the trajectory point with the smallest distance from each candidate point to form a candidate point - trajectory point pair.

[0142] In this embodiment, for each candidate point, calculate its distance from all trajectory points, find the trajectory point with the smallest distance from it, and form a candidate point - trajectory point pair. The number of candidate point - trajectory point pairs is equal to the number of candidate points.

[0143] S80. Calculate the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point - trajectory point pair, and select the candidate point - trajectory point pair with the largest resultant force after the interaction of the gravitational force and the repulsive force.

[0144] In this embodiment, calculate the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point - trajectory point pair, and regard the candidate point and the trajectory point as a whole to calculate the numerical sum rather than the vector sum of the resultant force. The gravitational force formula of the candidate point is:

[0145]

[0146] In the formula, p represents the position of the candidate point, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, ξ represents the gravitational field coefficient, which determines the size of the gravitational field. ξ is a manually set value and is adjusted according to the map size and the obstacle density in the map. This coefficient is not a fixed value but is related to the obstacle distribution around the candidate point. Its expression is:

[0147]

[0148] In the formula, N shadow is the number of grids in the shaded occlusion area in the local grid map.

[0149] The repulsive force formula of the trajectory point is:

[0150]

[0151] In the formula, p 1 represents the position of the trajectory point, p c represents the current position of the robot, d(p 1 , p c ) represents moving from the trajectory point p 1 along the historical trajectory to the current position p of the robot cThe distance to be traveled, η represents the repulsive force field coefficient, which determines the size of the repulsive force field. η is a value set artificially and is adjusted according to the size and complexity of the map.

[0152] After obtaining the gravitational force of the candidate point and the repulsive force of the trajectory point, calculate the resultant force of the gravitational force of the candidate point and the repulsive force of the trajectory point. It should be noted that the calculation of the resultant force only considers the numerical superposition and does not consider the direction. The resultant force calculation formula is:

[0153] U t (p cand ,p traj )=U a (p cand )+U r (p traj ),

[0154] In the formula, p cand 、p traj respectively represent the candidate point and the trajectory point with the smallest distance corresponding to it. U a (p cand )、U r (p traj ) respectively represent the gravitational force of the candidate point and the repulsive force of the trajectory point with the smallest distance corresponding to it.

[0155] S90, so that the robot moves along the recorded trajectory points to the trajectory point in the candidate point - trajectory point pair selected in step S80, and returns to step S40 until the environmental exploration is completed.

[0156] Taking the multi-room simulation environment scene with obstacles built in Webots shown in (a) in Figure 6 as an example, using the method described in this embodiment for exploration, the exploration trajectory of the robot is as shown in (b) in Figure 6 . The real environment map constructed after the exploration is as shown in (c) in Figure 6 . It can be observed that the entire environment has been fully explored.

[0157] This embodiment provides an autonomous exploration method for unknown environments with small computational complexity, good real-time performance, and high efficiency. By training a local exploration strategy through reinforcement learning and combining the artificial potential field method, the robot can achieve fast, efficient, and complete autonomous exploration in unknown complex environments.

[0158] Embodiment 2

[0159] As another aspect of the embodiments of the present disclosure, an autonomous environment exploration system 100 for a mobile robot in an unknown environment is also provided. As shown in Figure 7 , it includes:

[0160] Robot position and environmental map acquisition module 1, which acquires the current position of the robot and the sensing range of the sensor, and constructs the required environmental map. Among them, the environmental map includes a global real map, a global exploration map, and a local exploration map;

[0161] Local environmental exploration network construction module 2, which constructs a local environmental exploration network based on the local exploration map;

[0162] Local environmental exploration network training module 3, which stores multiple five-tuple data obtained by the interaction between the robot and the environment in an experience pool, and uses the multiple five-tuple data in the experience pool to train the local environmental exploration network. Among them, the multiple five-tuple data includes the executed action, the local maps before and after the executed action, the action reward, and the end flag;

[0163] Local environmental exploration module 4, which uses the trained local environmental exploration network to perform local environmental exploration and records the trajectory points until the local environmental exploration ends;

[0164] Center point calculation module 5, which uses a randomness algorithm to screen out the nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes through a clustering algorithm, and divides them into several different clusters, and calculates and screens out the center points of each cluster;

[0165] Candidate point screening module 6, which screens out candidate points from the center points and determines the number of candidate points;

[0166] Candidate point-trajectory point pair acquisition module 7, which calculates the distances between each candidate point and all the trajectory points, and finds the trajectory point with the smallest distance from each candidate point to form a candidate point-trajectory point pair;

[0167] Candidate point-trajectory point pair with the largest resultant force acquisition module 8, which calculates the gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the largest resultant force after the interaction between the gravitational force and the repulsive force;

[0168] Environmental exploration module 9, which makes the robot move along the recorded trajectory points to the trajectory point with the largest resultant force, returns to the local environmental exploration module to perform local environmental exploration until the environmental exploration is completed.

[0169] Without contradiction, the above modules in the system of the embodiments of the present disclosure can implement any of the above implementation manners of the method.

[0170] Based on the description of the above embodiments, the embodiments of the present disclosure can achieve the following technical effects:

[0171] 1) The environmental global exploration strategy designed by the present disclosure determines boundary points through clustering and uses the artificial potential field method for target point selection. The combination of the two constitutes a perfect global point selection strategy, avoiding the robot from falling into local optimality relying only on local information and ensuring the integrity of environmental exploration. At the same time, clustering reduces the number of target points, reduces the computational amount, and lowers the requirement of the algorithm for device computing power.

[0172] 2) The method of the present disclosure uses reinforcement learning for obstacle avoidance and exploration in local exploration and adopts an artificial strategy for point selection at the global level. Reinforcement learning optimizes the efficiency of obstacle avoidance and exploration in unknown environments, while the artificial strategy ensures the reasonable selection of global exploration points. This design not only ensures the flexibility and environmental adaptability of the strategy but also effectively avoids omission and repeated exploration, improving the comprehensiveness and stability of exploration.

[0173] 3) During the process of local environmental exploration, the method of the present disclosure marks the area that cannot be accurately sensed due to being blocked by obstacles within the sensing range of the robot sensor as an obstacle shadow area, effectively reducing the logical misjudgment caused by unknowable factors due to obstacle occlusion. At the same time, a convolutional layer and a pooling layer are introduced into the neural network structure to facilitate the automatic extraction of environmental features, thereby improving the accuracy of environmental exploration and enhancing the adaptability to complex unknown environments.

[0174] 4) The method of the present disclosure records the environmental exploration process through trajectory points, matches the nearest trajectory point for each candidate point, and forms a candidate point - trajectory point pair. After selecting the candidate point, the robot only needs to move along the historical trajectory to the target point without an additional path planning module. This method simplifies path planning, reduces computational overhead, realizes fast and accurate navigation using the existing trajectory, improves the exploration efficiency and system response speed, and enhances the stability and reliability.

[0175] The embodiment of the present disclosure also proposes an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to execute the above-mentioned autonomous environmental exploration method of a mobile robot in an unknown environment. Among them, the electronic device can be provided as a terminal, a server, or other forms of devices.

[0176] The embodiment of the present disclosure also proposes a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned autonomous environmental exploration method of a mobile robot in an unknown environment is implemented. The computer-readable storage medium can be a non-volatile computer-readable storage medium.

[0177] Those skilled in the art can understand that in the autonomous environment exploration method and system of a mobile robot in the above-mentioned unknown environment of the specific implementation manner, the writing order of each step does not mean a strict execution order and does not impose any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, and the module, the segment of a program, or the part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the block may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0179] The various embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A method for autonomous environment exploration of a mobile robot in an unknown environment, characterized in that: The steps include: S10, obtaining the current position of the robot and the sensing range of the sensor, and constructing a required environment map, wherein the environment map includes a global real map, a global exploration map, and a local exploration map; S20. Based on the local exploration map, a local environment exploration network is constructed; S30, storing multiple quintuple data obtained by the interaction between the robot and the environment into an experience pool, and using the multiple quintuple data in the experience pool to train a local environment exploration network, wherein the multiple quintuple data include an execution action, a local map before and after the execution of the action, an action reward, and an end flag; S40, using the trained local environment exploration network to perform local environment exploration, and recording trajectory points until the local environment exploration is completed; S50, using a random algorithm to filter out nodes at the junction of an explored area and an unexplored area from the global exploration map, clustering the nodes using a clustering algorithm, and dividing them into several different clusters, and calculating and filtering to obtain the center point of each cluster; S60, screening out candidate points from the central point, and determining the number of candidate points; S70, calculating the distance between each candidate point and all trajectory points, finding the trajectory point with the smallest distance to each candidate point, and forming a candidate point-trajectory point pair; S80, calculating the attraction of the candidate point and the repulsion of the trajectory point in each candidate point-trajectory point pair, and selecting the candidate point-trajectory point pair with the largest resultant force after the attraction and repulsion interact with each other; S90, moving the robot along the recorded trajectory points to the trajectory points in the candidate point-trajectory point pair selected in step S80, and returning to step S40 until the environment exploration is completed.

2. The method according to claim 1, characterized in that The construction of the local environment exploration network includes constructing a reward function based on path length, collision and exploration area increase, and the reward function is expressed as: r t ＝r explore +r dis +r collision In the formula, r explore For exploration rewards, r dis is the path length reward, r collision For collision rewards; in, r explore =c1·ΔA explore , r dis =-c2·||P t+1 -P t ||, In the formula, ΔA explore is the increase in the area of ​​the explored region from time t to time t+1, c1 is the weight coefficient of the exploration reward, P t+1 and P t They represent the positions of the robot at time t+1 and time t respectively, c2 is the weight coefficient of the path length reward, which is used to control the influence of the path length on the total reward, and c3 is a constant used to represent the collision reward (c3>0).

3. The method according to claim 1, characterized in that: The multiple five-tuple data obtained by the robot interacting with the environment are stored in the experience pool, and the multiple five-tuple data are: exp t =(s t ,a t ,s t+1 ,r t ,done), In the formula, s t is the local map observed by the agent at time t, a t In the observed state s t The action output by the neural network and executed by the agent, s t+1 Execute action a for the agent t The local map observed later, r t is the action reward of the evaluation function for the output action of the neural network, and done is the sign of whether the task is completed.

4. The method according to claim 1, characterized in that: The center point of each cluster is obtained by calculating the average coordinates of all points in each cluster, expressed as: In the formula, x i and i Cluster C k Interior point p i The coordinates of , m is the number of points in the cluster.

5. The method according to any one of claims 1 or 4, characterized in that: Filter candidate points from the center point, specifically: Taking the center point as the center, a square area of ​​a certain size is intercepted from the global exploration map as a local map, the ratio Ue of the number of grids in the unexplored area to the total number of grids in the local map is calculated, and the center point with Ue>δ is screened out as a candidate point.

6. The method according to claim 5, characterized in that Determine the number of candidate points, specifically: determine whether the number of candidate points is 0, if not 0, continue to execute step S70, if it is 0, there is no node at the junction of the explored area and the unexplored area, and the environment exploration is completed.

7. The method according to claim 1, characterized in that The gravitational force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair are calculated. The gravitational force calculation formula of the candidate point is: In the formula, p represents the position of the candidate point, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, ξ represents the gravitational field coefficient, which determines the size of the gravitational field; The repulsive force calculation formula of the trajectory point is: Where p1 represents the position of the trajectory point, p c represents the current position of the robot, d(p1, p c ) indicates that the robot moves from the trajectory point p1 along the historical trajectory to the current position p of the robot. c The distance that needs to be traveled, η represents the repulsive field coefficient, which determines the size of the repulsive field.

8. An autonomous environment exploration system for a mobile robot in an unknown environment, characterized in that: include: A robot position and environment map acquisition module obtains the robot's current position and the sensor's sensing range, and constructs the required environment map, wherein the environment map includes a global real map, a global exploration map, and a local exploration map; The local environment exploration network construction module builds a local environment exploration network based on the local exploration map; A local environment exploration network training module stores multiple five-tuple data obtained by the robot interacting with the environment into an experience pool, and uses the multiple five-tuple data in the experience pool to train the local environment exploration network, wherein the multiple five-tuple data include an execution action, a local map before and after the execution of the action, an action reward, and an end flag; The local environment exploration module uses the trained local environment exploration network to perform local environment exploration and record trajectory points until the local environment exploration is completed; The center point calculation module uses a random algorithm to filter out nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes through a clustering algorithm, and divides them into several different clusters. The center point of each cluster is calculated and filtered: A candidate point screening module is used to screen candidate points from the central point and determine the number of candidate points; The candidate point-trajectory point pair acquisition module calculates the distance between each candidate point and all trajectory points, finds the trajectory point with the smallest distance to each candidate point, and forms a candidate point-trajectory point pair; A candidate point-trajectory point pair acquisition module with the largest resultant force calculates the attraction of the candidate point and the repulsion of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the largest resultant force after the interaction of the attraction and repulsion; The environment exploration module enables the robot to move along the recorded trajectory points to the trajectory points with the largest combined force, and return to the local environment exploration module to conduct local environment exploration until the environment exploration is completed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for autonomous environment exploration of a mobile robot in an unknown environment as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for autonomous environment exploration of a mobile robot in an unknown environment as described in any one of claims 1 to 7 is implemented.

Images