A Method and System for Autonomous Environment Exploration of Mobile Robots in Unknown Environments

Abstract

This disclosure relates to a method and system for autonomous environment exploration of a mobile robot in an unknown environment. The method includes: acquiring the robot's current position and constructing a required environmental map; training a local environment exploration network through robot-environment interaction; using the trained local environment exploration network to explore the local environment and record trajectory points until the local environment exploration is completed; using a random algorithm to select the boundary points between explored and unexplored areas, calculating the center points, and selecting candidate points from the center points; calculating the trajectory points with the smallest distance to each candidate point to form candidate point-trajectory point pairs; calculating the attractive force of the candidate points and the repulsive force of the trajectory points, and selecting the candidate point-trajectory point pair with the largest resultant force; the robot moving along the historical trajectory to the trajectory point with the largest resultant force and returning to the local environment exploration process until the environment exploration is completed. This method enables robots to conduct rapid and efficient autonomous exploration in unknown and complex environments.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of intelligent robot environment exploration, in particular to a mobile robot autonomous environment exploration method and system in unknown environment. BACKGROUND

[0002] With the rapid development of technology, intelligent mobile robots have become an indispensable key technology in many industries, widely used in industrial production, logistics distribution, medical health, military defense and other fields. In disaster relief, city inspection, intelligent agriculture, resource exploration and other tasks, intelligent mobile robots have shown great potential and value. In these applications, the autonomous navigation and environment perception technology of the robot plays a crucial role. It not only can effectively reduce the labor burden, improve work efficiency and safety, but more importantly, it can replace humans to perform complex and high-risk tasks in extreme or dangerous environments, such as work in harsh conditions of high temperature, high radiation and high pressure.

[0003] Autonomous exploration technology in unknown environment refers to the process of gradually acquiring information about unknown areas through self-perception and decision-making when the robot or agent lacks complete environmental information. This technology is widely used in indoor and outdoor robot navigation, unmanned driving, disaster rescue and other fields. Traditional path planning methods usually rely on known maps, but in actual scenarios, robots often need to deal with dynamic changes, complex obstacles and environments that cannot be known in advance. Therefore, autonomous exploration technology must combine environment perception, real-time map updating and planning strategies to achieve the goal from the initial state to full exploration. Common autonomous exploration algorithms include artificial potential field method, boundary method, sampling method and graph search method, etc. In recent years, with the development of neural networks, deep reinforcement learning technology has also been gradually applied to autonomous exploration in unknown environments. However, when facing large-scale and complex unknown environments, these methods still face problems such as repeated exploration, incomplete environment coverage, and easy to fall into local optimum, and there is still room for improvement in terms of environment exploration efficiency and coverage. SUMMARY

[0004] To solve the problems of repeated exploration, incomplete environment coverage, and easy to fall into local optimum when traditional robot exploration methods face dynamic changes, complex obstacles, and environments that cannot be known in advance, the present disclosure proposes a mobile robot autonomous environment exploration method in unknown environment to solve the above problems.

[0005] According to an aspect of the present disclosure, a mobile robot autonomous environment exploration method in unknown environment is provided, comprising:

[0006] S10, acquire a current position of the robot and a sensing range of a sensor, and construct a required environment map, wherein the environment map comprises a global real map, a global exploration map and a local exploration map;

[0007] S20, construct a local environment exploration network based on the local exploration map;

[0008] S30, store a plurality of five-tuple data obtained by the robot interacting with the environment in an experience pool, and train the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data comprises an executed action, local maps before and after the action is executed, an action reward and an end flag;

[0009] S40, perform local environment exploration using the trained local environment exploration network, and record a trajectory point until the local environment exploration ends;

[0010] S50, use a random algorithm to screen out nodes at a junction of an explored area and an unexplored area in the global exploration map, cluster the nodes by a clustering algorithm, divide the nodes into several different clusters, and calculate and screen out a center point of each cluster;

[0011] S60, screen out candidate points from the center points, and determine a number of the candidate points;

[0012] S70, calculate a distance between each candidate point and all trajectory points, find out a trajectory point with a minimum distance to each candidate point, and form a candidate point-trajectory point pair;

[0013] S80, calculate an attractive force of the candidate point and a repulsive force of the trajectory point in each candidate point-trajectory point pair, and select a candidate point-trajectory point pair with a maximum resultant force after interaction of the attractive force and the repulsive force;

[0014] S90, make the robot move to the trajectory point in the selected candidate point-trajectory point pair in step S80 along the recorded trajectory point, and return to step S40 until the environment exploration is completed.

[0015] Preferably, the constructing the local environment exploration network comprises constructing a reward function based on a path length, a collision and an increase amount of an exploration area, and the reward function is expressed as:

[0016] r t =r explore +r dis +r collision ,

[0017] In the formula, r explore is an exploration reward, r dis is a path length reward, and r collision is a collision reward.

[0018] wherein,

[0019] r explore = c1·ΔA explore ,

[0020] r dis = -c2·||P t+1 -P t ||,

[0021]

[0022] wherein, ΔA explore is the increase of the explored area from time t to time t+1, c1 is the weight coefficient of the exploration reward, P t+1 and P t represent the positions of the robot at time t+1 and time t respectively, c2 is the weight coefficient of the path length reward, used to control the influence of the path length on the total reward, and c3 is a constant (c3>0) used to represent the collision reward.

[0023] Preferably, the plurality of five-tuple data obtained by the robot interacting with the environment is stored in the experience pool, and the plurality of five-tuple data is:

[0024] exp t = (s t , a t , s t+1 , r t , done),

[0025] wherein, s t is the local map observed by the agent at time t, a t is the action output by the neural network under the observation state s t and executed by the agent, s t+1 is the local map observed after the agent executes the action a t , r t is the action reward of the evaluation function on the action output by the neural network, and done is a flag indicating whether the task is completed.

[0026] Preferably, the center point of each cluster is obtained by calculating the average coordinates of all points in each cluster, and is represented as:

[0027]

[0028] wherein, x i and y i are the coordinates of the point p k in the cluster C i , and m is the number of points in the cluster.

[0029] Preferably, the candidate points are selected from the center points, and specifically, the candidate points are selected according to the following formula:

[0030] A square region of a certain size is cut from the global exploration map as a local map with the center point as the center, the proportion Ue of the number of unexplored region grids in the local map to the total number of grids is calculated, and the center points with Ue>δ are screened out as candidate points.

[0031] Preferably, the number of candidate points is determined, specifically: it is judged whether the number of candidate points is 0, if not, step S70 is continued to be executed, and if yes, there is no node at the junction of the explored region and the unexplored region, and the environment exploration is completed.

[0032] Preferably, the attractive force of the candidate point in each candidate point-trail point pair and the repulsive force of the trail point are calculated, the attractive force calculation formula of the candidate point is:

[0033]

[0034] In the formula, p represents the candidate point position, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, and ξ represents the attractive force field coefficient, which determines the size of the attractive force field;

[0035] The repulsive force calculation formula of the trail point is:

[0036]

[0037] In the formula, p1 represents the trail point position, p c represents the current position of the robot, d(p1, p c ) represents the distance from the trail point p1 to the current position p c of the robot, and η represents the repulsive force field coefficient, which determines the size of the repulsive force field.

[0038] According to an aspect of the present disclosure, a mobile robot autonomous environment exploration system in an unknown environment is provided, comprising:

[0039] A robot position and environment map acquisition module acquires the current position of the robot and the sensing range of the sensor, and constructs a required environment map, wherein the environment map comprises a global real map, a global exploration map and a local exploration map;

[0040] A local environment exploration network construction module constructs a local environment exploration network based on the local exploration map;

[0041] The local environment exploration network training module stores a plurality of five-tuple data obtained by the robot interacting with the environment in an experience pool, and trains the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data comprises an executed action, a local map before and after the action, an action reward, and an end flag;

[0042] The local environment exploration module uses the trained local environment exploration network to perform local environment exploration and records trajectory points until the local environment exploration ends.

[0043] The center point calculation module uses a random algorithm to filter out nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes by a clustering algorithm, divides them into several different clusters, and calculates and filters out the center point of each cluster.

[0044] The candidate point filtering module filters out candidate points from the center points and determines the number of candidate points.

[0045] The candidate point-trajectory point pair acquisition module calculates the distance between each candidate point and all trajectory points, finds the trajectory point with the smallest distance to each candidate point, and forms a candidate point-trajectory point pair.

[0046] The candidate point-trajectory point pair with the largest resultant force acquisition module calculates the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the largest resultant force after the interaction of the attractive force and the repulsive force.

[0047] The environment exploration module causes the robot to move to the trajectory point with the largest resultant force along the recorded trajectory points, returns to the local environment exploration module for local environment exploration, and continues until the environment exploration is completed.

[0048] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-mentioned method for autonomous environment exploration of a mobile robot in an unknown environment.

[0049] According to an aspect of the present disclosure, a computer-readable storage medium having computer program instructions stored thereon is provided, wherein the computer program instructions are executed by a processor to implement the above-mentioned method for autonomous environment exploration of a mobile robot in an unknown environment.

[0050] Compared with the prior art, the present disclosure has the following beneficial effects:

[0051] 1) The global exploration strategy designed in the present disclosure determines the boundary points through clustering and selects the target points using the artificial potential field method, which together constitute a perfect global point selection strategy, avoiding the robot relying solely on local information to fall into local optimization and ensuring the integrity of environmental exploration. At the same time, clustering reduces the number of target points, reduces the computational load, and reduces the demand for device computing power of the algorithm.

[0052] 2) The method of the present disclosure uses reinforcement learning for obstacle avoidance and exploration in local exploration, and uses artificial strategy for point selection at the global level. Reinforcement learning optimizes the obstacle avoidance and exploration efficiency in unknown environments, while the artificial strategy ensures the reasonable selection of global exploration points. This design not only guarantees the flexibility and environmental adaptability of the strategy, but also effectively avoids missing and repeated exploration, improving the comprehensiveness and stability of exploration.

[0053] 3) In the process of local environmental exploration, the present disclosure method marks the area that cannot be accurately perceived by the robot sensor within the perception range due to obstruction by obstacles as an obstacle shadow area, effectively reducing the logical misjudgment caused by unknown factors due to obstruction by obstacles. At the same time, convolutional and pooling layers are introduced into the neural network structure, which facilitates automatic extraction of environmental features, thereby improving the accuracy of environmental exploration and enhancing the adaptability to complex unknown environments.

[0054] 4) The method of the present disclosure records the environmental exploration process through trajectory points, matches the nearest trajectory point for each candidate point, and forms a candidate point-trajectory point pair. After selecting the candidate point, the robot only needs to move to the target point along the historical trajectory, without the need for an additional path planning module. This method simplifies path planning, reduces computational overhead, uses existing trajectories to achieve fast and accurate navigation, improves exploration efficiency and system response speed, and improves stability and reliability.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure.

[0056] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0057] The drawings herein are incorporated into the specification and form part of the specification, which illustrate embodiments consistent with the present disclosure, and together with the specification serve to explain the technical solutions of the present disclosure.

[0058] Figure 1 A flowchart of a mobile robot autonomous environmental exploration method in an unknown environment is shown;

[0059] Figure 2A flow chart of the autonomous exploration method of the mobile robot in the unknown environment in the present disclosure is shown.

[0060] Figure 3 A schematic diagram of the global real map, the global exploration map and the local exploration map in the embodiment of the present disclosure is shown.

[0061] Figure 4 A schematic diagram of the network structure of reinforcement learning in the local exploration map in the embodiment of the present disclosure is shown.

[0062] Figure 5 A schematic diagram of the system architecture of the local environment exploration algorithm based on reinforcement learning in the embodiment of the present disclosure is shown.

[0063] Figure 6 A schematic diagram of the exploration result of the robot in the unknown environment in the embodiment of the present disclosure is shown.

[0064] Figure 7 A structural block diagram of a mobile robot autonomous environment exploration system in an unknown environment is shown. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings represent functionally the same or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0066] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0067] The term "and / or", merely an associative relationship of the associated objects described herein, can represent the existence of three kinds of relationships, for example, A and / or B, can represent: the existence of A alone, the existence of A and B, and the existence of B alone. In addition, the term "at least one" herein means any one of the plurality or any combination of at least two of the plurality, for example, including at least one of A, B, C, can mean including any one or more elements selected from the set consisting of A, B and C.

[0068] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail, in order to highlight the main idea of the present disclosure.

[0069] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0070] Embodiment 1

[0071] Based on the above idea, the present application provides a mobile robot autonomous environment exploration method in an unknown environment. Figure 1 A flowchart of a mobile robot autonomous environment exploration method in an unknown environment is shown. The method comprises:

[0072] S10, acquiring a current position of a robot and a sensing range of a sensor, and constructing a required environment map, wherein the environment map comprises a global real map, a global exploration map and a local exploration map;

[0073] S20, constructing a local environment exploration network based on the local exploration map;

[0074] S30, storing a plurality of five-tuple data obtained by the robot interacting with the environment in an experience pool, and training the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data comprises an executed action, a local map before and after the action is executed, an action reward and an end flag;

[0075] S40, performing local environment exploration using the trained local environment exploration network, and recording trajectory points until the local environment exploration ends;

[0076] S50, using a randomness algorithm to screen out nodes at the junction of an explored area and an unexplored area in the global exploration map, clustering the nodes by a clustering algorithm, and dividing them into several different clusters, and calculating and screening to obtain a center point of each cluster;

[0077] S60, screening candidate points from the center points, and determining the number of candidate points;

[0078] S70, calculating the distance between each candidate point and all trajectory points, finding out the trajectory point with the minimum distance to each candidate point, and constituting a candidate point-trajectory point pair;

[0079] S80, calculating the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selecting the candidate point-trajectory point pair with the maximum resultant force after the interaction of the attractive force and the repulsive force;

[0080] S90, moving the robot along the recorded trajectory points to the trajectory point in the candidate point-trajectory point pair selected in step S80, returning to step S40 until the environment exploration is completed.

[0081] The embodiment of the present disclosure provides a mobile robot autonomous environment exploration method in an unknown environment, and a flowchart of the mobile robot autonomous environment exploration method is as shown in the figure. Figure 2 The embodiment of the present disclosure provides a mobile robot autonomous environment exploration method in an unknown environment, and a flowchart of the mobile robot autonomous environment exploration method is as shown in the figure.

[0082] S10, obtaining the current position of the robot and the sensing range of the sensor, and constructing the required environment map, wherein the environment map includes a global real map, a global exploration map and a local exploration map.

[0083] In the embodiment, a simulation environment of a space to be explored and a robot model are built, the current position of the robot in the space to be explored and radar data are obtained, and the required environment map is constructed based on this, including a global real map, a global exploration map and a local exploration map. Figure 3 The update process for expressing three environment maps is as follows: Figure 3 In (a), (b) and (c), the construction processes of the global real map, the global exploration map and the local exploration map are respectively.

[0084] The global exploration map refers to an environment map for exploring the entire scene. As a core tool when the robot explores the entire scene, the global exploration map is continuously updated in the process of environment exploration and guides the robot to perform subsequent exploration actions. The global exploration map update method is as follows: assuming that the actual sensing range of the laser radar carried by the robot is L, a value l0 smaller than L is set as the current exploration range of the robot. In the exploration process, if the robot does not detect an obstacle, the space within the range from the position of the robot to L-l0 is marked as an explored area, and if an obstacle is detected, the position of the obstacle is updated as an obstacle area, and the space between the position of the robot and the obstacle is updated as an explored area. Specifically, as shown in (b) of the figure, Figure 3 In (b), the white part represents the explored area, the gray part represents the unexplored area, and the black part represents the edge of the obstacle.

[0085] The global real map is a map drawn by the robot in the exploration process to record the real features of the environment. The main difference between it and the global exploration map is that the sensing range of the laser radar used when drawing the map is different, and the specific difference is as follows: Figure 3As shown in (a), to ensure the accuracy and integrity of the exploration mapping and effectively reduce the interference of noise information in the environment, the laser radar detection distance used by the global exploration map when updating is intentionally set to be slightly smaller than the real perception range. Therefore, when the robot completes the exploration task using the global exploration map, the global real map drawn using the real laser radar data often exhibits higher exploration coverage. In particular, the global real environment map mentioned in this embodiment is strictly based on the actual perception ability of the laser radar carried by the robot, and is mainly used for performance analysis to provide a more accurate reference for the exploration actions of the robot.

[0086] The local exploration map is a map representation form used by the robot during autonomous exploration of an unknown environment to focus on the surrounding area and support immediate decision-making, and is mainly used for detailed exploration of the local environment. In terms of space, the local exploration map defines a limited but sufficient area around the current position of the robot to deal with sudden situations at close range and support detailed perception and motion planning. The size of this area is flexibly adjusted according to the effective range of the laser radar carried by the robot, the motion flexibility, and the importance of the task, and usually covers a surrounding range of several meters to several tens of meters. The local exploration map is obtained as follows: a square area of a certain range is extracted from the global exploration map with the current position of the robot as the center, and is updated by real-time sensor data. The specific updating method is as follows: the obstacles and the part of the area behind them perceived by the robot are marked as obstacle-occluded areas, and the attributes of other areas remain unchanged, as shown in (b). The local exploration map is updated in real time based on the real-time sensor data of the robot, and the size of the local exploration map is dynamically adjusted according to the real-time sensor data of the robot. Figure 3 As shown in (c), this is because the uncertainty of the area behind the obstacle may affect the accuracy of the decision-making during the robot's exploration of the environment. In particular, due to the different relative positions of the robot and the obstacle, the occlusion relationship of the obstacle will change accordingly, so each position has a unique local exploration map. It should be noted that the side length of the local map should be greater than the detection range of the robot's laser radar to ensure sufficient perception of the surrounding environment.

[0087] S20, based on the local exploration map, a local environment exploration network is constructed.

[0088] In the local map exploration problem, the local exploration map of the current position of the robot needs to be obtained, and based on this local exploration map, the local exploration problem can be simplified as an optimization problem: selecting a target point P for the robot, so that when the robot moves from the current position to P, the newly added unexplored area is maximized, while satisfying the obstacle avoidance constraint and the path feasibility.

[0089] The target point P in the local environment exploration is represented as:

[0090]

[0091] wherein, denotes the set of all possible target points in the local map, T is the time instant at which the robot is exploring the local map, p denotes the current position of the robot, and g(p) is a function that calculates the area of the newly explored region in the process of the robot moving from the current position to the target point p.

[0092] In order to solve the robot local map exploration problem using reinforcement learning, the exploration process needs to be modeled, and a corresponding Markov Decision Process (MDP) model needs to be constructed. It should be noted that due to the limitation of the detection range of the laser radar, the robot can usually only observe part of the environmental information around itself, and cannot fully perceive the state of the entire environment, therefore, the autonomous exploration process of the local map of the robot is more suitable to be described by a Partially Observable Markov Decision Process (POMDP).

[0093] The model of the Partially Observable Markov Decision Process constructed for the robot local map exploration problem can be represented by a seven-tuple (S, A, T, R, O, Z, γ). Wherein: S is the set of all possible states of the environment; A is the set of all actions that the robot can execute; S is the state transition equation, T(s t+1 |s t , a) represents the probability that the environment will be transferred from state s t to state s t+1 after the robot executes action a; S x A→R represents the immediate reward given by the environment after the robot takes action a in the current state s; O represents the set of observations, i.e. the finite set of all observations that the robot's sensors can observe; Z represents the observation probability distribution, Z(o|s t+1 , a t ) represents the probability that the robot can observe o after executing action a t and transferring from state s t+1 to state s

[0094] Based on the design of the above seven-tuple, the basic process of the agent in the local environment exploration can be described as follows: at time instant t, the state of the agent (Agent) is s t . According to the policy function π θ (a t |s t ), the agent selects action a t , and then according to the state transition function T(s t+1 |s t , a t ), the state of the environment is transferred to the next state s t+1 . At the same time, the agent according to the observation probability distribution Z(o|s t+1, a t ) obtaining an environment observation o∈Z and obtaining an immediate reward r t (s t , a t ). This interaction process is repeated until the local exploration task is completed.

[0095] Further, the SAC algorithm is used to construct the network architecture, including the design of the state space, action space, reward function and network structure. First, the local exploration map is used to construct the state space of the robot's local environment exploration problem. Since the reinforcement learning algorithm used can generate continuous actions based on probability distribution, the action space is set as a two-dimensional continuous action space. The target point output by the network is represented in polar coordinates relative to the current position of the robot, with the current position of the robot as the origin and the orientation of the robot as the reference direction of the coordinate axis. The first dimension of the action space represents the angle of the target point relative to the orientation of the robot, which can be set to (θ min , θ max ); the second dimension represents the distance between the target point and the robot, which can be set to (0, l0). The specific value of l0 can be adjusted according to the actual problem requirements, and is usually not more than the maximum detection range of the laser radar carried by the robot to ensure the safety of the robot. Therefore, the action vector of the robot can be represented as (θ, l), and this polar coordinate representation clearly describes the orientation and distance of the target point relative to the robot, which helps to improve the efficiency and accuracy of the algorithm in target point planning.

[0096] In order to improve the numerical stability of the model, speed up the convergence speed, and improve the training effect, thereby improving the generality and portability of the method, normalization processing is performed on each dimension of the action space, and the normalization formula is:

[0097]

[0098] where θ norm is the normalized angle value, ranging between -1 and 1, and l norm is the normalized distance value, ranging between 0 and 1.

[0099] The reward function is a key component of reinforcement learning, which serves as a feedback mechanism for the agent, defining the quality of the agent's behavior in different states, thereby driving the agent to continuously optimize its decision-making strategy and accelerating the training process. In the local map exploration task, the reward function needs to provide environmental feedback to the robot, guiding it to explore as much of the environment as possible in each decision while minimizing the length of the exploration path. In addition, considering the safety of the robot during travel, the selection of actions should avoid known obstacle areas as much as possible. Therefore, three basic rewards are introduced for the robot: exploration reward r explore, path length reward r dis , collision reward r collision .

[0100] Specifically, for time t, the action reward r t obtained by the mobile robot is:

[0101] r t = r explore + r dis + r collision ,

[0102] Exploration reward r explore is defined as the reward given according to the increase in the area of the explored region in the global exploration map after the robot performs an action. The global exploration map is transformed accordingly, and the explored region and the unexplored region are represented by binary matrix E t (i, j), with a size of N x M, where the explored region is represented by 1 and the unexplored region is represented by 0. Specifically, at time t, the exploration state of grid (i, j) can be represented by variable E t (i, j), where E t (i, j) = 1 if it has been explored, otherwise E t (i, j) = 0. Then at time t, the number of grids A explore，t of the explored region can be represented as:

[0103]

[0104] The increase in the area of the explored region ΔA explore from time t to time t + 1 is represented as:

[0105]

[0106] Exploration reward r explore is represented as:

[0107] r explore = c1·ΔA explore ,

[0108] where c1 is the weight coefficient of the exploration reward.

[0109] Path length reward r dis is defined as the reward given according to the length of the path traveled by the robot. In order to encourage the robot to explore as much as possible with the shortest path, a negative reward can be set for the path length, that is, a small penalty will be generated for each action, and this penalty should not be too large to avoid excessive inhibition of the exploration behavior of the robot. The calculation formula of the path length reward is represented as:

[0110] r dis = -c2·||Pt+1 -P t ||,

[0111] In the formula, P t+1 and P t respectively represent the position of the robot at t+1 time and t time, c2 is a weight coefficient of the path length reward, used to control the influence of the path length on the total reward.

[0112] The collision reward r collision is defined as the reward given according to whether the robot collides or not. Specifically, when the robot collides, the reward is negative; if the robot does not collide, the reward is zero, expressed as:

[0113]

[0114] In the formula, c3 is a constant used to represent the collision reward (c3>0).

[0115] As shown in Figure 4 , since the input of the neural network is the local exploration map, in order to extract the planar structure information of the environment, a convolution layer is introduced in the network, and a Relu function is activated after each convolution layer to introduce a nonlinear transformation. In addition, a max pooling layer is also added after the activation function to reduce the size of the feature map, reduce the computational overhead and alleviate overfitting, while retaining key information. Finally, the features are further processed through a fully connected layer, and the final result is given at the output layer.

[0116] S30, store a plurality of five-tuple data obtained by interacting the robot with the environment in an experience pool, and train the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data includes an executed action, a local map before and after the action is executed, an action reward, and an end flag.

[0117] In this embodiment, a plurality of five-tuple data obtained by interacting the robot with the environment is stored in an experience pool for agent training. To ensure the diversity of experience and avoid the model falling into local optimum, the robot randomly takes actions to obtain experience data in the early stage, and accumulates experience data while learning in the later stage. The plurality of five-tuple data is:

[0118] exp t =(s t ,a t ,s t+1 ,r t ,done),

[0119] In the formula, s t is a local map observed by the agent at t time, a t is an action taken by the agent at the observation state s tan action outputted by the neural network and executed by the agent, s t+1 execute the action a t a local map observed after, r t the action reward of the evaluation function for the action outputted by the neural network, and done is a flag indicating whether the task is finished. These data are the basis for training of the reinforcement learning, through multiple interactions and accumulation of experience, the agent learns how to select an action policy that can maximize the cumulative reward, the interaction of the agent with the environment and the training process of the neural network are as shown in Figure 5

[0120] Further, the local environment exploration network is trained using multiple five-tuple data in the experience pool, and the optimization target of the local environment exploration network is:

[0121]

[0122] H(π(·|s t ))=E[-logπ(·|s t )],

[0123] In the formula, (s t , a t ) ~ π(·|s t ) represents a state-action sequence at time t generated under the policy function π, r(s t , a t ) is the immediate reward corresponding to the state s t and the action a t at time t, H(π(·|s t )) represents the entropy value of the policy function π at state s t , the higher the entropy value, the greater the randomness of the policy, α is a temperature coefficient for controlling the importance of the entropy term, thereby achieving a balance between exploration and reward, γ is a discount factor for balancing the importance of current rewards and future rewards, and its value range is [0, 1].

[0124] S40, local environment exploration is performed using the trained local environment exploration network, and trajectory points are recorded until the local environment exploration ends.

[0125] In this embodiment, the trained local environment exploration network is transplanted to the robot for local environment exploration, the environment map is gradually updated, and each trajectory point outputted by the neural network is recorded until the local environment exploration ends.

[0126] S50, using a randomness algorithm to screen out nodes at the junction of the explored area and the unexplored area from the global exploration map, clustering the nodes by a clustering algorithm, dividing them into several different clusters, and calculating and screening the center points of each cluster.​

[0127] In this embodiment, when the local environment exploration ends, the RRT algorithm is used to select the nodes at the junction of the explored area and the unexplored area, i.e., the boundary points, from the global exploration map. First, several points in the trajectory recorded in the local exploration process are selected as the initial growth points, which are the root nodes of the RRT tree, and all the tree expansion steps, directions, and other parameters are shared. The step controls the distance of each expansion, which is usually set to a fixed value Δx. When expanding each time, a target point q rand is randomly selected from the current tree, which can be located in the explored area or the unexplored area. After determining q rand , the algorithm finds the node q near closest to the target point q rand from the existing tree nodes, and expands a new node through this node. This is achieved by calculating the distance between each node and the target point:

[0128]

[0129] where (x rand , y current ) and (x current , y near ) represent the coordinates of the target point and the current node, respectively.

[0130] Next, the tree expands from the node q rand to the target point q new , generating a new node q new :

[0131]

[0132] In addition, the new node q new must pass collision detection to ensure that it is not located in the obstacle area, which can be determined by querying the grid map to determine whether the node is located in the space area.

[0133] During the expansion process, the key is how to select the boundary points between the explored area and the unexplored area from the generated RRT tree. Specifically, the boundary points are those nodes located at the junction of the explored area and the unexplored area. Whenever the RRT tree expands to a new node, the algorithm checks the relationship between the node and its adjacent area. Specifically: if the node q new is located in the unexplored area, and some of its adjacent nodes are located in the explored area, then the node q new is a boundary point. As the RRT tree continues to expand, the algorithm repeatedly performs the above steps until enough boundary points are generated or a certain termination condition is reached. Finally, through the expansion of the RRT tree and the boundary determination, we can obtain a set of points representing the boundary between the explored area and the unexplored area.

[0134] The boundary points are clustered using the DBSCAN algorithm, and the boundary points are divided into different clusters, and the center point of each cluster is calculated. The center point of each cluster is obtained by calculating the average coordinates of all points in each cluster, and is represented as:

[0135]

[0136] In the formula, x i and y i are the coordinates of the point p i in the cluster C k , and m is the number of points in the cluster.

[0137] Next, the point closest to the geometric center point in the cluster C k is found, and this point is taken as the center point of the cluster.

[0138] S60, screening candidate points from the center points, and determining the number of candidate points.

[0139] In this embodiment, candidate points are screened from the center points, specifically: taking the center point as the center, a square region of a certain size is intercepted from the global exploration map as a local map, the proportion Ue of the number of unexplored region grids in the local map to the total number of grids is calculated, and the center points with Ue>δ are screened as candidate points, where δ is a threshold value, which is adjusted according to the complexity of the environment.

[0140] Further, the number of candidate points is determined, specifically: judging whether the number of candidate points is 0, if not, continue to execute step S70, if 0, there is no node at the junction of the explored region and the unexplored region, and the environment exploration is completed.

[0141] S70, calculating the distance of each candidate point from all trajectory points, finding the trajectory point with the smallest distance from each candidate point, and constructing a candidate point-trajectory point pair.

[0142] In this embodiment, for each candidate point, the distance from all trajectory points is calculated, and the trajectory point with the smallest distance is found to form a candidate point-trajectory point pair. There are as many candidate point-trajectory point pairs as there are candidate points.

[0143] S80, calculating the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selecting the candidate point-trajectory point pair with the largest resultant force after the interaction of the attractive force and the repulsive force.

[0144] In this embodiment, the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair are calculated, and the numerical sum of the resultant force of the candidate point and the trajectory point is calculated as a whole instead of a vector sum, the calculation formula of the attractive force of the candidate point is as follows:

[0145]

[0146] In the formula, p represents the position of the candidate point, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, ξ represents the coefficient of the attractive force field, and determines the size of the attractive force field. ξ is a value artificially set, and is adjusted according to the size of the map and the density of obstacles in the map. The coefficient is not a fixed value, but is related to the obstacle distribution around the candidate point, and its expression is as follows:

[0147]

[0148] In the formula, N shadow is the number of grids in the shadow occluded area in the local grid map.

[0149] The calculation formula of the repulsive force of the trajectory point is as follows:

[0150]

[0151] In the formula, p1 represents the position of the trajectory point, p c represents the current position of the robot, d(p1, p c ) represents the distance from the trajectory point p1 to the current position p c of the robot along the historical trajectory, and η represents the coefficient of the repulsive force field, which determines the size of the repulsive force field. η is a value artificially set, and is adjusted according to the size of the map and the complexity.

[0152] After the attractive force of the candidate point and the repulsive force of the trajectory point are obtained, the resultant force of the attractive force of the candidate point and the repulsive force of the trajectory point is calculated. It should be noted that the calculation of the resultant force only considers the numerical superposition, and does not consider the direction. The calculation formula of the resultant force is as follows:

[0153] U t (p cand , p traj ) = U a (p cand ) + U r (p traj ),

[0154] In the formula, p cand , p traj respectively represent the candidate point and the trajectory point corresponding to the candidate point with the smallest distance, U a (p cand ), U r (ptraj ) represent the attraction of the candidate point and the repulsion of the trajectory point with the smallest distance to it, respectively.

[0155] S90, move the robot along the recorded trajectory points to the trajectory point in the candidate point-trajectory point pair selected in step S80, and return to step S40 until the environmental exploration is completed.

[0156] by Figure 6 Taking the multi-room simulation environment with obstacles built in Webots as shown in Figure (a), the robot is explored using the method described in this embodiment. The robot's exploration trajectory is as follows: Figure 6 As shown in (b), the real-world environment map constructed after the exploration is completed is as follows: Figure 6 As shown in (c), it can be observed that the entire environment has been fully explored.

[0157] This embodiment provides a method for autonomous exploration of unknown environments that is computationally efficient, real-time, and highly effective. By training local exploration strategies through reinforcement learning and combining them with artificial potential field methods, the robot can achieve rapid, efficient, and complete autonomous exploration in unknown and complex environments.

[0158] Example 2

[0159] As another aspect of the embodiments of this disclosure, a mobile robot autonomous environment exploration system 100 in an unknown environment is also provided, such as... Figure 7 As shown, it includes:

[0160] Robot position and environment map acquisition module 1 acquires the robot's current position and the sensor's perception range, and constructs the required environment map, wherein the environment map includes a global real map, a global exploration map, and a local exploration map;

[0161] Module 2 for building a local environment exploration network: Based on the local exploration map, a local environment exploration network is constructed.

[0162] The local environment exploration network training module 3 stores multiple quintuple data obtained from the robot's interaction with the environment into the experience pool, and uses the multiple quintuple data in the experience pool to train the local environment exploration network. The multiple quintuple data includes the action to be performed, the local map before and after the action, the action reward, and the end flag.

[0163] Local environment exploration module 4 uses a trained local environment exploration network to explore the local environment and record trajectory points until the local environment exploration ends;

[0164] The center point calculation module 5 uses a random algorithm to screen out nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes through a clustering algorithm, divides them into several different clusters, and calculates and screens the center points of each cluster;

[0165] The candidate point screening module 6 screens candidate points from the center points and determines the number of candidate points;

[0166] The candidate point-trajectory point pair acquisition module 7 calculates the distance between each candidate point and all trajectory points, finds the trajectory point with the minimum distance to each candidate point, and constitutes a candidate point-trajectory point pair;

[0167] The candidate point-trajectory point pair with the maximum resultant force acquisition module 8 calculates the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the maximum resultant force after the interaction of the attractive force and the repulsive force;

[0168] The environment exploration module 9 makes the robot move to the trajectory point with the maximum resultant force along the recorded trajectory points, returns to the local environment exploration module for local environment exploration, and continues until the environment exploration is completed.

[0169] Without causing contradictions, the above-mentioned modules in the system of the embodiment of the disclosure can implement any of the embodiments in the above-mentioned method.

[0170] Based on the description of the above-mentioned embodiments, the embodiment of the disclosure can achieve the following technical effects:

[0171] 1) The environment global exploration strategy designed by the disclosure determines the boundary points through clustering, and selects the target points using the artificial potential field method, which combines the two to form a perfect global point selection strategy, avoids the robot from falling into local optimization only relying on local information, and ensures the integrity of the environment exploration. At the same time, clustering reduces the number of target points, reduces the calculation amount, and reduces the demand of the algorithm on the device computing power.

[0172] 2) The method of the disclosure uses reinforcement learning for obstacle avoidance and exploration in local exploration, and uses an artificial strategy for point selection at the global level. Reinforcement learning optimizes the obstacle avoidance and exploration efficiency in the unknown environment, and the artificial strategy ensures the reasonable selection of global exploration points. This design not only guarantees the flexibility and environmental adaptability of the strategy, but also effectively avoids missing and repeated exploration, and improves the comprehensiveness and stability of exploration.

[0173] 3) The method of the present disclosure effectively reduces the logical misjudgment caused by the unknown factors due to the obstruction of obstacles in the local environment exploration process by marking the area that cannot be accurately perceived by the robot sensor within the sensing range due to the obstruction of obstacles as an obstacle shadow area. Meanwhile, the convolution layer and the pooling layer are introduced into the neural network structure, which facilitates the automatic extraction of environmental features, thereby improving the accuracy of environmental exploration and enhancing the adaptability to complex unknown environments.

[0174] 4) The method of the present disclosure records the environmental exploration process by trajectory points, matches the nearest trajectory point for each candidate point to form a candidate point-trajectory point pair. After selecting the candidate point, the robot only needs to move to the target point along the historical trajectory without the need for an additional path planning module. This method simplifies the path planning, reduces the computational overhead, realizes fast and accurate navigation using the existing trajectory, improves the exploration efficiency and system response speed, and improves the stability and reliability.

[0175] The embodiments of the present disclosure also propose an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-mentioned method for autonomous environmental exploration of mobile robots in unknown environments. The electronic device can be provided as a terminal, a server or other forms of devices.

[0176] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above-mentioned method for autonomous environmental exploration of mobile robots in unknown environments. The computer-readable storage medium can be a non-volatile computer-readable storage medium.

[0177] Those skilled in the art can understand that in the above-mentioned method and system for autonomous environmental exploration of mobile robots in unknown environments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible internal logic.

[0178] The computer program product of the second aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to enable a user equipment device to receive a configuration message from a base station, the configuration message comprising an indication of a set of one or more parameters for a first type of hybrid automatic repeat request process, the first type of hybrid automatic repeat request process being associated with a first type of data; and instructions for causing a computer to enable a user equipment device to receive a configuration message from a base station, the configuration message comprising an indication of a set of one or more parameters for a first type of hybrid automatic repeat request process, the first type of hybrid automatic repeat request process being associated with a first type of data.

[0179] Embodiments of the present disclosure have been described above, with the understanding that these embodiments are exemplary only, and are not restrictive, in terms of the disclosed embodiments. Many modifications and variations of the described embodiments are possible, without departing from the scope and spirit of the described embodiments. The choice of words in the specification is intended to best explain the principles of the embodiments, practical application, or technical improvement in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for autonomous environment exploration of a mobile robot in an unknown environment, characterized in that, Comprising the following steps: S10, acquiring the current position of the robot and the sensing range of the sensor, and constructing the required environment map, wherein the environment map comprises a global real map, a global exploration map and a local exploration map; S20, constructing a local environment exploration network based on the local exploration map; the construction of the local environment exploration network comprises constructing a reward function based on path length, collision and exploration area increase, and the reward function is represented as: r t ＝r explore +r dis +r collision ， where r explore is the exploration reward, r dis is the path length reward, and r collision is the collision reward. wherein, r explore = c1- ΔA explore , r dis = -c2·||P t+1 -P t ||, where ΔA explore is the increase of the explored area between time t and time t+1, c1 is a weight coefficient of the exploration reward, P t+1 and P t represent the position of the robot at time t+1 and at time t, respectively, c2 is a weight coefficient of the path length reward, used to control the influence of the path length on the total reward, and c3 is a constant used to represent the collision reward (c3>0). S30, storing a plurality of five-tuple data obtained by interacting the robot with the environment in an experience pool, and training the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data comprises an executed action, local maps before and after the action, an action reward and an end flag; S40, performing local environment exploration using the trained local environment exploration network, and recording trajectory points until the local environment exploration ends; S50, selecting nodes at the junction of the explored area and the unexplored area from the global exploration map using a randomness algorithm, clustering the nodes by a clustering algorithm, and dividing them into several different clusters, and calculating and selecting the center points of each cluster; S60, selecting candidate points from the center points, and determining the number of candidate points; S70, calculating the distance between each candidate point and all trajectory points, finding the trajectory point with the smallest distance to each candidate point, and forming a candidate point-trajectory point pair; S80, calculating the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selecting the candidate point-trajectory point pair with the largest resultant force after the interaction of the attractive force and the repulsive force; S90, making the robot move to the trajectory point in the candidate point-trajectory point pair selected in step S80 along the recorded trajectory points, returning to step S40, and ending the environment exploration.

2. The method of claim 1, wherein, The plurality of five-tuple data obtained by interacting the robot with the environment is stored in an experience pool, and the plurality of five-tuple data is: exp t = (s t , a t , s t+1 , r t , done), where s t is the local map observed by the agent at time t, a t is the action output by the neural network at observation s t and executed by the agent, s t+1 is the local map observed by the agent after executing action a t , r t is the action reward of the action output by the neural network as evaluated by the reward function, and done is a flag indicating whether the task is finished.

3. The method of claim 1, wherein, The center point of each cluster is obtained by calculating the average coordinates of all points in each cluster, and is represented as: where x i and y i are the coordinates of the points p k in the cluster C i and m is the number of points in the cluster.

4. The method according to claim 1 or 3, characterized in that, The candidate points are selected from the center points, specifically: A certain size of square area is intercepted from the global exploration map as a local map with the center point as the center, the proportion Ue of the number of unexplored area grids in the local map to the total number of grids is calculated, and the center points with Ue>δ are selected as candidate points.

5. The method of claim 4, wherein, The number of candidate points is determined, specifically: whether the number of candidate points is 0 is judged, if not 0, step S70 is continuously executed, if 0, there is no node at the junction of the explored area and the unexplored area, and the environment exploration is completed.

6. The method of claim 1, wherein, The attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair are calculated, and the calculation formula of the attractive force of the candidate point is: where p represents the candidate point position, N unexplore is the number of unexplored grids in the local grid map, N total is the total number of grids, and ξ represents the gravitational field coefficient, which determines the size of the gravitational field; The calculation formula of the repulsive force of the trajectory point is: In the formula, p1 represents the position of the trajectory point, p c represents the current position of the robot, d(p1, p c ) represents the distance from the trajectory point p1 to the current position p c of the robot along the historical trajectory, and η represents a repulsive force field coefficient, which determines the size of the repulsive force field.

7. An autonomous environment exploration system for a mobile robot in an unknown environment, characterized in that, Comprise: A robot position and environment map acquisition module acquires the current position of the robot and the sensing range of the sensor, and constructs the required environment map, wherein the environment map comprises a global real map, a global exploration map and a local exploration map; A local environment exploration network construction module constructs a local environment exploration network based on the local exploration map; The constructing the local environment exploration network comprises constructing a reward function based on path length, collision and exploration area increase amount, and the reward function is represented as: r t ＝r explore +r dis +r collision ， where r explore is the exploration reward, r dis is the path length reward, and r collision is the collision reward. wherein, r explore = c1- ΔA explore , r dis = -c2·||P t+1 -P t ||, where ΔA explore is the increase of the explored area between time t and time t+1, c1 is a weight coefficient of the exploration reward, P t+1 and P t represent the position of the robot at time t+1 and at time t, respectively, c2 is a weight coefficient of the path length reward, used to control the influence of the path length on the total reward, and c3 is a constant used to represent the collision reward (c3>0). The local environment exploration network training module stores a plurality of five-tuple data obtained by interacting the robot with the environment in an experience pool, and trains the local environment exploration network using the plurality of five-tuple data in the experience pool, wherein the plurality of five-tuple data comprises an executed action, a local map before and after the action, an action reward and an end flag; The local environment exploration module uses the trained local environment exploration network to perform local environment exploration, and records a trajectory point until the local environment exploration ends; The center point calculation module uses a random algorithm to screen out nodes at the junction of the explored area and the unexplored area from the global exploration map, clusters the nodes by a clustering algorithm, divides them into several different clusters, and calculates and screens the center points of each cluster; The candidate point screening module screens candidate points from the center points and determines the number of candidate points; The candidate point-trajectory point pair acquisition module calculates the distance between each candidate point and all trajectory points, finds the trajectory point with the smallest distance to each candidate point, and constitutes a candidate point-trajectory point pair; The candidate point-trajectory point pair with the largest resultant force acquisition module calculates the attractive force of the candidate point and the repulsive force of the trajectory point in each candidate point-trajectory point pair, and selects the candidate point-trajectory point pair with the largest resultant force after the interaction of the attractive force and the repulsive force; The environment exploration module makes the robot move to the trajectory point with the largest resultant force along the recorded trajectory point, returns to the local environment exploration module for local environment exploration, and continues until the environment exploration is completed.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the unknown environment mobile robot autonomous environment exploration method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the unknown environment mobile robot autonomous environment exploration method of any one of claims 1 to 6.

Images