Mobile robot autonomous exploration navigation method based on deep reinforcement learning

Abstract

A mobile robot autonomous exploration navigation method based on deep reinforcement learning is mainly composed of a data input module, a candidate action point selection module and a candidate target point selection module. An occupation map of the mobile robot, position data information of the mobile robot and surrounding environment data information detected by the sensor are transmitted to the next module through the data input module. In the candidate action point selection module, map construction is performed based on Shannon entropy, a mutual information reward curved surface of an existing occupied map is predicted through a Gaussian process regression model, a prediction result of Gaussian process regression is improved by using a Bayesian optimization technology, and then candidate action points are selected by using a learning-based perception strategy. A perception network model is trained. And finally, selecting an optimal candidate target point from the plurality of candidate action points. And finally, a mobile robot navigation module plans a motion path according to the candidate target points and drives the mobile robot to explore towards the target points.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep reinforcement learning, and specifically to an autonomous exploration and navigation method for mobile robots based on deep reinforcement learning. Background Art

[0002] In an unknown environment, a robot needs to possess the key capabilities of autonomous exploration and mapping. The navigation control methods of traditional mobile robots mainly rely on accurate environmental models or artificially designed control rules. These strategies may exhibit good performance in known environments. However, their limitations gradually become apparent when faced with complex or unknown environments. With the rapid progress of artificial intelligence technology, especially the deep integration of deep learning and reinforcement learning, new paths have been opened up for the autonomous navigation of mobile robots. Deep reinforcement learning combines the advantages of deep neural networks and reinforcement learning.

[0003] In the field of mobile robot navigation, deep reinforcement learning has become a research hotspot and has been widely applied. With the help of deep reinforcement learning, a robot can directly learn navigation strategies from raw sensor data (such as depth images, lidar information, etc.), without relying on an accurate environmental model. The autonomous exploration and navigation method for mobile robots based on deep reinforcement learning can not only achieve efficient navigation in known environments but also conduct autonomous exploration in unknown environments. Through continuous interaction with the environment, the robot can learn navigation strategies such as how to avoid obstacles and how to formulate optimal paths. This ability of autonomous exploration enables the robot to show higher adaptability and robustness when faced with complex and changing environments.

[0004] Aiming at the deficiencies of existing autonomous exploration and navigation methods for mobile robots, the present invention proposes an autonomous exploration and navigation method for mobile robots based on deep reinforcement learning.

[0005] The differences compared with the prior art are as follows:

[0006] Technical comparison with patent CN119200601A "An autonomous exploration method for mobile robots based on deep reinforcement learning"

[0007] In patent CN119200601A, lidar is used to collect environmental data, the information map is sparsified by the bidirectional A* algorithm, and the target path points are generated by the reinforcement learning policy network. This method emphasizes model-free deep reinforcement learning technology and can learn exploration strategies in a simulation system and seamlessly transplant them into the real environment. While we propose an autonomous exploration and navigation method for mobile robots based on deep reinforcement learning, which realizes the autonomous exploration and navigation of the robot in an unknown environment by combining Gaussian process regression, Bayesian optimization sampling, and deep reinforcement learning. The adaptability and robustness of the robot in complex unknown environments are improved. Summary of the Invention

[0008] For the autonomous exploration and navigation task of a mobile robot in an unknown environment, based on the method of deep reinforcement learning, the present invention proposes an autonomous exploration and navigation method for a mobile robot based on deep reinforcement learning. In an unknown environment, the system relies on sensors carried by the wheeled mobile robot (such as lidar, camera, etc.) to detect the surrounding environment and generate an environmental map in real time. The best path is selected using an algorithm to achieve the autonomous exploration and navigation of the mobile robot.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] An autonomous exploration and navigation method for a mobile robot based on deep reinforcement learning, characterized in that it includes the following steps:

[0011] Step 1: Data input module;

[0012] Step 2: Candidate action evaluation and selection module;

[0013] Step 3: Candidate target evaluation and selection module;

[0014] Step 4: Build a simulation scenario platform.

[0015] As a further improvement of the present invention, the specific steps of the above step 1 are as follows:

[0016] (1) Collection of surrounding environment data:

[0017] The lidar measures the distance to surrounding objects by emitting laser pulses and receiving the reflected light. These measurement results form point cloud data, providing data support for map construction;

[0018] (2) Use Simultaneous Localization and Mapping (SLAM) technology:

[0019] The SLAM algorithm fuses sensor data such as lidar, and simultaneously estimates the position of the robot and the environmental map. The SLAM algorithm consists of a prediction step and a correction step. The prediction step estimates the new position of the robot based on the previous position and motion information of the robot provided by the sensor; the correction step uses environmental measurement values to refine the predicted position. By iteratively executing these two steps, the SLAM algorithm continuously updates the position of the robot and the surrounding environmental map;

[0020] (3) Generate an occupancy map:

[0021] The occupancy map is represented in the form of occupancy grids, and each grid cell indicates the probability of being occupied by an object. The state of these grid cells is updated through lidar data. If the ray of the lidar hits an obstacle, the occupancy value of the corresponding grid increases.

[0022] As a further improvement of the present invention, the specific steps of step two are as follows:

[0023] (1) Construction of a map using Shannon entropy:

[0024] Introduce Shannon entropy as an index to measure the uncertainty of map construction. In map construction, Shannon entropy measures the amount of information in the exploration process, and then generates an autonomous exploration strategy for the target point;

[0025] Under the grid map m, the expression of Shannon entropy is described as:

[0026]

[0027] where p(m i,j ) refers to the probability that the cell at the intersection of the i-th row and the j-th column is occupied;

[0028] The calculation result will give a quantitative measure of uncertainty, which is used to evaluate the uncertainty of the map;

[0029] (2) Predict the mutual information reward surface of the existing occupancy map through a Gaussian process regression model:

[0030] Use the Gaussian process regression model to estimate the mutual information reward surface of the existing occupancy map, aiming to optimize the allocation of system computing resources, and then improve the operation efficiency of the system. The mutual information reward surface helps the robot determine the optimal exploration path. By calculating the mutual information between the unknown area and the explored area, the robot selects those areas that can maximize the information gain for exploration;

[0031] When predicting the mutual information reward surface of the existing occupancy map through the Gaussian process regression model, only calculate the training sample points within a certain range around the mobile robot, rather than calculating all the training sample points in the entire occupancy map;

[0032] Set a set of training sample data x = {x 1 , x 2 , …, x n}. Based on these data x and their corresponding results y, a Gaussian process regression model is constructed to predict the MI gain at a specified point on the existing occupancy map. This Gaussian process regression model will predict the output value y * associated with the test data x * , and calculate the corresponding covariance cov(y * );

[0033] y * = k(x * , x)[k(x, x) + σ n 2 I] -1y

[0034]

[0035] In the above expression, the estimated value I(m, x * ) corresponds to the result of the test data x * , denoted as y * , while cov(y * ) represents the covariance associated with the predicted output y * . The vector σ n 2 represents the Gaussian noise variance characteristics related to the training output y. In addition, the kernel function k(x, x'), also known as the covariance matrix, is used to represent the relationship between the input data x and x'. In the system studied in the present invention, the widely used Matérn-type kernel is adopted, and the specific expression is as follows:

[0036]

[0037] where the parameter v is responsible for adjusting the smoothness of the covariance function, the characteristic length is represented by l, Γ represents the gamma function, and the modified Bessel function is denoted as K v ;

[0038] (3) Improving the prediction result of Gaussian process regression using Bayesian optimization technology:

[0039] When sampling with the Bayesian optimization acquisition function, sampling is only performed within a certain range around the mobile robot, rather than on the entire occupancy map, so that points that are not useful for Bayesian optimization will not be sampled;

[0040] The sampling strategy of candidate points is calculated and determined according to a specific acquisition function under Bayesian optimization. The Gaussian process upper confidence bound GP-UCB algorithm is selected as the acquisition function, and its expression is as follows:

[0041]

[0042] In this formula, the parameter k represents the trade-off parameter between exploration and exploitation, and represents the action point consistent with the occupancy map resolution, μ(x) and σ(x) represent the predicted mean and variance obtained by Gaussian process regression respectively, and x sample * refers to the selected optimal sampling point;

[0043] (4) Selecting candidate action points:

[0044] After the iterative process of Bayesian optimization is completed, a set of sampled point data is obtained, and the mutual information reward surface is predicted through Gaussian process regression. The points collected on the occupancy map are defined as sampled points, and then N points with the highest gains are selected from the sampled points and used as candidate action points. Finally, a set of candidate action points for the mobile robot is formed;

[0045] When the mobile robot faces multiple regions with similar distances but different gains and not yet explored, the learning-based perception strategy then tends to select those regions with relatively lower gains for exploration. Its objective function is expressed as:

[0046]

[0047] where G action represents the set of candidate action points on the current occupancy map. φ and serve as weight coefficients and satisfy that φ is less than x global * represents the candidate action point selected as the optimal one. L(·) represents the A* path length for the mobile robot to reach this candidate action point.

[0048] As a further improvement of the present invention, the specific steps of step three are as follows:

[0049] The perception network model designs a combined image input, which includes the current occupancy map, the current position of the mobile robot, and all candidate action points on the current occupancy map;

[0050] When constructing the network model, a triple convolutional layer is adopted to extract the feature information of the input image. After each convolutional operation, the rectified linear unit ReLU is applied to eliminate negative feature values. The features extracted by the convolutional layer will be classified through a fully connected layer;

[0051] Then the output is fed to the Actor layer and the Critic layer for use. After the output of the Actor layer is processed by the Sigmoid activation function, the weight parameter ω∈(0,1) is generated. In the evaluation function, the filtering layer uses the weight parameter ω to estimate the score value of each candidate action point. This evaluation function is composed of the length of the A* path between the mobile robot and the candidate action point and the mutual information gain of the candidate action point. The expression of the evaluation function is as follows:

[0052]

[0053] where score global represents the score value obtained through the evaluation of the candidate action point; and and respectively represent the distance and the mutual information gain after normalization processing, and the specific expressions are as follows:

[0054]

[0055]

[0056] Among them, L(·) represents the length of the A* path from the mobile robot to the candidate action point; I(m, x t ) is the mutual information gain of the candidate action point; I ht (m, x) is the upper threshold of the preset mutual information gain.

[0057] As a further improvement of the present invention, the specific steps of step four are as follows:

[0058] In this simulation scenario platform, first, a beam-based sensing model is used to simulate the perception and detection activities of the mobile robot for the surrounding environment. This sensor model endows the mobile robot with a 360° omnidirectional coverage field of view and a resolution of 0.5°, enabling it to observe the surrounding environment in detail. At the same time, this module has the function of effectively identifying obstacles within a circular detection area with a diameter of 40 pixels. Subsequently, the A* path search algorithm is applied to determine the optimal path of the robot from the starting point to the target point. Finally, several maps with different scales and layouts are selected from the indoor layout dataset for the training and testing of the algorithm.

[0059] As a further improvement of the present invention, the autonomous exploration and navigation method of the mobile robot based on deep reinforcement learning is completed on an Intel Core i9-12900HX CPU with a main frequency of 4.5 GHz.

[0060] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solutions and has the following advantages:

[0061] The autonomous exploration navigation method of the present invention mainly consists of a data input module, a candidate action point selection module, and a candidate target point selection module. Among them, the data input module mainly transmits the occupancy map of the mobile robot, the position of the mobile robot, and the data of the surrounding environment information detected by the sensor to the next module. The candidate action point selection module mainly consists of Shannon entropy map construction, Gaussian process regression map prediction, Bayesian optimization sampling, and candidate action point selection. This module mainly selects the candidate action points of the mobile robot through some algorithms. The candidate target point selection module inputs data such as the occupancy map of the mobile robot, the candidate action points, and the position of the mobile robot into the network, and finally selects the optimal candidate target point from multiple candidate action points through training the network model. The mobile robot navigation module plans the optimal path according to the current position of the robot and the candidate target point, and drives the mobile robot to explore towards the target point. The method proposed by the present invention is applicable to providing a solution for the autonomous exploration navigation of mobile robots in an unknown and dynamically changing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is the overall framework of the autonomous exploration navigation method of the mobile robot of the present invention;

[0063] Figure 2 In (a) is the test map used in the present invention, and (b) is the initial position and initial occupancy map of the mobile robot;

[0064] Figure 3 In (a) is the current occupancy map and the current position of the mobile robot of the present invention, (b) is the mutual information reward surface predicted by the Gaussian process regression model and the initial training samples, (c) is the mutual information reward surface and sampling points after Bayesian optimization, and (d) is the selected candidate action points;

[0065] Figure 4 is the structural diagram of the perception network model of the present invention;

[0066] Figure 5 are the candidate target points selected by the present invention;

[0067] Figure 6 In (a) is the mobile robot of the present invention, and (b) is the actual experimental scenario. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] The present invention will be further described below in conjunction with calculation examples. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the protection scope of this application.

[0069] The technical solution of the present invention is as follows:

[0070] A mobile robot autonomous exploration and navigation method based on deep reinforcement learning, comprising the following steps:

[0071] Step 1: Data input module

[0072] 1. Collection of surrounding environment data: The lidar measures the distance to surrounding objects by emitting laser pulses and receiving the reflected light. These measurement results form point cloud data, providing data support for map construction.

[0073] 2. Use of Simultaneous Localization and Mapping (SLAM) technology: The SLAM algorithm (Gmapping algorithm) fuses sensor data such as lidar to estimate the position of the robot and the environmental map simultaneously. The SLAM algorithm consists of a prediction step and a correction step. The prediction step estimates the new position of the robot based on the previous position and motion information of the robot provided by the sensor; the correction step uses environmental measurement values (the distance of landmarks detected by the lidar sensor) to refine the predicted position. By iteratively executing these two steps, the SLAM algorithm can continuously update the position of the robot and the surrounding environment map.

[0074] 3. Generation of occupancy map: The occupancy map is represented in the form of occupancy grids, and each grid cell indicates the probability of being occupied by an object. The states of these grid cells are updated through lidar data. If the ray of the lidar hits an obstacle, the occupancy value of the corresponding grid increases.

[0075] Step 2: Candidate action point evaluation and selection module

[0076] 1. Construction of a map using Shannon entropy: In autonomous exploration and navigation, a mobile robot must consider the uncertainty factors of both the map and its own pose during the exploration phase in order to select appropriate candidate action points. Given the unknown nature of the surrounding environment, the mobile robot cannot directly obtain the uncertainty of the current map construction. Therefore, the present invention introduces Shannon entropy as an indicator to measure the uncertainty of map construction. In map construction, Shannon entropy can measure the amount of information in the exploration process and thus generate an autonomous exploration strategy for target points.

[0077] Under the grid map m, the expression of Shannon entropy can be described as:

[0078]

[0079] where p(m i,j ) refers to the probability that the cell at the intersection of the i-th row and the j-th column is occupied.

[0080] The calculation result will give a quantitative uncertainty measure, which can be used to evaluate the uncertainty of the map. The higher the entropy value, the greater the uncertainty of the map, that is, the less the mobile robot knows about the occupancy of the areas on the map. Conversely, the lower the entropy value, the smaller the uncertainty of the map.

[0081] 2. Predict the mutual information reward surface of the existing occupancy map through the Gaussian process regression model: The present invention uses the Gaussian process regression model to estimate the mutual information reward surface of the existing occupancy map, aiming to optimize the configuration of system computing resources and thus improve the operation efficiency of the system. The mutual information reward surface can help the robot determine the optimal exploration path. By calculating the mutual information between the unknown area and the explored area, the robot can select the areas that can maximize the information gain for exploration. This method can reduce the number of times of exploring repetitive areas and improve the exploration efficiency. The Gaussian process regression model is used to predict the uncertainty in the environment and utilize this information to improve the efficiency and effectiveness of decision-making.

[0082] When predicting the mutual information reward surface of the existing occupancy map through the Gaussian process regression model, only the training sample points within a certain range around the mobile robot are calculated, rather than all the training sample points in the entire occupancy map. This can improve the running efficiency of the algorithm and reduce the resources used for calculation.

[0083] Set a set of training sample data x = {x 1 , x 2 , …, x n}. Based on these data x and their corresponding results y, a Gaussian process regression model is constructed to predict the MI gain at specified points on the existing occupancy map. The Gaussian process regression model will predict the output value y * associated with the test data x * , and calculate the corresponding covariance cov(y * ).

[0084] y * = k(x * , x)[k(x, x) + σ n 2 I] -1 y

[0085]

[0086] In the above expression, the estimate I(m, x * ) corresponds to the result of the test data x * , denoted as y * , and cov(y * ) represents the covariance associated with the predicted output y * . The vector σn 2 It represents the Gaussian noise variance characteristic related to the training output y. In addition, the kernel function k(x, x'), also known as the covariance matrix, is used to represent the mutual relationship between the input data x and x'. In the system studied in the present invention, the widely used Matérn-type kernel is adopted, and the specific expression is as follows:

[0087]

[0088] Among them, the parameter v is responsible for adjusting the smoothness of the covariance function, the characteristic length is represented by l, Γ represents the gamma function, and the modified Bessel function is denoted as K v . Compared with the widely used radial basis function kernel (RBF), the Matérn kernel can effectively simulate the drastic changes in the mutual information reward surface in the presence of obstacles. Therefore, it is more suitable for application in autonomous exploration navigation.

[0089] 3. Improving the prediction result of Gaussian process regression using Bayesian optimization technology: Given that there are significant errors in predicting the mutual information reward surface by Gaussian process regression on the initial training data set, the Bayesian optimization strategy is applied to determine the sampling points using the variance predicted by the Gaussian process regression model and incorporate these points into the training set, aiming to improve the prediction accuracy of the mutual information reward surface.

[0090] Similarly, to improve the running efficiency of the algorithm, when sampling using the Bayesian optimization acquisition function, sampling is only performed within a certain range around the mobile robot, rather than on the entire occupancy map, so that the points that are useless for Bayesian optimization will not be sampled. Thus, the resources used for calculation are saved.

[0091] The sampling strategy of candidate points is calculated and determined according to a specific acquisition function under Bayesian optimization. In the present invention, the Gaussian process upper confidence bound (GP-UCB) algorithm is selected as the acquisition function, and its expression is as follows:

[0092]

[0093] In this formula, the parameter κ represents the trade-off parameter between exploration and exploitation, and represents the action point consistent with the occupancy map resolution. μ(x) and σ(x) represent the predicted mean and variance obtained by Gaussian process regression respectively, and x sample *It refers to the selected optimal sampling points. In the system of the present invention, the goal is to evenly distribute the candidate action points of the mobile robot among multiple potential high-reward regions, rather than over-concentrating on a single potential high-reward region. Therefore, the acquisition function designed and adopted in the present invention aims to strongly favor exploration behavior rather than concentration on exploitation. The acquisition function balances the trade-off between exploration and exploitation. In Gaussian process regression, the acquisition function can help select points that are both likely to bring high returns and have uncertainty for sampling.

[0094] 4. Select candidate action points: After the iterative process of Bayesian optimization is completed, a set of sampling point data can be obtained, and the mutual information reward surface is predicted through Gaussian process regression. The points collected on the occupancy map are defined as sampling points. Then, N points with the highest gains are selected from the sampling points and used as candidate action points. Finally, a set of candidate action points for the mobile robot is formed.

[0095] The selection of candidate action points is based on a learned perception strategy. The core goal of the perception strategy is to improve the detection path of the mobile robot to prevent missing any area during the detection process and end the exploration at the appropriate time. The perception strategy takes a more comprehensive consideration, taking into account multiple factors such as mutual information gain and path length to ensure the efficiency of exploration. When the mobile robot faces multiple unprobed regions with similar distances but different gains, the strategy tends to select those regions with relatively lower gains for exploration. Its objective function can be expressed as:

[0096]

[0097] where G action represents the set of candidate action points on the current occupancy map. φ and are used as weight coefficients and satisfy that φ is less than x global * denotes the selected optimal candidate action point. L(·) represents the A* path length for the mobile robot to reach this candidate action point.

[0098] Step 3: Candidate target evaluation module

[0099] The present invention designs a combined image input for the perception network model, which includes the current occupancy map, the current position of the mobile robot, and all candidate action points on the current occupancy map.

[0100] When constructing the network model, the present invention adopts a triple convolutional layer to extract the feature information of the input image. After each convolutional operation, a rectified linear unit (ReLU) is applied to eliminate negative feature values. The features extracted by the convolutional layer will be classified through a fully connected layer. Then the output is fed to the Actor layer and the Critic layer for use. After the output of the Actor layer is processed by the Sigmoid activation function, the weight parameter ω ∈ (0, 1) is generated. In the evaluation function, the filtering layer uses the weight parameter ω to estimate the score value of each candidate action point. The evaluation function is composed of the length of the A* path between the mobile robot and the candidate action point and the mutual information gain of the candidate action point. The expression of the evaluation function is as follows:

[0101]

[0102] where score global represents the score value obtained by evaluating through the candidate action point; and and respectively represent the distance and the mutual information gain after normalization processing, and the specific expressions are as follows:

[0103]

[0104]

[0105] where L(·) represents the length of the A* path from the mobile robot to the candidate action point; I(m, x t ) is the mutual information gain of the candidate action point; I ht (m, x) is the upper threshold of the mutual information gain preset by the present invention.

[0106] After determining the score values of all candidate action points, the mobile robot will select the candidate action point with the highest score among them and use it as the candidate target point for the next movement. Subsequently, the current position of the mobile robot and the candidate target point are transmitted to the navigation module, and the navigation module will plan the optimal travel path and drive the mobile robot to move to the candidate target point for autonomous exploration and navigation.

[0107] Step Four: Build a simulation scenario platform

[0108] The present invention constructs a training and simulation platform based on Python, aiming to effectively train and verify the autonomous exploration navigation system and evaluate the performance of the exploration navigation algorithm. In this simulation scenario platform, first, a beam-based sensing model is used to simulate the perception and detection activities of the mobile robot in the surrounding environment. This sensor model endows the mobile robot with a 360° panoramic field of view and a high-precision resolution of 0.5°, enabling it to observe the surrounding environment in detail. At the same time, this module has the function of effectively identifying obstacles within a circular detection area with a diameter of 40 pixels. Subsequently, the A* path search algorithm is applied in the present invention to determine the optimal path of the robot from the starting point to the target point. Finally, several maps with different scales and layouts are selected from the indoor layout dataset for the training and testing of the algorithm. These maps cover different complexity levels and different obstacle distributions, thus enabling a more comprehensive evaluation of the algorithm's performance and its generalization ability in different scenarios.

[0109] The autonomous exploration navigation method for mobile robots based on deep reinforcement learning proposed in this invention is trained on an Intel Core i9-12900HX CPU with a main frequency of 4.5GHz. During the execution of the algorithm, the starting point of each exploration of the mobile robot is randomly selected. This random design helps the mobile robot adapt to diverse exploration environments and enhances the generalization ability of the algorithm.

[0110] The present invention is trained and verified on the Python training and simulation platform. After training the network model for about 6 hours on a device with an Intel Core i9-12900HX CPU with a main frequency of 4.5GHz, a convergence and stability effect is achieved.

[0111] Figure 1 This is the overall framework of the autonomous exploration navigation method for mobile robots based on deep reinforcement learning in the present invention. It shows the overall process of the autonomous exploration navigation method for mobile robots.

[0112] Figure 2 In (a) is the test map used in the present invention. First, the mobile robot collects the surrounding environmental data information through the sensors it carries in the unknown environment; determines the initial position of the mobile robot; generates Figure 2 the initial occupancy map shown in (b) of

[0113] Figure 3Shown is a complete process of selecting candidate action points. Among them, (a) is the current occupancy map and the current position of the mobile robot, and the blue dots in the figure represent the mobile robot. (b) is the mutual information reward surface predicted by the Gaussian process regression model based on the initial training samples, and the red dots in the figure mark the positions of the initial training sample points. (c) is the mutual information reward surface and the distribution of training samples after Bayesian optimization iteration, and the green dots in the figure are the sampling points selected by the Bayesian optimization sampling function. (d) are the selected candidate action points, and the green dots in the figure are 20 candidate action points.

[0114] Figure 4 This is the perception network model architecture of the candidate target point selection module in the present invention. The present invention designs a combined image input for the perception network, which includes the current occupancy map, the current position of the mobile robot, and all candidate action points on the current occupancy map. Then, through the perception network model, the candidate target points are finally selected.

[0115] Figure 5 The selected candidate target points are shown as red dots in the figure.

[0116] Figure 6 In (a) is the mobile robot used in the present invention, which mainly consists of an intelligent car chassis, NVIDIA Jetson Nano, a robot operating system (ROS) main control board, a lidar, etc. (b) is the constructed real experimental scenario. The trained autonomous exploration navigation algorithm is deployed on the NVIDIA Jetson Nano. The lidar can detect the surrounding environment in real time and accurately capture the position information of obstacles; the ROS main control board is responsible for controlling the movement of the car and uploading the information of its own odometer and inertial measurement unit (IMU) data; the NVIDIA Jetson Nano is the core processing unit, which receives the data transmitted by the lidar and the ROS main control board and constructs and navigates the map in real time. Until the exploration of the entire unknown environment is completed.

[0117] The above is only a preferred embodiment of the present invention, and it is not any other form of limitation to the present invention. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope protected by the present invention.

Claims

1. A mobile robot autonomous exploration and navigation method based on deep reinforcement learning, characterized by: The following steps are involved: Step 1: Data input module; Step 2: Candidate action point selection module; Step 3: Candidate target point selection module; Step 4: Build a simulation scenario platform.

2. The method for autonomous exploration and navigation of a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step 1 are as follows: (1) Collection of surrounding environment data: LiDAR measures the distance to surrounding objects by emitting laser pulses and receiving reflected light. These measurement results form point cloud data, which provides data support for map construction. (2) Using real-time positioning and mapping to build SLAM technology: The SLAM algorithm integrates data from sensors such as LiDAR to estimate the robot's position and the environment map at the same time. The SLAM algorithm consists of a prediction step and a correction step. The prediction step estimates the robot's new position based on the robot's previous position and motion information provided by the sensor; the correction step uses environmental measurements to refine the predicted position. By iteratively executing these two steps, the SLAM algorithm continuously updates the robot's position and the surrounding environment map. (3) Generate occupancy map: The occupancy map is represented in the form of an occupancy grid, where each grid cell indicates the probability of being occupied by an object. The status of these grid cells is updated by the lidar data. If the lidar's ray hits an obstacle, the occupancy value of the corresponding grid increases.

3. The method for autonomous exploration and navigation of a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step 2 are as follows: (1) Construction of the map using Shannon entropy: Shannon entropy is introduced as an indicator to measure the uncertainty of map construction. In map construction, Shannon entropy measures the amount of information in the exploration process and then generates an autonomous exploration strategy for the target point. Under the definition of the grid map m, the expression of Shannon entropy is described as: Among them, p(m i,j ) refers to the probability that the cell at the intersection of the i-th row and the j-th column is occupied; The calculation results will give a quantitative uncertainty measure to evaluate the uncertainty of the map; (2) Predict the mutual information reward surface of the existing occupancy map through the Gaussian process regression model: The Gaussian process regression model is used to estimate the mutual information reward surface of the existing occupied map, aiming to optimize the configuration of the system's computing resources and thus improve the system's operating efficiency. The mutual information reward surface helps the robot determine the optimal exploration path. By calculating the mutual information between unknown areas and explored areas, the robot selects those areas that can maximize information gain for exploration. When predicting the mutual information reward surface of the existing occupancy map through the Gaussian process regression model, only the training sample points within a certain range around the mobile robot are calculated, rather than the training sample points in the entire occupancy map; Suppose a set of training sample data x = {x1, x2, ..., x n Based on these data x and their corresponding results y, a Gaussian process regression model is constructed to predict the MI gain of a specified point on the existing occupancy map. The Gaussian process regression model will predict and test data x * The associated output value y * , and calculate the corresponding covariance cov(y * ); y * =k(x * ,x)[k(x,x)+σ n 2 I] -1 y In the above expression, the estimated value I(n,x * ) corresponds to the test data x * The result is denoted as y * , and cov(y * ) represents the predicted output y * The associated covariance. The vector σ n 2 represents the Gaussian noise variance characteristic related to the training output y. In addition, the kernel function k(x,x'), also known as the covariance matrix, is used to represent the relationship between the input data x and x'. In the system studied in the present invention, the widely used Matérn kernel is adopted, and the specific expression is as follows: The parameter v is responsible for adjusting the smoothness of the covariance function, the characteristic length is represented by l, Γ represents the gamma function, and the modified Bessel function is recorded as K v ; (3) Using Bayesian optimization techniques to improve the prediction results of Gaussian process regression: When sampling the Bayesian optimization acquisition function, only a certain range around the mobile robot is sampled instead of the entire occupancy map, so that points that are useless for Bayesian optimization will not be sampled; The sampling strategy of candidate points is calculated and determined based on a specific acquisition function under Bayesian optimization. The Gaussian process upper confidence bound GP-UCB algorithm is selected as the acquisition function, and its expression is as follows: In this formula, the parameter κ represents the trade-off parameter between exploration and development, and represents the action point consistent with the resolution of the occupancy map, μ(x) and σ(x) represent the predicted mean and variance obtained by Gaussian process regression, respectively, x sample * It refers to the optimal sampling point selected; (4) Selection of candidate action points: After the iteration process of Bayesian optimization is completed, a set of sampling point data is obtained, and the mutual information reward surface is obtained through Gaussian process regression prediction. The points collected on the occupancy map are defined as sampling points, and then N points with the highest gain are selected from the sampling points as candidate action points. Finally, a set of candidate action points for the mobile robot is formed. When a mobile robot faces multiple areas with similar distances but different gains and that have not yet been explored, the learning-based perception strategy tends to select those areas with relatively low gains for exploration. Its objective function is expressed as: Among them, G action represents the set of candidate action points on the current occupied map. As the weight coefficient, and satisfy φ is less than x global * represents the candidate action point selected as the best. L(·) represents the A* path length of the mobile robot to reach the candidate action point.

4. The method for autonomous exploration and navigation of a mobile robot based on deep reinforcement learning according to claim 3, characterized in that: The specific steps of step three are as follows: The perception network model is designed with a combined image input that includes the current occupancy map, the current position of the mobile robot, and all candidate action points on the current occupancy map; When constructing the network model, a triple convolution layer is used to extract the feature information of the input image. After each convolution operation, the linear rectification function ReLU is applied to eliminate negative eigenvalues. The features extracted by the convolution layer will be classified through the fully connected layer. The output is then fed to the Actor layer and the Critic layer for use. The output of the Actor layer is processed by the Sigmoid activation function to generate a weight parameter ω∈(0,1). In the evaluation function, the filter layer uses the weight parameter ω to estimate the score of each candidate action point. The evaluation function is composed of the length of the A* path between the mobile robot and the candidate action point and the mutual information gain of the candidate action point. The expression of the evaluation function is as follows: Among them, score global represents the score value obtained by evaluating the candidate action point; and and They represent the normalized distance and mutual information gain, respectively, and are expressed as follows: Where L(·) represents the length of the A* path from the mobile robot to the candidate action point; I(m,x t ) is the mutual information gain of candidate action points; I ht (m,x) is the preset upper limit threshold of mutual information gain.

5. The method for autonomous exploration and navigation of a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of step 4 are as follows: In this simulation scenario platform, the first step is to use a beam-based sensing model to simulate the mobile robot's perception and detection of the surrounding environment. This sensor model gives the mobile robot a 360° field of view and a resolution of 0.5°, enabling it to observe the surrounding environment in detail. At the same time, the module has the function of effectively identifying obstacles within a circular detection area with a diameter of 40 pixels. Subsequently, the A* path search algorithm was applied to determine the optimal path for the robot from the starting point to the target point. Finally, several maps of different scales and layouts were selected from the indoor layout dataset for algorithm training and testing.

6. The method for autonomous exploration and navigation of a mobile robot based on deep reinforcement learning according to claim 1, characterized in that: The autonomous exploration and navigation method for mobile robots based on deep reinforcement learning was trained on an Intel Corei9-12900HX CPU with a main frequency of 4.5GHz.

Images