Learning-based map-free autonomous navigation obstacle avoidance method, medium and equipment

Abstract

The invention provides a learning-based map-free autonomous navigation obstacle avoidance method, a medium and equipment. The method comprises the following steps: carrying out map-free autonomous navigation obstacle avoidance on the mobile robot by adopting a reusable planning capability library; the multiplexing planning capability library comprises a global planning module and a local planning module; the global planning module is used for guiding a global navigation direction and generating ordered and discrete path points; the local planning module generates real-time actions based on a learned local navigation obstacle avoidance model; and controlling the mobile robot to move according to the action generated by the local planning module so as to sequentially navigate the mobile robot to each path point generated by global planning. According to the method, through a global and local combined planning architecture, an algorithm is modularized, so that the method adapts to different navigation tasks, a large amount of retraining based on a deep reinforcement learning navigation method is avoided, and the universality and interpretability of a model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of navigation and obstacle avoidance, and more specifically to a map-free autonomous navigation and obstacle avoidance method, medium and device based on learning. Background Art

[0002] Autonomous navigation and obstacle avoidance of mobile robots plays an important role in many tasks, but still faces challenges in complex and uncertain environments such as post-disaster rescue, dangerous environment exploration, and enemy monitoring.

[0003] The basic idea of ​​navigation and obstacle avoidance is to plan a collision-free path to the target location for the unmanned system, which is usually divided into traditional methods and intelligent methods. The traditional method relies on prior information, that is, before planning, various methods such as SLAM and other technologies are used to obtain the environment map, and then the path planning is performed based on the known map. The classic planning algorithms are A* and Dijkstra, which can guarantee the shortest path from the starting point to the target point. Although the research on traditional autonomous navigation and obstacle avoidance methods is relatively mature, in the above-mentioned mission scenarios, there are situations such as difficulty in obtaining the global map in advance, limited detection accuracy, and unpredictable dynamic obstacles, which cause the traditional method to require a lot of calculations, lack of real-time performance, and easily lead to mission failure. Therefore, it has gradually become a trend to study intelligent methods with a higher level of autonomy.

[0004] Intelligent methods are highly adaptable to the environment, and mainly include group optimization, fuzzy logic, and learning-based algorithms. Among them, the group optimization algorithm has high computational complexity and also faces real-time issues. Although the fuzzy logic algorithm has a small amount of computation, it relies heavily on expert experience and is therefore prone to failure in uncertain environments. The deep reinforcement learning algorithm does not need to explicitly build an environmental model and solve optimization problems. It directly obtains navigation strategies based on sensor inputs, and has great potential in improving autonomy and coping with dynamic unknown environments.

[0005] However, deep reinforcement learning algorithms learn by interacting with the environment, navigation task rewards are sparse, algorithm training efficiency is low, and it is even difficult to converge in complex scenarios. In addition, current learning-based methods are mostly implemented in static, ideal environments, and there is a large gap between simulation and reality. In addition, trained models are usually only applicable to specific scenarios and have poor scalability. Even if the navigation task changes slightly, the agent needs to be retrained. Summary of the invention

[0006] In order to overcome the shortcomings and deficiencies in the prior art, the purpose of the present invention is to provide a mapless autonomous navigation obstacle avoidance method, medium and device based on learning; the method realizes mapless dynamic obstacle avoidance navigation to the specified target point by improving the double-delayed deep deterministic policy gradient (TD3) algorithm; through the planning architecture combining global and local, the fixed-size end-to-end model is suitable for large-scale navigation task scenarios. In order to realize the transfer of the algorithm in the real environment, a variable random simulation training environment is designed, and the input and output data are optimized; the proposed method can improve the obstacle avoidance performance of mobile robots in dynamic and complex environments, the algorithm model has high versatility, can adapt to different navigation tasks without a large amount of repeated training, and provides a safer and more reliable autonomous navigation obstacle avoidance method for non-deterministic complex real environments.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solutions: a mapless autonomous navigation and obstacle avoidance method based on learning, which uses a reusable planning capability library to perform mapless autonomous navigation and obstacle avoidance on a mobile robot; the reusable planning capability library includes a global planning module and a local planning module;

[0008] The global planning module is used to guide the global navigation direction and generate orderly and discrete path points;

[0009] The local planning module generates real-time actions based on the learned local navigation and obstacle avoidance model;

[0010] Control the movement of the mobile robot according to the actions generated by the local planning module, so as to navigate the mobile robot to each path point generated by the global planning in sequence;

[0011] The local navigation obstacle avoidance model refers to the use of Actor-Critic architecture; in the Actor-Critic architecture, the Actor network is based on the state s i is the input, action a i is the output;

[0012] a i =π(s i ; φ)

[0013] Among them, i represents the time, π represents the strategy function of the Actor network; φ is the parameter of the Actor network;

[0014] The Critic network is used to evaluate the quality of the Actor network:

[0015] q n =Q n (s i , a i θ n )

[0016] Among them, qn , n = 1, 2, respectively represent the outputs of the two Critic networks; Q n Represents the function corresponding to the Critic network; θ n It is the parameter corresponding to the Critic network.

[0017] Preferably, the training method of the local navigation obstacle avoidance model comprises the following steps:

[0018] Step S1, establishing a local navigation obstacle avoidance simulation environment with obstacles; the obstacles include static obstacles and dynamic obstacles of different shapes;

[0019] Step S2: In the local navigation obstacle avoidance simulation environment, when the mobile robot reaches the target point or collides or runs to the maximum set number of steps, the current round ends and the next round starts again; the local navigation obstacle avoidance model is trained for several rounds to learn the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point, and the parameters of the local navigation obstacle avoidance model are updated;

[0020] For each round, the following steps are included:

[0021] Step S2-1, in a local navigation obstacle avoidance simulation environment, setting a random mobile robot starting position, target point position, dynamic obstacle position and movement speed;

[0022] Step S2-2: Under the guidance of the reward function, the mobile robot learns the navigation and obstacle avoidance strategy and sets the reward function R i :

[0023] R i =R g +R c +R a +R p

[0024] Among them, R g is the reward for reaching the goal, R c is the penalty for collision with an obstacle, R a is the action reward; R p It is a small time penalty imposed on each step in the round when the target point is not reached;

[0025] Step S2-3, at the current time i=t, change the state s t Input the Actor network and get the current action a t , and add random noise ∈ 1 ; Action a after adding random noise t for:

[0026] a t =clip(π(st ;φ)+∈ 1 , a min , a max )

[0027]

[0028] Among them, a min represents the minimum action of the mobile robot; a max represents the maximum action of the mobile robot; clip represents the clipping function, which is used to limit the action value to [a min , a max ], when the calculated action value is less than the lower limit, the value is a min , if it is greater than the upper limit, the value is a max , in [a min , a max ] range, then take itself; Represents a mean of 0 and a standard deviation of σ 1 Normal distribution of 1 represents a constant;

[0029] Step S2-4: Simulate the mobile robot to perform action a t After that, calculate the reward function R t , give the next moment state s t+1 and done flag, forming an experience tuple (s t , a t , R t ,s t+1 , done), and stored in the experience replay pool;

[0030] Step 2-5: Update the network parameters and randomly extract N experience tuples from the experience replay pool;

[0031] First use the target network of the Actor network and the Critic network to calculate the target value y i Then use the gradient descent method to minimize the loss function L(θ n ) to update the Critic network parameters θ n :

[0032]

[0033] Among them, θ n ′ and φ′ represent the parameters of the Actor network and the Critic network corresponding to the target network respectively; γ represents the discount factor; ∈ 2 Represents a mean of 0 and a standard deviation of σ 2 Gaussian noise; c is a small constant, representing the clipping ∈ 2 the scope and limits of

[0034] The deterministic policy gradient is used to update the Actor network parameters, and the gradient ascent method is used to maximize the Q value of the Critic network; the deterministic policy gradient is as follows:

[0035]

[0036] in, is the gradient of the objective function J(φ) with respect to the Actor network parameter φ; the gradient is approximated by taking the average of the N samples in the experience replay pool; Represents the value function Q n about the gradient of the action, and a i By π(s i ; φ) calculation; represents the gradient of the Actor network with respect to the parameter φ;

[0037] Target network parameters θ n ′ and φ′ are soft updated, and the formula is as follows:

[0038] θ n ′=τθ n +(1-τ)θ n '

[0039] φ′=τφ+(1-τ)φ′

[0040] Among them, τ∈(0,1) is the learning rate, which takes a smaller value.

[0041] Preferably, in step S2-2, the reward R for reaching the target g , Penalty R for collision with obstacles c , action reward R a , a small time penalty R imposed on each step in the round when the target point is not reached p They are:

[0042]

[0043] R a =α·v t -β·abs(ω t )+μ·Δd g

[0044] R p =r p

[0045] Among them, d g is the distance between the mobile robot and the target point, δ 1 is the threshold; d o is the distance between the mobile robot and the obstacle, δ 2 is the threshold value; vt is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance between the mobile robot and the target point at the previous moment and the Euclidean distance at the current moment; α, β and μ are the set proportional coefficients; r p is a negative constant.

[0046] Preferably, in the Actot-Critic architecture, LSTM networks are respectively added between the fully connected layers of the Actor network and between the fully connected layers of the Critic network;

[0047] The Actor network is in state s i As input, it is processed by the fully connected layer and LSTM network, and the output action is a i ; The Critic network is in state s i is the input, processed by the fully connected layer and LSTM network, and then combined with the action a output by the Actor network i After concatenation, the output is obtained by passing through the fully connected layer and the Q value layer.

[0048] Preferably, the output action a of the Actor network i =[v i ,ω i ]; where v i and ω i are the linear velocity and angular velocity output by the Actor network respectively; after the Actor network, the tanh activation function is used, and then the maximum linear velocity v of the mobile robot is used. max and the maximum angular velocity ω max Scaling is performed to obtain the output action a of the local navigation obstacle avoidance model:

[0049] a=[v,ω]

[0050] v=v max v i

[0051] ω=ω max ω i

[0052] Among them, v and ω are linear velocity and angular velocity respectively; v i and ω i The value range is (-1, 1).

[0053] Preferably, the state s of the Actot network input iIt includes: local perception environment information at the current moment, local perception environment historical information, action at the previous moment, and polar coordinates of the target point with the mobile robot as the origin. Local perception environment information and local perception environment historical information refer to: data detected by laser sensors on the local environment and data detected by laser sensors on the local environment at historical moments.

[0054] Preferably, the global planning module uses a global planner in the robot operating system navigation stack to generate a global path; then uses a Ramer-Douglas-Peucker algorithm to downsample the path point set of the global path to extract ordered, discrete path points on the global path.

[0055] A readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the learning-based map-free autonomous navigation obstacle avoidance method.

[0056] A computer device comprises a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the map-free autonomous navigation obstacle avoidance method based on learning is implemented.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] 1. The method of the present invention enables unmanned systems to avoid obstacles without maps in complex dynamic environments, eliminating the complex rules based on traditional methods and the reliance on prior information, improving the obstacle avoidance and navigation performance in complex and uncertain environments, and providing more reliable safety guarantees for the navigation tasks of unmanned systems;

[0059] 2. The method of the present invention is different from the pure end-to-end method. By adopting a framework combining global and local planning, the algorithm is modularized to adapt to different navigation tasks, avoiding a large amount of retraining based on deep reinforcement learning navigation methods, and improving the versatility and interpretability of the algorithm model;

[0060] 3. The present invention takes into account the reliability of implementation in a real environment, designs a variable random simulation training environment, optimizes the processing of input and output data, and can achieve direct transfer of learning-based methods from simulation to reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a structural block diagram of a reusable planning capability library for navigation tasks of the map-free autonomous navigation and obstacle avoidance method of the present invention;

[0062] Figure 2 It is a structural schematic diagram of a local navigation obstacle avoidance model of the map-free autonomous navigation obstacle avoidance method of the present invention. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0064] Embodiment 1

[0065] The present embodiment is a mapless autonomous navigation and obstacle avoidance method based on learning, which adopts a reusable planning capability library. Through a modular architecture that combines the global and local aspects, it can be used for navigation tasks in complex non-deterministic environments, and the reusable planning capability library has high versatility. The mapless autonomous navigation and obstacle avoidance system based on learning needs to include at least a mobile robot chassis, a laser sensor, and an odometer.

[0066] The reuse planning capability library includes global planning modules and local planning modules, such as Figure 1 As shown in the figure, following the hierarchical idea of ​​combining global and local planning, the task is divided into global and local planning modules. The global planning module guides the global navigation direction according to the task requirements and generates orderly and discrete path points. The local planning module is based on the learning method and is responsible for responding to real-time changes in the environment and navigating to each path point provided by the global planning module in turn, so as to autonomously navigate and avoid obstacles in a complex dynamic environment in an end-to-end manner.

[0067] Preferably, in order to combine the advantages of traditional planning and make full use of existing mature modules, the global planning module directly uses the global planner in the robot operating system (ROS) navigation stack, such as A* or RRT global planner, to generate a global path according to task requirements; then the Ramer-Douglas-Peucker (RDP) algorithm is used to downsample the path point set of the global path, thereby retaining the basic shape of the global path and extracting ordered, discrete path points. When performing navigation tasks, the global planning module runs at a lower frequency.

[0068] The local planning module generates real-time action instructions based on the learned local navigation and obstacle avoidance model to navigate to each path point generated by the global planning in sequence.

[0069] The local navigation obstacle avoidance model adopts the Actor-Critic architecture; the Actor-Critic architecture includes the Actor network and the Critic network, such as Figure 2 As shown in the figure. LSTM networks are added between the fully connected layers of the Actor network and the fully connected layers of the Critic network; the Actor network is based on the state s i As input, it is processed by the fully connected layer and LSTM network, and the output action is a i ; Critic network with state s i is the input, processed by the fully connected layer and LSTM network, and then combined with the action a output by the Actor networki After concatenation, the output is obtained by passing through the fully connected layer and the Q value layer.

[0070] The Actor-Critic architecture is adopted, and a deterministic Actor network is used to process high-dimensional continuous actions. The input data is preprocessed and then connected to the LSTM network to help process historical information and extract features, capture the changing characteristics of dynamic obstacles in the environment, and thus guide the model to learn obstacle avoidance decisions for unknown dynamic obstacles.

[0071] In the Actor-Critic architecture, the Actor network is based on the state s i is the input, action a i is the output; state s i It includes: the local perceived environment information at the current moment, the local perceived environment history information, the action at the previous moment and the polar coordinates of the target point with the mobile robot as the origin; action a i for:

[0072] a i =π(s i ; φ)

[0073] Among them, i represents the time, π represents the strategy function of the Actor network; φ is the parameter of the Actor network;

[0074] The Critic network is used to evaluate the quality of the Actor network:

[0075] q n =Q n (s i , a i θ n )

[0076] Among them, q n , n = 1, 2, respectively represent the outputs of the two Critic networks; Q n Represents the function corresponding to the Critic network; θ n It is the parameter corresponding to the Critic network.

[0077] The output action of the Actor network is a i =[v i ,ω i ]; where v i and ω i are the linear velocity and angular velocity output by the Actor network respectively; they are consistent with the action of the mobile robot and use a continuous action space. In order to limit the speed range, the tanh activation function is used after the Actor network, and then the maximum linear velocity v of the mobile robot is used. max and the maximum angular velocity ω maxScaling is performed to obtain the output action a of the local navigation obstacle avoidance model:

[0078] a=[v,ω]

[0079] v=v max v i

[0080] ω=ω max ω i

[0081] Among them, v and ω are linear velocity and angular velocity respectively; v i and ω i The value range is (-1, 1).

[0082] The training method of the local navigation obstacle avoidance model includes the following steps:

[0083] Step S1, establish a local navigation obstacle avoidance simulation environment with obstacles; obstacles include static obstacles and dynamic obstacles of different shapes; the density of obstacles in different sub-areas of the simulation environment is different. In order to reduce the gap between simulation and reality, the simulator used needs to be equipped with a high-performance physics engine. Taking the position of the mobile robot as the starting point and the path point generated by the global planning module as the target point, interact with the local navigation obstacle avoidance simulation environment to learn the navigation obstacle avoidance strategy.

[0084] Step S2: In a local navigation obstacle avoidance simulation environment, the local navigation obstacle avoidance model is trained for several rounds to learn the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point, and the parameters of the local navigation obstacle avoidance model are updated.

[0085] When the mobile robot reaches the target point or collides or runs to the maximum set number of steps, the current round ends and the next round starts again; the local navigation obstacle avoidance model interacts with the local navigation obstacle avoidance simulation environment through the reward function R i The guidance to maximize the discounted return U t The expectation is taken as the goal to learn the navigation and obstacle avoidance strategy.

[0086] Specifically, the discounted return U t for:

[0087]

[0088] where γ∈[0, 1] is the discount factor.

[0089] Discount ReturnU t The expectation is the action value function Q π , the formula is:

[0090]

[0091] Among them, Q π Indicates that when using strategy π, in state s t and take action a t The expected discounted return that can be obtained.

[0092] For each round, the following steps are included:

[0093] Step S2-1, in the local navigation obstacle avoidance simulation environment, randomly set the starting position, target point position, dynamic obstacle position and movement speed of the mobile robot; the mobile robot position, target point position and dynamic obstacle position are set in the unoccupied area of ​​the local navigation obstacle avoidance simulation environment; in order to improve the training efficiency, in the first M rounds, that is, in the initial stage of training, the target point position is set near the mobile robot, and the distance between the target point position and the mobile robot position is less than 2 meters; so that the mobile robot can meet the target as soon as possible to obtain a high reward, and help the local navigation obstacle avoidance model learn the correct strategy. After reaching a certain number of rounds, the restriction is cancelled and randomly generated.

[0094] Step S2-2: Set the reward function R i :

[0095] R i =R g +R c +R a +R p

[0096] Among them, R g is the reward for reaching the goal, R c is the penalty for collision with an obstacle, R a is the action reward, which is used to optimize the trajectory of the mobile robot, encourage forward movement and move toward the target direction, and punish rapid turning; R p It is a small time penalty imposed on each step in the round when the target point is not reached, which is used to guide the mobile robot to reach the target as soon as possible. Since reaching the target point and collision are sparse rewards, a dense reward R is designed. a and R p To improve training efficiency.

[0097] The reward for reaching the target is R g , Penalty R for collision with obstacles c , action reward R a , a small time penalty R imposed on each step in the round when the target point is not reached p They are:

[0098]

[0099] R a =α·v t -β·abs(ωt )+μ·Δd g

[0100] R p =r p

[0101] Among them, d g is the distance between the mobile robot and the target point, δ 1 is the threshold; when d g Less than the threshold δ 1 When the target point is reached, a reward will be given; o is the distance between the mobile robot and the obstacle, δ 2 is the threshold; when d o Less than the threshold δ 2 When the vehicle collides with an obstacle, a penalty will be imposed; t is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance between the mobile robot and the target point at the previous moment and the Euclidean distance at the current moment; α, β and μ are the set proportional coefficients; r p is a negative constant.

[0102] Step S2-3, at the current time i=t, change the state s t Input the Actor network and get the current action a t , and add random noise ∈ 1 ; During the training process, in order to ensure the exploratory nature of the deterministic strategy, the actual action a t We also need to add random noise ∈ 1 , action a after adding random noise t for:

[0103] a t =clip(π(s t ;φ)+∈ 1 , a min , a max )

[0104]

[0105] Among them, a min represents the minimum action of the mobile robot; a max represents the maximum action of the mobile robot; clip represents the clipping function, which is used to limit the action value to [a min , a max ], when the calculated action value is less than the lower limit, the value is a min , if it is greater than the upper limit, the value is a max , in [amin , a max ] range, then take itself; Represents a mean of 0 and a standard deviation of σ 1 Normal distribution of 1 represents a constant;

[0106] Step S2-4: Simulate the mobile robot to perform action a t After that, calculate the reward function R t , give the next moment state s t+1 and done flag, forming an experience tuple (s t , a t , R t ,s t+1 , done), and stored in the experience replay pool;

[0107] Step 2-5: Update the network parameters and randomly extract N experience tuples from the experience replay pool;

[0108] First use the target network of the Actot network and the Critic network to calculate the target value y i Then use the gradient descent method to minimize the loss function L(θ n ) to update the Critic network parameters θ n :

[0109]

[0110]

[0111] Among them, θ n ′ and φ′ represent the parameters of the Actor network and the Critic network corresponding to the target network, ∈ 2 Represents a mean of 0 and a standard deviation of σ 2 Gaussian noise; c is a small constant, representing the clipping ∈ 2 The range limit of ; introduce Gaussian noise ∈ 2 The target strategy is smoothed so that similar actions have similar values, allowing the Critic network to smoothly estimate nearby actions and reduce overfitting of a single action. In addition, the target action with added noise needs to be based on the maximum action a of the real mobile robot. max and the minimum action a min Make sure the cuts are effective.

[0112] The deterministic policy gradient is used to update the Actor network parameters, and the gradient ascent method is used to maximize the expectation of the discounted return, that is, the Q value of the Critic network; when updating, delayed updates are used to update the Critic network multiple times and then update the Actor network once, thereby improving the stability of training. The deterministic policy gradient is as follows:

[0113]

[0114] in, is the gradient of the objective function J(φ) with respect to the Actor network parameter φ; the gradient is approximated by taking the average of the N samples in the experience replay pool; represents the gradient of the Q-value function with respect to the action, and a is composed of π(s i ; φ) calculation; represents the gradient of the Actor network with respect to the parameter φ;

[0115] Target network parameters θ n ′ and φ′ are soft updated, and the formula is as follows:

[0116] θ n ′=τθ n +(1-τ)θ n '

[0117] φ′=τφ+(1-τ)φ′

[0118] Among them, τ∈(0,1) is the learning rate, which takes a smaller value.

[0119] Embodiment 2

[0120] This embodiment provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the map-free autonomous navigation obstacle avoidance method based on learning as described in the first embodiment.

[0121] Embodiment 3

[0122] This embodiment provides a computer device, including a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the map-free autonomous navigation obstacle avoidance method based on learning described in the first embodiment is implemented.

[0123] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A map-free autonomous navigation and obstacle avoidance method based on learning, characterized by: A reusable planning capability library is used to enable mobile robots to navigate and avoid obstacles without maps. The reuse planning capability library includes global planning modules and local planning modules; The global planning module is used to guide the global navigation direction and generate orderly and discrete path points; The local planning module generates real-time actions based on the learned local navigation and obstacle avoidance model; Control the movement of the mobile robot according to the actions generated by the local planning module, so as to navigate the mobile robot to each path point generated by the global planning in sequence; The local navigation obstacle avoidance model refers to the use of Actor-Critic architecture; in the Actor-Critic architecture, the Actor network is based on the state s i is the input, action a i is the output; a i =π(s i ;f) Among them, i represents the time, π represents the strategy function of the Actor network; φ is the parameter of the Actor network; The Critic network is used to evaluate the quality of the Actor network: q n =Q n (s i ,a i ;θ n ) Among them, q n , n = 1, 2, respectively represent the outputs of the two Critic networks; Q n Represents the function corresponding to the Critic network; θ n It is the parameter corresponding to the Critic network.

2. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The training method of the local navigation obstacle avoidance model comprises the following steps: Step S1, establishing a local navigation obstacle avoidance simulation environment with obstacles; the obstacles include static obstacles and dynamic obstacles of different shapes; Step S2: In the local navigation obstacle avoidance simulation environment, when the mobile robot reaches the target point or collides or runs to the maximum set number of steps, the current round ends and the next round starts again; the local navigation obstacle avoidance model is trained for several rounds to learn the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point, and the parameters of the local navigation obstacle avoidance model are updated; For each round, the following steps are included: Step S2-1, in a local navigation obstacle avoidance simulation environment, setting a random mobile robot starting position, target point position, dynamic obstacle position and movement speed; Step S2-2: Set the reward function R i : R i =R g +R c +R a +R p Among them, R g is the reward for reaching the goal, R c is the penalty for collision with an obstacle, R a is the action reward; R p It is a small time penalty imposed on each step in the round when the target point is not reached; Step S2-3, at the current time i=t, change the state s t Input the Actor network and get the current action a t , and add random noise ∈ 1; the action a after adding random noise t for: a t =clip(π(s t ;φ)+∈1,a min ,a max ) Among them, a min represents the minimum action of the mobile robot; a max represents the maximum motion of the mobile robot; clip represents the clipping function; represents a normal distribution with a mean of 0 and a standard deviation of σ1; σ1 represents a constant; Step S2-4: Simulate the mobile robot to perform action a t After that, calculate the reward function R t , give the next moment state s t+1 and done flag, forming an experience tuple (s t , a t , R t ,s t+1 , done), and stored in the experience replay pool; Step 2-5: Update the network parameters and randomly extract N experience tuples from the experience replay pool; first use the target network of the Actor network and the Critic network to calculate the target value y i Then use the gradient descent method to minimize the loss function L(θ n ) to update the Critic network parameters θ n : Among them, θ n ′ and φ′ represent the parameters of the target network corresponding to the Actor network and the Critic network respectively; γ represents the discount factor; ∈2 represents Gaussian noise with a mean of 0 and a standard deviation of σ2; c represents the range limit of clipping ∈2; The deterministic policy gradient is used to update the Actor network parameters, and the gradient ascent method is used to maximize the Q value of the Critic network; the deterministic policy gradient is as follows: in, is the gradient of the objective function J(φ) with respect to the Actor network parameter φ; the gradient is approximated by taking the average of the N samples in the experience replay pool; represents the gradient of the Q-value function with respect to the action, and a i By π(s i ; φ) calculation; represents the gradient of the Actor network with respect to the parameter φ; Target network parameters θ n ′ and φ′ are soft updated, and the formula is as follows: i n ′=τθ n +(1-τ)θ n ′ φ′=τφ+(1-τ)φ′ Among them, τ∈(0,1) is the learning rate, which takes a smaller value.

3. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 2, characterized in that: In step S2-2, the reward R for reaching the target g , Penalty R for collision with obstacles c , action reward R a , a small time penalty R imposed on each step in the round when the target point is not reached p They are: R a =α·v t -β·abs(ω t )+μ·Δd g R p ＝r p Among them, d g is the distance between the mobile robot and the target point, δ1 is the threshold; d o is the distance between the mobile robot and the obstacle, δ2 is the threshold; v t is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance between the mobile robot and the target point at the previous moment and the Euclidean distance at the current moment; α, β and μ are the set proportional coefficients; r p is a negative constant.

4. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: In the Actor-Critic architecture, LSTM networks are added between the fully connected layers of the Actor network and between the fully connected layers of the Critic network respectively; The Actor network is in state s i As input, it is processed by the fully connected layer and LSTM network, and the output action is a i ; The Critic network is in state s i is the input, processed by the fully connected layer and LSTM network, and then combined with the action a output by the Actor network i After concatenation, the output is obtained by passing through the fully connected layer and the Q value layer.

5. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The output action a of the Actor network i =[v i ,ω i ]; Among them, v i and ω i are the linear velocity and angular velocity output by the Actor network respectively; after the Actor network, the tanh activation function is used, and then the maximum linear velocity v of the mobile robot is used. max and the maximum angular velocity ω max Scaling is performed to obtain the output action a of the local navigation obstacle avoidance model: a=[v,ω] v=v max v i oh = oh max oh i Among them, v and ω are linear velocity and angular velocity respectively; v i and ω i The value range is (-1,1).

6. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The state s of the Actor network input i It includes: the local perceived environment information at the current moment, the local perceived environment history information, the action at the previous moment, and the polar coordinates of the target point with the mobile robot as the origin.

7. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The global planning module uses the global planner in the robot operating system navigation stack to generate a global path; then uses the Ramer-Douglas-Peucker algorithm to downsample the path point set of the global path to extract ordered and discrete path points on the global path.

8. A readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the learning-based map-free autonomous navigation and obstacle avoidance method according to any one of claims 1 to 7.

9. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the map-free autonomous navigation obstacle avoidance method based on learning described in any one of claims 1 to 7 is implemented.

Images