A learning-based map-free autonomous navigation obstacle avoidance method, medium and device

Abstract

The application provides a learning-based map-free autonomous navigation obstacle avoidance method, medium and equipment; wherein, the method is: reusable planning capability library is used to carry out map-free autonomous navigation obstacle avoidance for a mobile robot; the reusable planning capability library comprises a global planning module and a local planning module; the global planning module is used to guide the global navigation direction and generate ordered and discrete path points; the local planning module generates real-time actions based on a learning-based local navigation obstacle avoidance model; the mobile robot is controlled to move according to the actions generated by the local planning module, so as to navigate the mobile robot to each path point generated by the global planning in turn. The method combines global and local planning architectures, modularizes algorithm, adapts to different navigation tasks, avoids a large amount of retraining based on a deep reinforcement learning navigation method, and improves the generality and interpretability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of navigation obstacle avoidance, and more particularly to a learning-based map-free autonomous navigation obstacle avoidance method, medium and device. BACKGROUND

[0002] Mobile robot autonomous navigation obstacle avoidance plays an important role in many tasks, but still faces challenges in complex uncertain environments such as post-disaster rescue, dangerous environment exploration and enemy monitoring.

[0003] The basic idea of navigation obstacle avoidance is to plan a collision-free path for an unmanned system to reach a target position, which is usually divided into traditional methods and intelligent methods. The traditional method relies on prior information, i.e. obtaining an environment map through various methods such as SLAM technology before planning, and then planning a path with the known map. The classic planning algorithms are A* and Dijkstra, which can guarantee the shortest path from the starting point to the target point. Although the traditional autonomous navigation obstacle avoidance method is relatively mature, in the above task scenarios, it is difficult to obtain a global map in advance, the detection accuracy is limited, and the dynamic obstacles are unpredictable, etc., which makes the traditional method need a lot of calculation, and the real-time performance is insufficient, which can easily lead to task failure. Therefore, the research of intelligent methods with higher autonomy has gradually become a trend.

[0004] Intelligent methods have strong adaptability to the environment, mainly including group optimization, fuzzy logic and learning-based algorithms. Among them, the group optimization algorithm has high computational complexity and also faces real-time problems. The fuzzy logic algorithm has a small amount of calculation, but it depends heavily on expert experience, so it is easy to fail in uncertain environments. The deep reinforcement learning algorithm does not need to explicitly construct an environment model and solve optimization problems, and directly obtains a navigation strategy according to the sensor input, which has great potential in improving autonomy and dealing with dynamic unknown environments.

[0005] However, the deep reinforcement learning algorithm learns through interaction with the environment, and the navigation task reward is sparse, the algorithm training efficiency is low, and it is even difficult to converge in complex scenarios. Moreover, the current learning-based methods are mostly implemented in static and ideal environments, and the simulation and reality gap is large. In addition, the trained model is usually only applicable to a specific scenario, and has poor scalability, i.e. even a slight change in the navigation task requires retraining of the agent. SUMMARY

[0006] To overcome the shortcomings and deficiencies in the prior art, the purpose of the present application is to provide a learning-based map-free autonomous navigation obstacle avoidance method, medium and equipment; the method realizes map-free dynamic obstacle avoidance navigation to reach the specified target point by improving the double-delay deep deterministic policy gradient (TD3) algorithm; through the planning architecture combining global and local, the fixed-size end-to-end model is suitable for large-scale navigation task scenarios. In order to realize the transfer of the algorithm in the real environment, a changeable random simulation training environment is designed, and the input and output data are optimized; the method can improve the obstacle avoidance performance of mobile robots in dynamic complex environments, the generality of the algorithm model is high, and it can adapt to different navigation tasks without a large amount of repeated training, and provides a safer and more reliable autonomous navigation obstacle avoidance method for non-deterministic complex real environment.

[0007] In order to achieve the above purpose, the present application is realized by the following technical scheme: a learning-based map-free autonomous navigation obstacle avoidance method, which adopts a reusable planning capability library to realize map-free autonomous navigation obstacle avoidance of mobile robots; the reusable planning capability library includes a global planning module and a local planning module;

[0008] The global planning module is used to guide the global navigation direction and generate ordered and discrete path points;

[0009] The local planning module generates real-time actions based on a learning-based local navigation obstacle avoidance model;

[0010] The mobile robot is controlled to move according to the actions generated by the local planning module, so as to navigate the mobile robot to each path point generated by the global planning in turn;

[0011] The local navigation obstacle avoidance model refers to the use of an Actor-Critic architecture; in the Actor-Critic architecture, the Actor network takes the state s i as input, and the action a i as output;

[0012] a i =π(s i ;φ)

[0013] Wherein, i represents the time, and π represents the policy function of the Actor network; φ is the parameter of the Actor network;

[0014] The Critic network is used to evaluate the Actor network:

[0015] q n =Q n (s i ,a i ;θ n )

[0016] Wherein, qn , n = 1, 2, respectively represent the outputs of two Critic networks; Q n represents a function corresponding to the Critic network; θ n is the parameter corresponding to the Critic network.

[0017] Preferably, the training method of the local navigation obstacle avoidance model comprises the following steps:

[0018] Step S1, a local navigation obstacle avoidance simulation environment with obstacles is established; the obstacles include static obstacles and dynamic obstacles of different shapes;

[0019] Step S2, in the local navigation obstacle avoidance simulation environment, when the mobile robot reaches the target point or collision occurs or the maximum set number of steps is reached, the current round ends and the next round is restarted; the local navigation obstacle avoidance model is trained for several rounds to learn the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point, and the parameters of the local navigation obstacle avoidance model are updated;

[0020] For each round, the following sub-steps are included:

[0021] Step S2-1, in the local navigation obstacle avoidance simulation environment, a random starting position of the mobile robot, a target point position, a position and a motion speed of the dynamic obstacle are set;

[0022] Step S2-2, the mobile robot learns the navigation obstacle avoidance strategy under the guidance of the reward function, and the reward function R i is set:

[0023] R i = R g + R c + R a + R p

[0024] Wherein, R g is the reward for reaching the target, R c is the punishment for collision with the obstacle, R a is the action reward; R p is a small time penalty applied to each step within the round when the target point is not reached.

[0025] Step S2-3, at the current time i = t, the state s t is input into the Actor network to obtain the action a t at the current time and a random noise ∈1 is added; the action a t after adding the random noise is:

[0026] a t = clip(π(s t; φ) + ∈1, a min , a max )

[0027]

[0028] where a min represents the minimum action of the mobile robot; a max represents the maximum action of the mobile robot; clip represents a clipping function, which is used to limit the action value in the range of [a min , a max ], and takes the value of a min when the calculated action value is less than the lower limit, takes the value of a max when the calculated action value is greater than the upper limit, and takes the value itself when the calculated action value is in the range of [a min , a max ]; represents a normal distribution with a mean of 0 and a standard deviation of σ1; σ1 represents a constant;

[0029] Step S2-4, after simulating the mobile robot to perform the action a t , the reward function R t is calculated, the state s t+1 and the done flag of the next moment are given, to form an experience tuple (s t , a t , R t , s t+1 , done) and store it in the experience replay pool;

[0030] Step 2-5, network parameter updating is performed, and N experience tuples are randomly extracted from the experience replay pool;

[0031] First, the target value y i is calculated using the target networks of the Actor network and the Critic network, and then the loss function L(θ n ) is minimized using the gradient descent method to update the Critic network parameters θ n :

[0032]

[0033] where θ n ' and φ' represent the parameters of the target networks of the Actor network and the Critic network, respectively; γ represents the discount factor; ∈2 represents Gaussian noise with a mean of 0 and a standard deviation of σ2; c is a small constant value, representing the range limit of clipping ∈2;

[0034] The Actor network parameters are updated using the deterministic policy gradient, and the Q value of the Critic network is maximized using the gradient ascent method; the deterministic policy gradient is as follows:

[0035]

[0036] wherein, is the gradient of the target function J(φ) with respect to the Actor network parameter φ; the gradient is approximated by averaging over N samples in the experience replay buffer; represents the value function Q n with respect to the action a i is calculated by π(s i ; φ). denotes the gradient of the Actor network with respect to the parameter φ;

[0037] The target network parameter θ n ′ and φ′ adopt soft update, and the formula is as follows:

[0038] θ n ′ = τθ n + (1 - τ)θ n ′

[0039] φ′ = τφ + (1 - τ)φ′

[0040] wherein, τ ∈ (0, 1) is a learning rate, and a smaller value is taken.

[0041] Preferably, in the step S2-2, the reward R g for reaching the target, the penalty R c for colliding with the obstacle, the action reward R a , and the small time penalty R p imposed on each step within the round when the target point is not reached are respectively:

[0042]

[0043] R a = α · v t - β · abs(ω t ) + μ · Δd g

[0044] R p = r p

[0045] wherein, d g is the distance between the mobile robot and the target point, δ1 is a threshold value; d o is the distance between the mobile robot and the obstacle, δ2 is a threshold value; v t is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance of the mobile robot and the target point position at the last time and the Euclidean distance at the current time; α, β and μ are set proportional coefficients; rp is a negative constant.

[0046] Preferably, in the Actot-Critic architecture, LSTM networks are additionally arranged between the fully connected layers of the Actor network and between the fully connected layers of the Critic network, respectively.

[0047] The Actor network takes the state s i as input, processes it through fully connected layers and LSTM networks, and outputs the action a i ; The Critic network takes the state s i as input, processes it through fully connected layers and LSTM networks, and then concatenates the action a i output by the Actor network, processes it through fully connected layers and Q-value layers, and obtains the output.

[0048] Preferably, the output action a i of the Actor network is [v i , ω i ]; wherein v i and ω i are the linear velocity and angular velocity output by the Actor network; tanh activation function is used after the Actor network, and then scaled by the maximum linear velocity v max and the maximum angular velocity ω max of the mobile robot, respectively, to obtain the output action a of the local navigation obstacle avoidance model:

[0049] a = [v, ω]

[0050] v = v max v i

[0051] ω = ω max ω i

[0052] wherein v and ω are linear velocity and angular velocity, respectively; the value range of v i and ω i is (-1, 1).

[0053] Preferably, the state s i input to the Actor network includes: current time local perception environment information, local perception environment historical information, action at the last time, and polar coordinates of the target point with the mobile robot as the origin. The local perception environment information and the local perception environment historical information refer to: data detected by the laser sensor on the local environment, data detected by the laser sensor on the local environment at the historical time.

[0054] Preferably, the global planning module generates a global path by employing a global planner in a robot operating system navigation stack; and then down-samples a path point set of the global path using a Ramer-Douglas-Peucker algorithm to extract ordered and discrete path points on the global path.

[0055] A readable storage medium, wherein the storage medium stores a computer program, the computer program causes a processor to execute the learning-based map-free autonomous navigation obstacle avoidance method when the computer program is executed by the processor.

[0056] A computer device, comprising a processor and a memory for storing a program executable by the processor, the processor implements the learning-based map-free autonomous navigation obstacle avoidance method when executing the program stored in the memory.

[0057] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0058] 1. The method can enable an unmanned system to perform map-free obstacle avoidance in a complex dynamic environment, eliminate the dependence on complex rules and prior information based on traditional methods, and improve the obstacle avoidance and navigation performance in a complex uncertain environment, thereby providing more reliable safety guarantee for the navigation task of the unmanned system.

[0059] 2. Different from a pure end-to-end method, the method combines global and local planning to modularize the algorithm, thereby adapting to different navigation tasks and avoiding a large amount of retraining based on a deep reinforcement learning navigation method, and improving the universality and interpretability of the algorithm model.

[0060] 3. The present application considers the reliability of implementation in a real environment, designs a changeable random simulation training environment, optimizes the input and output data, and can realize the direct transfer of the learning-based method from simulation to reality. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a structural diagram of a reusable planning ability library of a navigation task of the map-free autonomous navigation obstacle avoidance method of the present application;

[0062] Figure 2 is a structural diagram of a local navigation obstacle avoidance model of the map-free autonomous navigation obstacle avoidance method of the present application. DETAILED DESCRIPTION

[0063] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0064] Embodiment One

[0065] The embodiment is a learning-based map-free autonomous navigation and obstacle avoidance method, which adopts a reusable planning capability library; through a modular architecture combining global and local planning, the method can be used for navigation tasks in complex uncertain environments, and the generalizability of the reusable planning capability library is high; the learning-based map-free autonomous navigation and obstacle avoidance system at least includes a mobile robot chassis, a laser sensor and an odometer.

[0066] The reusable planning capability library includes a global planning module and a local planning module, as shown in Figure 1 ; following the hierarchical idea of combining global and local planning, the task is divided into global and local planning modules. The global planning module directs the global navigation direction according to the task requirements, and generates ordered and discrete path points. The local planning module uses a learning-based method to respond to real-time changes in the environment, and navigates to each path point provided by the global planning module in turn, thereby autonomously navigating and avoiding obstacles in a complex dynamic environment through an end-to-end manner.

[0067] Preferably, in order to combine the advantages of traditional planning and make full use of existing mature modules, the global planning module directly uses a global planner in the robot operating system (ROS) navigation stack, such as an A* or RRT global planner, to generate a global path according to the task requirements; then the Ramer-Douglas-Peucker (RDP) algorithm is used to downsample the path point set of the global path, thereby retaining the basic shape of the global path and extracting ordered and discrete path points. The global planning module runs at a lower frequency when performing a navigation task.

[0068] The local planning module generates real-time action instructions based on a learning-based local navigation and obstacle avoidance model to navigate to each path point generated by the global planning in turn.

[0069] The local navigation and obstacle avoidance model adopts an Actor-Critic architecture; the Actor-Critic architecture includes an Actor network and a Critic network, as shown in Figure 2 . LSTM networks are additionally added between the fully connected layers of the Actor network and between the fully connected layers of the Critic network; the Actor network takes the state s i as input, processes it through the fully connected layers and the LSTM network, and outputs the action a i ; the Critic network takes the state s i as input, processes it through the fully connected layers and the LSTM network, then concatenates the action a i output by the Actor network, and finally processes it through the fully connected layers and the Q value layer to obtain the output.

[0070] An Actor-Critic architecture is adopted, and a deterministic Actor network is used to handle high-dimensional continuous actions. The input data is preprocessed and then input into an LSTM network to help process historical information and extract features, capture the changing characteristics of dynamic obstacles in the environment, and guide the model to learn obstacle avoidance decisions for unknown dynamic obstacles.

[0071] In the Actor-Critic architecture, the Actor network takes the state s i as input and outputs the action a i . The state s i includes the current local perception environment information, the local perception environment historical information, the action at the previous time, and the polar coordinates of the target point with the mobile robot as the origin. i The action a i is:

[0072] a i = π(s n ; φ)

[0073] where i represents the time, π represents the policy function of the Actor network, and φ is the parameter of the Actor network.

[0074] The Critic network is used to evaluate the performance of the Actor network:

[0075] q n = Q i (s i , a n ; θ n )

[0076] where q n , n = 1, 2, represent the outputs of the two Critic networks, respectively. Q n represents the function of the corresponding Critic network, and θ i is the parameter of the corresponding Critic network.

[0077] The output action a i of the Actor network is [v i , ω i ], where v i and ω max are the linear and angular velocities output by the Actor network, respectively. To maintain consistency with the actions of the mobile robot, a continuous action space is used. To limit the speed range, a tanh activation function is used after the Actor network, and then scaled by the maximum linear velocity v max and the maximum angular velocity ω max of the mobile robot to obtain the output action a of the local navigation and obstacle avoidance model:

[0078] a = [v, ω]

[0079] v = v max v i

[0080] ω = ω max ω i

[0081] wherein v and ω are linear and angular velocities respectively; v i and ω i are in the range (-1, 1).

[0082] The training method of the local navigation obstacle avoidance model comprises the following steps:

[0083] Step S1, a local navigation obstacle avoidance simulation environment with obstacles is established; the obstacles include static obstacles and dynamic obstacles of different shapes; the density of obstacles in different sub-regions of the simulation environment is different. In order to reduce the gap between simulation and reality, the simulator needs to be equipped with a high-performance physics engine. Starting from the position of the mobile robot, the path points generated by the global planning module are taken as the target points, and the local navigation obstacle avoidance simulation environment is interacted to learn the navigation obstacle avoidance strategy.

[0084] Step S2, the local navigation obstacle avoidance model is trained for several rounds in the local navigation obstacle avoidance simulation environment, the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point is learned, and the parameters of the local navigation obstacle avoidance model are updated.

[0085] When the mobile robot reaches the target point or collides or runs to the maximum set number of steps, the current round ends and the next round starts; the local navigation obstacle avoidance model interacts with the local navigation obstacle avoidance simulation environment, and is guided by the reward function R i , aiming to maximize the expected value of the discounted return U t , to learn the navigation obstacle avoidance strategy.

[0086] Specifically, the discounted return U t is:

[0087]

[0088] wherein γ ∈ [0, 1] is a discount factor.

[0089] The expected value of the discounted return U t is the action value function Q π , and the formula is:

[0090]

[0091] wherein Q π represents the expected return when the strategy π is used in the state s t and the action a tThe expected discounted return that can be obtained later.

[0092] For each episode, the following steps are included:

[0093] Step S2-1, in the local navigation obstacle avoidance simulation environment, randomly set the starting position of the mobile robot, the target point position, the position and motion speed of the dynamic obstacle; the position of the mobile robot, the target point position and the dynamic obstacle are set in the unoccupied area of the local navigation obstacle avoidance simulation environment; in order to improve the training efficiency, in the first M episodes, that is, in the early stage of training, the target point position is set near the mobile robot, and the distance between the target point position and the mobile robot position is <2 meters; so that the mobile robot meets the target as soon as possible to obtain high rewards, helping the local navigation obstacle avoidance model to learn the correct strategy, and canceling the restriction after a certain number of episodes, and randomly generating.

[0094] Step S2-2, set the reward function R i :

[0095] R i =R g +R c +R a +R p

[0096] Wherein, R g is the reward for reaching the target, R c is the penalty for collision with obstacles, R a is the action reward, which is used to optimize the trajectory of the mobile robot, encourage forward and towards the target direction, and punish rapid turning; R p is a small time penalty applied to each step within the episode when the target point is not reached, which is used to guide the mobile robot to reach the target as soon as possible. Since reaching the target point and collision belong to sparse rewards, dense rewards R a and R p are designed to improve training efficiency.

[0097] The reward R g for reaching the target, the penalty R c for collision with obstacles, the action reward R a , and the small time penalty R p applied to each step within the episode when the target point is not reached are as follows:

[0098]

[0099] R a =α·v t -β·abs(ω t )+μ·Δd g

[0100] Rp = r p

[0101] wherein d g is the distance between the mobile robot and the target point, δ1 is a threshold value; when d g is less than the threshold value δ1, it is considered to reach the target point and a reward is given; d o is the distance between the mobile robot and the obstacle, δ2 is a threshold value; when d o is less than the threshold value δ2, it is considered to collide with the obstacle and a penalty is given; v t is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance of the mobile robot and the target point position at the previous time and the Euclidean distance at the current time; α, β and μ are the set proportional coefficients; r p is a negative constant.

[0102] Step S2-3, at the current time i = t, the state s t is input into the Actor network to obtain the action a t at the current time, and a random noise ∈1 is added; in the training process, in order to ensure the exploratory of the deterministic policy, the actually executed action a t also needs to add a random noise ∈1, and the action a t after adding the random noise is:

[0103] a t = clip(π(s t ; φ) + ∈1, a min , a max )

[0104]

[0105] wherein a min represents the minimum action of the mobile robot; a max represents the maximum action of the mobile robot; clip represents a clipping function, which is used to limit the action value in [a min , a max ], when the calculated action value is less than the lower limit, the value is a min , and when the calculated action value is greater than the upper limit, the value is a max , and when the calculated action value is in the range of [a min , a max ], the value is itself; represents a normal distribution with a mean of 0 and a standard deviation of σ1; σ1 represents a constant;

[0106] Step S2-4, after the mobile robot executes the action a t , the reward function R is calculatedt , the state s of the next moment t+1 and the done flag, forming experience tuples (s t , a t , R t , s t+1 , done) and storing them in the experience replay pool;

[0107] Step 2-5, network parameter update is performed, and N experience tuples are randomly extracted from the experience replay pool;

[0108] First, the target value y i is calculated using the target network of the Actot network and the Critic network n , and then the loss function L(θ n ) is minimized using the gradient descent method to update the Critic network parameters θ n :

[0109]

[0110]

[0111] where θ max ' and φ' represent the parameters of the target network of the Actor network and the Critic network, respectively, ∈2 represents Gaussian noise with a mean of 0 and a standard deviation of σ2; c is a small constant value, representing the range limit of clipping ∈2; Gaussian noise ∈2 is introduced for target policy smoothing, so that similar actions have similar values, allowing the Critic network to make smooth estimates of nearby actions and reduce overfitting to a single action. Moreover, the target action with added noise needs to be clipped according to the maximum action a min and the minimum action a i of the real mobile robot to ensure that it is a valid action.

[0112] The Actor network parameters are updated using the deterministic policy gradient, which maximizes the expected discounted return, i.e., the Q value of the Critic network, using the gradient ascent method. During updating, delayed updating is used, where the Critic network is updated multiple times before the Actor network is updated, thereby improving the stability of training. The deterministic policy gradient is as follows:

[0113]

[0114] where is the gradient of the objective function J(φ) with respect to the Actor network parameters φ; the average of N samples in the experience replay pool is taken to approximate the gradient; represents the gradient of the Q value function with respect to the action, and a is calculated from π(s i ; φ); denotes the gradient of the Actor network with respect to the parameter φ;

[0115] target network parameter θ n and φ' are updated softly, as follows:

[0116] θ n ' = τθ n + (1-τ)θ n '

[0117] φ' = τφ + (1-τ)φ'

[0118] where τ ∈ (0, 1) is the learning rate, which is taken to be small.

[0119] Embodiment Two

[0120] The embodiment provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the learning-based map-free autonomous navigation and obstacle avoidance method in the embodiment one.

[0121] Embodiment Three

[0122] The embodiment provides a computer device, which comprises a processor and a memory for storing a program executable by the processor, and the processor, when executing the program stored in the memory, implements the learning-based map-free autonomous navigation and obstacle avoidance method in the embodiment one.

[0123] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application shall be equivalent replacement modes and shall be included in the protection scope of the present application.

Claims

1. A learning-based map-less autonomous navigation and obstacle avoidance method, characterized by: A reusable planning capability library is used to enable mobile robots to navigate and avoid obstacles without maps. The reuse planning capability library includes global planning modules and local planning modules; The global planning module is used to guide the global navigation direction and generate orderly and discrete path points; The local planning module generates real-time actions based on the learned local navigation and obstacle avoidance model; Control the movement of the mobile robot according to the actions generated by the local planning module to navigate the mobile robot to each path point generated by the global planning in sequence; The local navigation obstacle avoidance model refers to the use of Actor-Critic architecture; in the Actor-Critic architecture, the Actor network is based on the state s i is input, action a i is the output; a i =π(s i ;f) Where i represents the time, π represents the policy function of the Actor network; φ is the parameter of the Actor network; The Critic network is used to evaluate the quality of the Actor network: q n =Q n (s i ,a i ;θ n ) Among them, q n ,n=1,2, respectively represent the outputs of the two Critic networks; Q n Represents the function corresponding to the Critic network; θ n is the parameter corresponding to the Critic network; In the training of the local navigation obstacle avoidance model, for each round, the reward function R is set i : R i =R g +R c +R a +R p Among them, R g is the reward for reaching the goal, R c is the penalty for collision with an obstacle, R a is the action reward; R p It is a small time penalty imposed on each step in the round when the target point is not reached; Reward R for reaching the goal g , Penalty R for collision with obstacles c , action reward R a , a small time penalty R imposed on each step in the round when the target point is not reached p They are: R a =α·v t -β·abs(ω t )+μ·Δd g R p ＝r p Among them, d g is the distance between the mobile robot and the target point, δ1 is the threshold; d o is the distance between the mobile robot and the obstacle, δ2 is the threshold; v t is the current linear velocity of the mobile robot; ω t is the current angular velocity of the mobile robot; Δd g is the difference between the Euclidean distance between the mobile robot and the target point at the previous moment and the Euclidean distance at the current moment; α, β and μ are the set proportional coefficients; r p is a negative constant.

2. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The training method of the local navigation obstacle avoidance model comprises the following steps: Step S1: establishing a local navigation obstacle avoidance simulation environment with obstacles; the obstacles include static obstacles and dynamic obstacles of different shapes; Step S2: In the local navigation obstacle avoidance simulation environment, when the mobile robot reaches the target point, collides, or runs to the maximum set number of steps, the current round ends and the next round begins again; the local navigation obstacle avoidance model is trained for several rounds to learn the navigation obstacle avoidance strategy of the mobile robot from the starting position to the target point, and the parameters of the local navigation obstacle avoidance model are updated; Each round includes the following steps: Step S2-1: In a local navigation obstacle avoidance simulation environment, set a random mobile robot starting position, target point position, dynamic obstacle position, and movement speed; Step S2-2: Set the reward function R i : R i =R g +R c +R a +R p Among them, R g is the reward for reaching the goal, R c is the penalty for collision with an obstacle, R a is the action reward; R p It is a small time penalty imposed on each step in the round when the target point is not reached; Step S2-3, at the current time i=t, change the state s t Input the Actor network and get the current action a t , and add random noise∈1; action a after adding random noise t for: a t =clip(π(s t ;φ)+∈1,a min ,a max ) Among them, a min Represents the minimum action of the mobile robot; a max represents the maximum motion of the mobile robot; clip represents the clipping function; represents a normal distribution with a mean of 0 and a standard deviation of σ1; σ1 represents a constant; Step S2-4, simulate the mobile robot to perform action a t After that, calculate the reward function R t , give the next moment state s t+1 and done flag, forming an experience tuple (s t ,a t ,R t ,s t+1 ,done), and stored in the experience replay pool; Step 2-5: Update the network parameters and randomly extract N experience tuples from the experience replay pool; first calculate the target value y using the target network of the Actor network and the Critic network i Then use the gradient descent method to minimize the loss function L(θ n ), to update the Critic network parameters θ n : Among them, θ n ′ and φ′ represent the parameters of the target network corresponding to the Actor network and the Critic network respectively; γ represents the discount factor; ∈2 represents Gaussian noise with mean 0 and standard deviation σ2; c represents the range limit of clipping ∈2; The deterministic policy gradient is used to update the Actor network parameters, and the gradient ascent method is used to maximize the Q value of the Critic network; the deterministic policy gradient is as follows: in, is the gradient of the objective function J(φ) with respect to the Actor network parameter φ; the gradient is approximated by taking the average of N samples in the experience replay pool; represents the gradient of the Q-value function with respect to the action, and a is given by π(s i ;φ) calculation; represents the gradient of the Actor network with respect to the parameter φ; Target network parameters θ n ′ and φ′ are soft updated, and the formula is as follows: i n ′=τθ n +(1-τ)θ n ′ φ′=τφ+(1-τ)φ′ Among them, τ∈(0,1) is the learning rate, which takes a smaller value.

3. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: In the Actor-Critic architecture, LSTM networks are added between the fully connected layers of the Actor network and the Critic network respectively. The Actor network is in state s i As input, it is processed by the fully connected layer and LSTM network and outputs action a i ; The Critic network is in state s i is input, processed by the fully connected layer and LSTM network, and then combined with the action a output by the Actor network i After concatenation, the output is obtained through the fully connected layer and the Q value layer.

4. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The output action a of the Actor network i =[v i ,ω i ]; Among them, v i and ω i are the linear velocity and angular velocity output by the Actor network respectively; after the Actor network, the tanh activation function is used, and then the maximum linear velocity v of the mobile robot is used. max and the maximum angular velocity ω max Scaling is performed to obtain the output action a of the local navigation obstacle avoidance model: a=[v,ω] v=v max v i oh = oh max oh i Among them, v and ω are linear velocity and angular velocity respectively; v i and ω i The value range is (-1,1).

5. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The state s of the Actor network input i It includes: local perception environment information at the current moment, local perception environment history information, action at the previous moment and polar coordinates of the target point with the mobile robot as the origin.

6. The map-free autonomous navigation and obstacle avoidance method based on learning according to claim 1, characterized in that: The global planning module uses the global planner in the robot operating system navigation stack to generate a global path; then uses the Ramer-Douglas-Peucker algorithm to downsample the path point set of the global path to extract ordered, discrete path points on the global path.

7. A readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the learning-based map-free autonomous navigation and obstacle avoidance method according to any one of claims 1 to 6.

8. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the map-free autonomous navigation and obstacle avoidance method based on learning described in any one of claims 1 to 6 is implemented.

Images