Robot navigation control method and system based on partially observable reinforcement learning

Abstract

The present invention discloses a robot navigation control method and system based on partially observable reinforcement learning, which are mainly used in the navigation task of the robot in an uncertain environment with unknown models. In order to complete the navigation task in an uncertain environment, the present invention adopts a reinforcement learning algorithm in a partially observable environment. The system includes a filtering unit, a planning unit, a replay pool, and a learning unit. In the present invention, state particles are used to represent belief states to reduce the computational complexity of belief state updates, simulation planning based on learned models is used to improve sample utilization, a resampling method is used to prevent particle degradation problems, and reward shaping based on negative information entropy of belief states is used to improve the training efficiency and stability of the algorithm in navigation tasks with sparse rewards. The present invention can achieve efficient and stable strategy learning in a partially observable environment with unknown models, and use the learned strategy in actual robot navigation tasks.

Description

Technical Field

[0001] The present invention relates to a robot navigation control method and system based on reinforcement learning in a partially observable environment, belonging to the technical field of robot control. Background Art

[0002] With the development of technology, robots have been widely used in various fields of production and life, and the various application scenarios that follow have also brought more new challenges to robotics. Robot navigation is one of the most important tasks in the field of robot control. There are a large number of robot navigation and control requirements in actual application scenarios, such as sweeping robots, storage and transportation robots, search and rescue robots, etc. Most traditional robot navigation algorithms require accurate modeling of the environment, which greatly limits the scope of application of the algorithm. Reinforcement learning can learn control strategies from data generated by interaction with the environment, so it is increasingly used in robot navigation tasks.

[0003] The environment in which the robot is located is usually very complex. Due to factors such as obstacles and the detection range of sensors, the robot can only obtain partial information about the environment through sensors. The decision-making task under incomplete information is much more difficult than that under complete information. At the same time, due to the limitations of the robot's sensor performance, the information obtained by the sensor itself is noisy, and the uncertainty caused by the noise will also interfere with the robot's decision-making. Therefore, how to control the robot in an uncertain environment is an urgent problem to be solved in the field of robot navigation.

[0004] Existing partially observable reinforcement learning algorithms often fail to effectively encourage robots to take actions to obtain environmental information, and it is difficult to obtain the optimal strategy in tasks where environmental information is critical. In addition, when performing navigation tasks, robots usually only receive rewards when they reach the target point, so it is an environment with sparse rewards. Existing partially observable reinforcement learning algorithms have slow training speeds and unstable algorithm performance in environments with sparse rewards. Summary of the invention

[0005] Purpose of the invention: In view of the common problems of existing robot navigation technology in uncertain environments, the present invention provides a robot navigation control method and system based on partially observable reinforcement learning. The robot navigation task is modeled as a partially observable Markov decision process (POMDP), and a reinforcement learning algorithm in a partially observable environment is used to solve this problem. This method effectively solves the problem of sparse rewards when using reinforcement learning to handle robot navigation tasks, and implicitly encourages the robot to take the initiative to obtain environmental information in a partially observable environment, thereby obtaining a better strategy and improving the efficiency and stability of the navigation control method.

[0006] Technical solution: A robot navigation control method based on partially observable reinforcement learning, specifically comprising the following steps:

[0007] S1. Initialize network parameters, including: transfer model D ψ The parameter ψ of the observation model Z θ The parameters θ of the policy network π ρ The parameter ρ of the double Q value network Q ω Parameter ω. Set the training time step counter t = 0 and enter S2;

[0008] S2. Generate K weighted belief state particles based on the initial state prior Initial weight are all set to 1, the robot obtains the initial observation o1 through the sensor and enters S3;

[0009] S3, if the training time step counter t is less than the maximum training step number L, then t←t+1, enter S4; otherwise enter S27;

[0010] S4, the robot observes the model Z θ (s,o) Update weights Enter S5;

[0011] S5. Calculate the average belief state Enter S6;

[0012] S6. Sampling The M particles with the largest weight in Enter S7;

[0013] S7. Normalize the weights of M particles Enter S8;

[0014] S8, the particles and the average belief state After combining, make N copies and assign each copy a weight. Get N new weighted particles, denoted as The superscript (n) indicates the nth replication, which proceeds to S9;

[0015] S9, set the planning time step counter i=t-1, and enter S10;

[0016] S10, if the planning time step counter is less than the maximum planning step number H, then i←i+1, enter S11; otherwise enter S19;

[0017] S11. For each copy, get the action according to the policy network Enter S12;

[0018] S12. For each particle in each copy, according to the transfer model D ψ Get the next moment status and reward Enter S13;

[0019] S13. For each copy, update the average belief state Enter S14;

[0020] S14. For each copy, find an estimate of the information entropy of the belief state Indicates the estimation of the current belief state, and enters S15;

[0021] S15. Update the weight of each replicated particle A (m)(n) represents the advantage function, and enters S16;

[0022] S16, if resampling is required, go to S17; otherwise go to S18;

[0023] S17, resample and copy particles, and enter S18;

[0024] S18, enter S10;

[0025] S19, uniformly sample n from 1 to N, and output the first action a in the robot planning trajectory of the nth copy t , enter S20;

[0026] S20, the robot takes action a t , interact with the training environment to get the next state s t+1 , the observation o at the next moment t+1 and reward r t , enter S21;

[0027] S21, if resampling is required, enter S22; otherwise enter S23;

[0028] S22, resample belief state particles and enter S23;

[0029] S23. Update belief state particles according to the transfer model Enter S24;

[0030] S24, the data Store in the replay pool and enter S25;

[0031] S25, the learning unit samples training data from the playback pool, updates network parameters, and enters S26;

[0032] S26, enter S3;

[0033] S27, end the training, output the trained network for robot navigation control. Cancel the playback pool and learning unit, skip steps S24 and S25 to get the specific steps of the robot navigation control use stage. At this time, the environment in S20 only needs to provide observation and reward information, but not real state information.

[0034] In the above technical solution, the environment (training environment) in which the robot is located is modeled as a POMDP, which can be represented by the following six-tuple:

[0035] (7) State space S, s t ∈S represents the state of the robot at time t;

[0036] (8) Action space A, a t ∈A represents the action taken by the robot at time t;

[0037] (9) Transition probability function T:S×A×S→[0,1], T(s t ,a t ,s t+1 ) indicates that the robot is in state s t Take action a t Transfer to t+1 The probability of

[0038] (10) Reward function R:S×A→,R(s t ,a t ) indicates that the robot is in state s t Take action a t The immediate rewards you can get;

[0039] (11) Observation space O, o t ∈O represents the observation obtained by the robot at time t;

[0040] (12) Observation probability function Z: S×A×O→[0,1], Z(s t ,a t-1 ,ot ) indicates that the robot is taking action a t-1 Transfer to t Get observation o t probability.

[0041] The goal of POMDP is to obtain a strategy π:H→A based on the historical action observation sequence to maximize the expected cumulative reward, the cumulative reward G t is defined as:

[0042]

[0043] Among them, γ∈(0,1] is a discount factor used to weigh the immediate reward and delayed reward. t Represents the reward obtained by the robot at time t.

[0044] In the above technical solution, the belief state b t (s) = p(s t =s|h t ), indicating the known history h t ={b0,a0,o1,…,a t-1 ,o t}, the hidden state s t is the probability distribution of s, and b0 represents the probability distribution of the initial state.

[0045] In the above technical solution, in S1, the network includes:

[0046] Transfer Model D ψ , ψ is the parameter of the transfer model;

[0047] Observation Model Z θ , θ is the parameter of the observation model;

[0048] Policy Network π ρ , ρ is the parameter of the policy network;

[0049] Double Q value network Q ω , ω is the parameter of the double Q value network.

[0050] The transfer model D ψ It is used to update the state particles in the filter unit and simulate in the planning unit. The input is the state and action, and the output is the state and reward at the next moment. The transfer model network structure is a fully connected network; the observation model Z θ Used to update particle weights in the filter unit. The input is state and observation, the output is the probability of observation, and the observation model network structure is a fully connected network; the strategy network π ρIt is used to provide strategies for the robot simulation in the planning module. The input is the belief state particle and the average belief state. The output is the action and the logarithm of the probability of outputting the action. The network structure of the strategy network is to output the mean μ and variance σ of the action through a fully connected network. 2 , and then from the Gaussian distribution N(μ,σ 2 ) to obtain the action, and use Gaussian distribution to calculate the logarithmic value of the probability of the output action; the double Q value network Q ω It is used to update particle weights in the planning module. The input is state and action, and the output is two Q values. The dual Q value network is two fully connected networks Q1 and Q2. i (i=1,2) each maintains a target Q value network TQ with the same network structure i (i=1,2), used for network parameter update.

[0051] In the above technical solution, in S2, the belief state is represented by weighted particles, which is a common approximate method for dealing with the problem of high computational complexity of belief state update. The process of updating particles is called particle filtering or temporal Monte Carlo method.

[0052] In the above technical solution, S7-S19 are planning units, and the robot uses multiple replicated belief state particles to perform simulation planning separately to select the optimal action.

[0053] In the above technical solution, in S14, in the estimation of the information entropy of the belief state, the belief state particle Estimating belief state probability distribution When , the kernel density estimation method (KDE) is used, and the Gaussian kernel density is used to estimate the probability distribution of belief states.

[0054] In the above technical solution, in S15, the calculation formula of the advantage function A is:

[0055]

[0056]

[0057] Where TD represents the timing differential error, Q ω is the smaller value of the two outputs of the dual Q value network.

[0058] When calculating the advantage function A, the reward is added with reward shaping based on the negative information entropy of the belief state, which encourages the robot to take actions to obtain information and improves the efficiency and stability of the algorithm.

[0059] In the above technical solution, in S17 and S22, the resampling is a technique commonly used in particle filtering to prevent particle degradation. Specifically, N weighted particles are randomly and repeatedly selected N times according to the weights, and then the weights of the new particles are all set to 1.

[0060] In the above technical solution, in S25, updating the network parameters includes:

[0061] (4) The transfer model and the observation model use the minimum mean square error between the predicted value and the true value as the loss function, and use the prescribed optimization method, such as stochastic gradient descent, Adam, etc., to optimize the loss function and update the network parameters.

[0062] (5) The two networks of the dual Q-value network are updated in the same way, both based on the temporal-difference (TD) error with the target Q-value network:

[0063]

[0064] Where α is the temperature coefficient, which controls the emphasis on policy entropy. Use the specified optimization method, such as stochastic gradient descent, Adam, etc., to optimize the loss function and update the network parameters. In addition, the Q value network parameters are copied to the target Q value network every fixed update step.

[0065] (6) The strategy function is updated based on the loss function:

[0066]

[0067] Where α is the temperature coefficient, Q ω The smaller value of the two outputs of the dual Q value network. Use the prescribed optimization method, such as stochastic gradient descent, Adam, etc., to optimize the loss function and update the network parameters.

[0068] To achieve the above-mentioned purpose of the invention, the robot navigation control system based on partially observable reinforcement learning of the present invention includes: a filtering unit, a planning unit, a playback pool, and a learning unit.

[0069] The filter unit is used to update the belief state particles and their weights, and use the actions obtained from the planning unit to interact with the training environment to obtain state, observation, and reward information. It is also used to process training data and store it in the replay pool.

[0070] The planning unit is used to receive the weighted particles provided by the filtering module, use the learned transfer model and strategy network to simulate planning, and output actions to the filtering unit.

[0071] The replay pool is a database that supports random access and is used to store processed training data and provide the learning unit with the training data required for learning, that is, the data set composed of tuples stored in the replay pool by the sampling and filtering unit.

[0072] The learning unit is used to sample the training data in the revisit pool, train the network using a given optimization method, and provide the updated network parameters to the filtering unit and the planning unit.

[0073] In the above system, the training environment, i.e., the actual application environment or the highly simulated virtual environment, is used for training the robot navigation control method, interacts with the filtering unit, and provides state, observation, and reward information for the filtering.

[0074] Based on the above technical solution, the neural network can be trained for practical use. The specific steps of the use phase can be obtained by canceling the playback pool and the learning unit and skipping steps S24 and S25. At this time, the environment in S20 only needs to provide observation and reward information, but does not need to provide real state information.

[0075] Beneficial effects: Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0076] The present invention uses reinforcement learning to process robot navigation control tasks, and can learn control strategies from data generated by interaction with the environment, thereby avoiding the problem that traditional control methods require precise modeling of the environment and expanding the application scope of the control method.

[0077] The present invention models the environment as a POMDP problem, and can define the uncertainty in the environment. Traditional methods are difficult to handle tasks with occlusion, limited sensor detection range, and sensor noise, and the present invention can effectively handle navigation control tasks in such environments.

[0078] The present invention adopts a model-based partially observable reinforcement learning algorithm, which can improve the utilization rate of training samples and enhance the training efficiency.

[0079] The present invention adopts reward shaping based on potential energy function, which can effectively deal with the problem of sparse rewards in robot navigation tasks in reality without changing its optimal strategy, thereby improving the algorithm training efficiency and stability.

[0080] The present invention adopts the negative information entropy of belief state as the potential energy function in the above-mentioned reward shaping method, which can encourage the robot to take actions to obtain information, and it is easier to obtain the optimal strategy than the traditional control method. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is an overall training framework diagram of an embodiment of the present invention;

[0082] Figure 2It is a unit interaction diagram of the training phase of an embodiment of the present invention;

[0083] Figure 3 It is a unit interaction diagram of the use phase of the embodiment of the present invention;

[0084] Figure 4 This is a diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0086] Figure 4 This is a top view of a robot's navigation environment. The robot is in one of the two rooms on the left and right that are exactly the same size. The robot's state is its absolute coordinates in the entire house. The robot can take actions with limited speed in any direction. The robot is equipped with sensors in four positive directions: up, down, left, and right. The sensors can obtain the distance from the robot to the nearest wall in that direction, and there is Gaussian noise. The robot's initial position is random. Its goal is to reach the charging point below the left room or above the right room. The robot will receive a reward of +100 after reaching the target position. The robot can only obtain observations when in use, so the robot cannot determine which room it is in, unless it reaches the shaded part of the figure, in which case it can determine the room it is in by the change in the distance to the wall.

[0087] The steps of the robot in the training phase are as follows:

[0088] S1. Initialize network parameters, including: transfer model D ψ The parameter ψ of the observation model Z θ The parameters θ of the policy network π ρ The parameter ρ of the double Q value network Q ω Parameter ω. Set the training time step counter t = 0 and enter S2;

[0089] S2. Generate K = 100 weighted belief state particles based on the prior of the initial state Initial weight are all set to 1, the robot obtains the initial observation o1 through the sensor and enters S3;

[0090] S3. If the training time step counter t is less than the maximum training step number L=10,000, then t←t+1, and enter S4; otherwise, enter S27;

[0091] S4, the robot observes the model Z θ (s,o) Update weights Enter S5;

[0092] S5. Calculate the average belief state Enter S6;

[0093] S6. Sampling The particles with the largest weight in the equation are M = 3, denoted as like Figure 2 , 3 The filter unit inputs the weighted particles to the planning unit to enter S7;

[0094] S7. Normalize the weights of M particles Enter S8;

[0095] S8, the particles and the average belief state After combining, copy N=30 copies and assign each copy a weight Get N new weighted particles, denoted as The superscript (n) indicates the nth replication, which proceeds to S9;

[0096] S9, set the planning time step counter i=t-1, and enter S10;

[0097] S10, if the planning time step counter is less than the maximum planning step number H=10, then i←i+1, and enter S11; otherwise, enter S19;

[0098] S11. For each copy, get the action according to the policy network Enter S12;

[0099] S12, for each particle in each copy, according to the transfer model D ψ Get the next moment status and reward Enter S13;

[0100] S13. For each copy, update the average belief state Enter S14;

[0101] S14. For each copy, find an estimate of the information entropy of the belief state Indicates the estimation of the current belief state, and enters S15;

[0102] S15. Update the weight of each replicated particle A (m)(n) represents the advantage function, and enters S16;

[0103] S16, if resampling is required, go to S17; otherwise go to S18;

[0104] S17, resample and copy particles, and enter S18;

[0105] S18, enter S10;

[0106] S19, uniformly sample n from 1 to N, and output the first action a in the planned trajectory of the nth replica robot t ,like Figure 2 , 3 The planning unit inputs the filter unit to enter S20;

[0107] S20, the robot takes action a t , interact with the training environment to get the next state s t+1 , the observation o at the next moment t+1 and reward r t , such as 2 Lieutenant Generals t+1 , o t+1 , r t Input to the filter unit and enter S21

[0108] S21, if resampling is required, enter S22, otherwise enter S23;

[0109] S22, resample belief state particles and enter S23;

[0110] S23. Update belief state particles according to the transfer model Enter S24;

[0111] S24, the data Deposit Figure 2 In the playback pool, enter S25;

[0112] S25, Figure 2 The learning unit in the process samples training data from the playback pool, updates the network parameters, and transmits the updated network parameters to the filtering unit and the planning unit, and enters S26;

[0113] S26, enter S3;

[0114] S27, end the training, and output the trained network for robot navigation control. Figure 3 , cancel the playback pool and the learning unit, skip steps S24 and S25 to obtain the specific steps of the robot navigation control use phase. At this time, the environment in S20 only needs to provide observation and reward information to the filtering module without providing real state information.

[0115] The entire training process framework diagram is referenced Figure 1 .

[0116] In the above embodiment, in S1, the network includes a transfer model D ψ, ψ is the parameter of the transfer model; the observation model Z θ , θ is the parameter of the observation model; the policy network π ρ , ρ is the parameter of the policy network; the double Q value network Q ω , ω is the parameter of the double Q value network. The transfer model D ψ The input is state and action, and the output is the state and reward at the next moment. The transfer model network structure is a 4-layer fully connected network, and the number of neurons in each layer is 256 / 256 / 256 / 3 respectively; the observation model Z θ The input is state and observation, the output is the probability of observation, the observation model network structure is a 4-layer fully connected network, and the number of neurons in each layer is 256 / 256 / 256 / 1; the policy network π ρ The input is the belief state particle and the average belief state, the output is the action and the logarithm of the probability of outputting the action, and the network structure of the policy network is to output the mean μ and variance σ of the action through the fully connected network 2 , and then from the Gaussian distribution N(μ,σ 2 ) to obtain the action, and use Gaussian distribution to calculate the logarithm of the probability of the output action. It is a 3-layer fully connected network with 256 / 256 / 4 neurons in each layer; the double Q value network Q ω The input is state and action, and the output is two Q values. The dual Q value network is two fully connected networks Q1 and Q2, both of which are 3-layer fully connected networks. The number of neurons in each layer is 256 / 256 / 1. At the same time, Q i (i=1,2) each maintains a target Q value network TQ with the same network structure i (i=1,2), used for network parameter update. Parameter initialization uses the Pytorch default parameter initialization method.

[0117] In the above embodiment, S10-S19 are planning units, and the robot uses the replicated multiple belief state particles to perform simulation planning respectively, so as to select the optimal action.

[0118] In the above embodiment, in S14, the belief state information entropy is estimated by using the belief state particle Estimating belief state probability distribution When , the multivariate Gaussian kernel density estimation method with Silverman empirical window width is used. The formula for kernel density estimation is:

[0119]

[0120] Where D is the dimension of the state, the window width matrix H is a diagonal matrix, and the calculation formula for the elements on the main diagonal is:

[0121]

[0122] in is the standard deviation of the state particles in dimension i.

[0123] In the above embodiment, in S15, the calculation formula of the advantage function A is:

[0124]

[0125]

[0126] Where TD represents the timing differential error, Q ω is the smaller value of the two outputs of the dual Q-value network. When calculating the advantage function A, the reward is added with the reward shaping based on the negative information entropy of the belief state, which encourages the robot to take actions to obtain information and improves the efficiency and stability of the algorithm.

[0127] In the above embodiment, in S17 and S22, the resampling is a technique commonly used in particle filtering to prevent particle degradation. Specifically, N weighted particles are randomly and repeatedly selected N times according to the weights, and then the weights of the new particles are all set to 1.

[0128] In the above embodiment, in S25, updating the network parameters includes:

[0129] (1) The transfer model and the observation model use the minimum mean square error between the predicted value and the true value as the loss function, use the Adam optimization method to optimize the loss function, and update the network parameters with a learning rate of 0.001.

[0130] (2) The two networks of the dual Q-value network are updated in the same way, both based on the temporal-difference (TD) error with the target Q-value network:

[0131]

[0132] Among them, α=1 is the temperature coefficient, which controls the emphasis on policy entropy, and γ=0.95 is the discount factor. The Adam optimization method is used to optimize the loss function, with a learning rate of 0.001, and the network parameters are updated. In addition, the Q value network parameters are copied to the target Q value network every 5 steps.

[0133] (3) The strategy function is updated based on the loss function:

[0134]

[0135] Among them, α = 1 is the temperature coefficient, Q ω is the smaller value of the two outputs of the dual Q value network. Use the Adam optimization method to optimize the loss function, with a learning rate of 0.001, and update the network parameters.

[0136] The following is the specific structure of this embodiment, including: a filtering unit, a planning unit, a playback pool, and a learning unit.

[0137] The filter unit is used to update the belief state particles and their weights, and use the actions obtained from the planning unit to interact with the training environment to obtain state, observation, and reward information. It is also used to process training data and store it in the replay pool.

[0138] The planning unit is used to receive the weighted particles provided by the filtering module, use the learned transfer model and strategy network to simulate planning, and output actions to the filtering unit.

[0139] The replay pool is a database that supports random access and is used to store processed training data and provide the learning unit with the training data required for learning, that is, the data set composed of tuples stored in the replay pool by the sampling and filtering unit.

[0140] The learning unit is used to sample the training data in the revisit pool, train the network using a given optimization method, and provide the updated network parameters to the filtering unit and the planning unit.

[0141] After the training phase is completed, the replay pool and the learning unit are canceled, and steps S24 and S25 are skipped to obtain the specific steps of the use phase. At this time, the environment in S20 only needs to provide observation and reward information, but does not need to provide real state information.

Claims

1. A robot navigation control method based on partially observable reinforcement learning, characterized in that: The steps include: S1. Initialize network parameters, including: transfer model D ψ The parameter ψ of the observation model Z θ The parameters θ of the policy network π ρ The parameter ρ of the double Q value network Q ω Parameter ω; set the training time step counter t = 0 and enter S2; S2. Generate K weighted belief state particles based on the initial state prior Initial weight are all set to 1, the robot obtains the initial observation o1 through the sensor and enters S3; S3, if the training time step counter t is less than the maximum training step number L, then t←t+1, enter S4; otherwise enter S27; S4, the robot observes the model Z θ (s,o) Update weights Enter S5; S5. Calculate the average belief state Enter S6; S6. Sampling The M particles with the largest weight in Enter S7; S7. Normalize the weights of M particles Enter S8; S8, the particles and the average belief state After combining, make N copies and assign each copy a weight. Get N new weighted particles, denoted as The superscript (n) indicates the nth replication, which proceeds to S9; S9, set the planning time step counter i=t-1, and enter S10; S10, if the planning time step counter is less than the maximum planning step number H, then i←i+1, enter S11; otherwise enter S19; S11. For each copy, get the action according to the policy network Enter S12; S12. For each particle in each copy, according to the transfer model D ψ Get the next moment status and reward Enter S13; S13. For each copy, update the average belief state Enter S14; S14. For each copy, find an estimate of the information entropy of the belief state Indicates the estimation of the current belief state, and enters S15; S15. Update the weight of each replicated particle A (m)(n) represents the advantage function, and enters S16; S16, if resampling is required, go to S17; otherwise go to S18; S17, resample and copy particles, and enter S18; S18, enter S10; S19, uniformly sample n from 1 to N, and output the first action a in the robot planning trajectory of the nth copy t , enter S20; S20, the robot takes action a t , interact with the training environment to get the next state s t+1 , the observation o at the next moment t+1 and reward r t , enter S21; S21, if resampling is required, enter S22; otherwise enter S23; S22, resample belief state particles and enter S23; S23. Update belief state particles according to the transfer model Enter S24; S24, the data Store in the replay pool and enter S25; S25, the learning unit samples training data from the playback pool, updates network parameters, and enters S26; S26, enter S3; S27, end the training, and output the trained network for robot navigation control; The robot training environment is modeled as a POMDP, which is represented by the following six-tuple: (1) State space S, s t ∈S represents the state of the robot at time t; (2) Action space A, a t ∈A represents the action taken by the robot at time t; (3) Transition probability function T: S×A×S→[0,1], T(s t ,a t ,s t+1 ) indicates that the robot is in state s t Take action a t Transfer to t+1 probability; (4) Reward Function R(s t ,a t ) indicates that the robot is in state s t Take action a t The immediate rewards you can get; (5) Observation space O, o t ∈O represents the observation obtained by the robot at time t; (6) Observation probability function Z: S×A×O→[0,1], Z(s t ,a t-1 ,o t ) indicates that the robot is taking action a t-1 Transfer to t Get observation o t probability; The goal of POMDP is to obtain a strategy π:H→A based on the historical action observation sequence to maximize the expected cumulative reward, the cumulative reward G t is defined as: Among them, γ∈(0,1] is a discount factor used to weigh the immediate reward and delayed reward; r t Represents the reward obtained by the robot at time t.

2. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: When the trained network is used for robot navigation control, the replay pool and the learning unit are cancelled, and steps S24 and S25 are skipped to obtain the specific steps of the robot navigation control use phase. At this time, the environment in S20 only needs to provide observation and reward information, but not real state information.

3. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: The belief state b t (s) = p(s t =s|h t ), indicating the known history h t ={b0,a0,o1,…,a t-1 ,o t }, the hidden state s t is the probability distribution of s, and b0 represents the probability distribution of the initial state.

4. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: The transfer model D ψ The input is state and action, the output is the state and reward at the next moment, and the transfer model network structure is a fully connected network; the observation model Z θ The input is state and observation, the output is the probability of observation, and the observation model network structure is a fully connected network; the policy network π ρ The input is the belief state particle and the average belief state, the output is the action and the logarithm of the probability of outputting the action, and the network structure of the policy network is to output the mean μ and variance σ of the action through the fully connected network 2 , and then from the Gaussian distribution N(μ,σ 2 ) to sample the action, and use Gaussian distribution to calculate the logarithmic value of the probability of the output action; the double Q value network Q ω The input is state and action, the output is two Q values, and the dual Q value network is two fully connected networks Q1 and Q2. i (i=1,2) each maintains a target Q value network TQ with the same network structure i (i=1,2), used for network parameter update.

5. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: In S14, in the estimation of the information entropy of the belief state, the belief state particle Estimating belief state probability distribution When , the kernel density estimation method is used, and the Gaussian kernel density is used to estimate the probability distribution of belief states.

6. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: In S15, the calculation formula of the advantage function A is: Where TD represents the timing differential error, Q ω is the smaller value of the two outputs of the dual Q value network.

7. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: In S17 and S22, the resampling is to randomly and repeatedly select N particles from N weighted particles according to the weights, and then set the weights of the new particles to 1.

8. The robot navigation control method based on partially observable reinforcement learning according to claim 1, characterized in that: In S25, updating the network parameters includes: (1) The transfer model and the observation model use the minimum mean square error between the predicted value and the true value as the loss function, use the optimization method to optimize the loss function, and update the network parameters; (2) The two networks of the dual Q-value network are updated in the same way, both based on the time difference error with the target Q-value network: Among them, α is the temperature coefficient, which controls the emphasis on policy entropy; use the optimization method to optimize the loss function and update the network parameters; copy the Q value network parameters to the target Q value network at every fixed update step; (3) The strategy function is updated based on the loss function: Where α is the temperature coefficient, Q ω It is the smaller value of the two outputs of the dual Q value network; use the optimization method to optimize the loss function and update the network parameters.

9. A system for implementing the robot navigation control method based on partially observable reinforcement learning according to any one of claims 1 to 8, characterized in that: include: Filtering unit, planning unit, playback pool, learning unit; The filter unit is used to update the belief state particles and their weights, and use the actions obtained from the planning unit to interact with the training environment to obtain state, observation, and reward information; it is also used to process the training data and store it in the replay pool; The planning unit is used to receive the weighted particles provided by the filtering module, use the learned transfer model and strategy network to simulate planning, and output actions to the filtering unit; Replay pool, a database that supports random access and is used to store processed training data and provide the learning unit with the training data required for learning; A learning unit, which is used to sample the training data in the revisit pool, train the network using a given optimization method, and provide the updated network parameters to the filtering unit and the planning unit; The training environment is an actual application environment or a simulated virtual environment, which is used for training the robot navigation control method, interacts with the filtering unit, and provides state, observation, and reward information for the filtering.

Images