Indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG

Abstract

An indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG belongs to the field of dynamic obstacle avoidance, and comprises the steps of data acquisition, preprocessing, construction of a Bayesian MADDPG network structure, design of a reward function and execution of a dynamic obstacle avoidance task. According to the method, Bayesian inference is embedded in an MADDPG framework, reinforcement learning of uncertainty perception is realized, and decision robustness in an unknown environment is improved; dynamic balance of path length, speed and safety is realized through multi-target reward fusion; the risk penalty function introduces a robot trend factor, not only depends on distance, and is more in line with decision logic of dynamic obstacle avoidance; and through significant differentiation of reward and punishment values, strategy convergence and experience memory are accelerated. Scene hierarchical experience management is introduced into the dynamic obstacle avoidance of the robot, so that the sample utilization rate and the convergence speed are improved; active learning is carried out through failure experience, repeated errors are avoided, the system safety is enhanced, the generalization ability is high, the safety is high, and the training efficiency is high.

Description

Technical Field

[0001] This invention belongs to the field of dynamic obstacle avoidance technology, and in particular relates to a dynamic obstacle avoidance method for indoor robots based on Bayesian MADDPG. Background Technology

[0002] Dynamic obstacle avoidance refers to the process by which a mobile robot, starting from a starting point, reaches a target point while avoiding surrounding dynamic obstacles. However, existing general dynamic obstacle avoidance methods for robots first require environmental perception and modeling, followed by dynamic obstacle detection and prediction, global dynamic obstacle avoidance planning, and finally motion control and execution. All of these require prior environmental modeling, making them difficult to apply to unknown environments. General dynamic obstacle avoidance methods suffer from an improper balance between path length, robot speed, and safety; for example, always aiming for the shortest path can lead to overly aggressive robot behavior. Furthermore, general dynamic obstacle avoidance methods often have flawed safety reward designs, such as prioritizing task completion in a short time, resulting in low safety. Finally, due to the probabilistic abrupt changes in the position, speed, direction, and interactions between dynamic obstacles, general dynamic obstacle avoidance methods cannot accurately capture the complex relationships between state information and actions, leading to low success rates and poor performance in dynamic obstacle avoidance tasks. Summary of the Invention

[0003] To address the problems of existing general robot dynamic obstacle avoidance methods, such as difficulty in applying them to unknown environments, improper balance between path length, robot speed, and safety, imprecise safety reward design, low safety, low success rate, and poor performance in dynamic obstacle avoidance tasks, this invention provides an indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG.

[0004] The technical solution adopted by this invention to solve the technical problem is as follows:

[0005] The present invention provides a dynamic obstacle avoidance method for indoor robots based on Bayesian MADDPG, comprising the following steps:

[0006] (1) Constructing the Bayesian MADDPG network structure: Replace the Actor and Critic networks in the traditional MADDPG network with Bayesian neural networks, and construct the Bayesian Actor network (BAN network) and the Bayesian Critic network (BCN network) respectively; the BAN network outputs the mean and variance of the action, supports probabilistic action sampling, and enhances the adaptability to uncertain environments; the BCN network outputs the mean and variance of the Q value, quantifying the uncertainty of the action value;

[0007] (2) Design a comprehensive heuristic reward function, including a distance reward function, a risk penalty function, a boundary penalty function, and a path reward function;

[0008] (3) A hierarchical priority experience replay mechanism is implemented by calculating hierarchical experience priority, hierarchical experience sampling probability, experience importance weight, and constructing an experience replay pool; the experience replay pool includes a key experience pool N. key Normal experience pool N normal and negative experience pool N neg The total experience pool is: Key Experience Pool N key Capacity allocation of 30%, storage of critical obstacle avoidance scenario experience; ordinary experience pool N normal Capacity utilization rate of 60%, storage experience in non-critical and stable scenarios; negative experience pool N neg 10% capacity utilization, experience in storage failure scenarios;

[0009] (4) Pre-set the accuracy threshold, initialize the parameters, and place the robot for dynamic obstacle avoidance training so that it can use the Bayesian MADDPG network to select the optimal action according to the state to achieve dynamic obstacle avoidance; when the accuracy of the robot in completing the dynamic obstacle avoidance task is higher than the accuracy threshold, increase the number of obstacles in the environment; update the network parameters once every 50 training rounds; when the maximum number of training rounds is reached, end the training.

[0010] As a further limitation, the BAN network includes three hidden layers and one output layer, each hidden layer consisting of a fully connected layer and an activation function; the hidden layers of the BAN network use the ReLU activation function, denoted as... W i s is the Bayesian weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i It is the bias vector of the i-th hidden layer; the mean μ of the output action of the output layer of the BAN network. a and variance σ a 2 , respectively represented as: ; W final It is the Bayesian weight matrix of the action mean output layer; b final W is the bias vector of the action mean output layer; σ It is the Bayesian weight matrix of the action variance output layer; b σ `output3` is the bias vector of the action variance output layer; `output3` is the output of the last hidden layer; the `Softplus` function is used to ensure the variance σ. a 2 Greater than zero.

[0011] As a further constraint, the Bayesian weight matrix W of the i-th hidden layer i With bias vector b iAll follow a Gaussian prior distribution with zero mean and fixed variance, defined as: N(⋅,⋅) represents a Gaussian distribution, σ0=0.1 is a hyperparameter used to adjust the intensity of prior uncertainty; the ReLU activation function is expressed as: x is the input data. When the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, it directly outputs 0.

[0012] As a further constraint, the final action sampling is performed by compressing the mean to [-1, 1] using the tanh function to adapt to the continuous action space, and then sampling action A from a Gaussian distribution. The tanh function is expressed as: Ultimately, the BAN network outputs a probabilistic action distribution based on the input state, rather than a single deterministic action.

[0013] As a further limitation, the BCN network comprises three hidden layers and one output layer, each hidden layer consisting of a fully connected layer and an activation function; the hidden layers of the BCN network use the ReLU activation function, denoted as... W i s is the linear weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i It is the bias vector of the i-th hidden layer; the mean μ of the Q-value of the output layer of the BCN network. q and variance σ q 2 To assess the uncertainty of action value, they are expressed as follows: ; W final It is the Bayesian weight matrix of the Q-mean output layer, b final W is the bias vector of the Q-mean output layer; σ,q It is the Bayesian weight matrix of the Q-value variance output layer; b σ,q This is the bias vector of the output layer, representing the Q-value variance; out3 is the output of the last hidden layer; the Softplus function is used to ensure the variance σ. q 2 Greater than zero; the BCN network first outputs the mean μ of the Q-value distribution based on the input global state and global action. q and variance σ q 2 Then through the sampling formula A Q-value sample is obtained.

[0014] As a further restriction, the linear weight matrix W of the i-th hidden layer i With bias vector b i All follow a Gaussian prior distribution with zero mean and fixed variance, defined as N(⋅,⋅) represents a Gaussian distribution, σ0=0.15 is a hyperparameter used to adjust the intensity of prior uncertainty; the ReLU activation function is expressed as: x is the input data. When the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, it directly outputs 0.

[0015] As a further limitation, the formula for calculating the hierarchical experience priority is: W pool As the pool type weight, when the experience belongs to the critical experience pool N key Normal experience pool N normal and negative experience pool N neg At that time, the weights for this pool type are 2.0, 0.5, and 5.0, respectively, to differentiate and amplify the priority of high-value experiences; u is a small constant; experience is a tuple containing environmental state information, a certain action taken by the robot, the reward obtained, and the result of the journey; k is the experience priority scaling factor, which is responsible for controlling the degree of influence of experience priority. When k is 0, it degenerates into uniform sampling, and when k is 1, it samples entirely according to the prediction error priority; j i This represents the prediction error of the i-th experience. The larger the prediction error, the more important the experience.

[0016] As a further limitation, the formula for calculating the prediction error is: Q current Q represents the output value of the BCN network; target This is the output value of the target Critic network, expressed as: ; r is the immediate reward obtained after performing action a; f is the discount factor, representing the importance attached to future rewards. The larger the value of f, the more the robot focuses on long-term rewards; s' is the next state after performing action a, and a' is the action chosen by the target Actor network in state s'.

[0017] As a further limitation, the formula for calculating the hierarchical empirical sampling probability is: ;p i It is the priority of the i-th experience; It is the sum of all experience priorities.

[0018] As a further limitation, the formula for calculating the empirical importance weight is: N represents the total number of experiences in the buffer; P(i) represents the sampling probability of the i-th experience; max(w) is the maximum value of the importance weights of all experiences; n is the compensation factor.

[0019] The beneficial effects of this invention are:

[0020] (1) In view of the problem that general robot dynamic obstacle avoidance methods require environmental modeling in advance and are difficult to apply to unknown environments, this invention uses the state information of the robot and the environment to train the Bayesian MADDPG network, so that it learns the ability to select the corresponding action strategy by acquiring the information. Since the object of its learning is the relationship between the robot and the environment information and actions rather than the relationship between position information and actions, this invention can be applied to environment B after training in environment A, and has stronger adaptability and superior generalization ability.

[0021] (2) To address the problem of improper balance between path length, robot speed and safety in general robot dynamic obstacle avoidance methods, a comprehensive heuristic reward function was designed, which includes a path reward function, a danger penalty function and a speed abnormality penalty function, to balance the three aspects of robot path length, speed and safety.

[0022] (3) In view of the problems of poor safety reward design and low safety in general robot dynamic obstacle avoidance methods, the present invention comprehensively considers the distance between the mobile robot and the obstacle and the robot's movement when designing the danger penalty function, thereby improving the safety of mobile robot tasks.

[0023] (4) In view of the problem that general robot dynamic obstacle avoidance methods cannot accurately capture the complex relationship between state information and actions, resulting in low success rate and poor effect of dynamic obstacle avoidance tasks, this invention adds a hierarchical priority experience playback mechanism. By hierarchically classifying experience, the probability of learning effective experience is increased, and ineffective training is reduced, thereby improving the utilization rate of samples, improving the convergence speed, learning efficiency and obstacle avoidance effect of the algorithm. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a specific implementation of the Bayesian MADDPG-based dynamic obstacle avoidance method for indoor robots according to the present invention.

[0025] Figure 2 This is a flowchart illustrating the specific implementation of step S3 in a Bayesian MADDPG-based dynamic obstacle avoidance method for indoor robots according to the present invention.

[0026] Figure 3 This is a flowchart illustrating the specific implementation of step S4 in a Bayesian MADDPG-based dynamic obstacle avoidance method for indoor robots according to the present invention.

[0027] Figure 4 This is a reward convergence graph based on ordinary experience sampling.

[0028] Figure 5 This is the reward convergence graph under stratified priority experience sampling.

[0029] Figure 6This is a task completion graph based on the traditional MADDPG algorithm.

[0030] Figure 7 This is a task completion graph based on the Bayesian MADDPG algorithm.

[0031] Figure 8 This is a diagram illustrating the effect of a single robot avoiding two dynamic obstacles, and it also serves as a scenario to verify the advantages and disadvantages of different methods.

[0032] Figure 9 This is a diagram illustrating the effect of a single robot avoiding six dynamic obstacles, based on the core inventive points of this invention.

[0033] Figure 10 This is a diagram illustrating the effect of two robots avoiding six dynamic obstacles, based on the core inventive points of this invention.

[0034] Figure 11 The illustration shows three robots avoiding six dynamic obstacles, based on the core inventive points of this invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings.

[0036] This invention discloses a dynamic obstacle avoidance method for indoor robots based on Bayesian MADDPG, which mainly includes steps such as data acquisition, data preprocessing, constructing a Bayesian MADDPG network structure, designing a reward function, and executing a dynamic obstacle avoidance task. Its core inventive points mainly include the following aspects:

[0037] I. Network architecture design based on Bayesian MADDPG;

[0038] Limitations of traditional MADDPG: Traditional methods output deterministic actions and Q-values, making it difficult to quantify decision uncertainty. In dynamic and unknown environments, they are prone to collisions or inefficient decisions due to overconfidence.

[0039] This invention replaces the Actor and Critic networks in the traditional MADDPG network with Bayesian Neural Networks (BNNs), constructing them as Bayesian Actor Networks (BANs) and Bayesian Critic Networks (BCNs), respectively. The BAN outputs the mean and variance of actions, supporting probabilistic action sampling. By sampling a Gaussian distribution to generate probabilistic actions, it enhances the policy's exploratory capabilities and adaptability to environmental uncertainty. The BCN outputs the mean and variance of Q-values, quantifying the uncertainty of action value, improving decision robustness, and providing a confidence reference for decision-making. This invention is the first to embed Bayesian inference into the MADDPG framework, achieving uncertainty-aware reinforcement learning and improving decision robustness in unknown environments.

[0040] II. Design of a comprehensive heuristic reward function;

[0041] Traditional reward functions are often singular (e.g., focusing only on path length or arrival reward), making it difficult to balance safety, efficiency, and path quality. This invention designs a comprehensive heuristic reward function that includes multiple sub-rewards and penalties, achieving a balance between path length, speed, and safety, thereby improving task completion quality. It mainly covers the following aspects:

[0042] Distance reward / penalty function: encourages the robot to move closer to the target point.

[0043] Danger penalty function: The penalty intensity is dynamically adjusted based on the distance and direction of the robot and the obstacle.

[0044] Boundary penalty function: Prevents the robot from approaching the map boundary.

[0045] Path reward function: Rewards robots that successfully reach the goal along the shortest path.

[0046] Multiple task success and failure rewards and penalties: set significant reward and penalty values ​​for key events such as collisions, timeouts, and reaching the target.

[0047] This invention achieves a dynamic balance between path length, speed, and safety through multi-objective reward fusion.

[0048] The hazard penalty function incorporates robot movement factors, relying not only on distance but also aligning better with the decision-making logic of dynamic obstacle avoidance. By significantly differentiating reward and penalty values ​​(e.g., a much higher reward for success), policy convergence and experience memorization are accelerated.

[0049] III. Tiered Priority Experience Replay Mechanism;

[0050] The shortcomings of traditional experience replay: uniform sampling or priority sampling based solely on TD error cannot distinguish the actual learning value of experience, resulting in low training efficiency.

[0051] This invention divides the experience pool into three categories:

[0052] Key Experience Pool (30%): Stores critical obstacle avoidance scenarios (such as successful close-range obstacle avoidance).

[0053] Ordinary experience pool (60%): Stores non-critical and stable scenarios.

[0054] Negative experience pool (10%): Stores failure scenarios (collision, timeout).

[0055] This invention amplifies the sampling priority of high-value experiences by differentiating pool type weights, thereby increasing the learning frequency of high-value experiences (critical and failed experiences) and improving learning efficiency; it also introduces importance sampling weights to correct the bias caused by priority sampling and enhance training stability.

[0056] This invention is the first to introduce scene-layered experience management in robot dynamic obstacle avoidance, improving sample utilization and convergence speed. By actively learning from failures, it avoids repeating mistakes and enhances system safety.

[0057] IV. Mechanisms for enhancing generalization and security;

[0058] Traditional methods rely on prior environmental modeling, making them difficult to transfer to unknown environments; safety is often used as a constraint rather than a learning objective. This invention achieves strong generalization capabilities without prior environmental knowledge and a decision-making mechanism with embedded safety awareness, maintaining high task success rates and low collision rates even in unknown dynamic environments. Specifically, it achieves this in the following aspects:

[0059] Strong generalization ability: The network learns the association between state information and actions, rather than specific environmental location information, and supports cross-environment transfer (trained in environment A, applied in environment B).

[0060] Safety Enhancement: The hazard penalty function comprehensively considers distance and robot movement, enhancing the safety of obstacle avoidance decisions.

[0061] High training efficiency: Through hierarchical experience replay and Bayesian uncertainty modeling, the convergence speed is accelerated and the sample utilization rate is improved.

[0062] The core innovation of this invention lies in combining Bayesian neural networks with a multi-robot reinforcement learning framework to construct a dynamic obstacle avoidance framework that combines uncertainty perception, multi-objective optimization, efficient learning, and strong generalization capabilities. Through probabilistic action output, uncertainty perception, hierarchical experience playback, and comprehensive reward design, it achieves efficient, safe, and generalized dynamic obstacle avoidance capabilities for robots in unknown dynamic environments.

[0063] See Figures 1 to 3 The present invention provides a method for dynamic obstacle avoidance of indoor robots based on Bayesian MADDPG, and its specific implementation process is as follows:

[0064] Step S1: Data Acquisition;

[0065] By collecting robot motion information and environmental information, we can obtain robot motion information such as position, speed, and direction, as well as environmental information such as map size, obstacle position, obstacle speed, obstacle size, target position, and actual path.

[0066] Step S2: Data preprocessing;

[0067] Preprocessed data is obtained by cleaning, normalizing, and labeling the collected data. Data cleaning removes noisy and erroneous data and handles missing values; normalization normalizes the data; and data labeling ensures that each data point includes environmental information, actions taken, and results.

[0068] Step S3: Construct the Bayesian MADDPG network structure;

[0069] like Figure 2 As shown, the specific implementation process is as follows:

[0070] Step S31: Overall design of the Bayesian MADDPG network structure;

[0071] The Bayesian MADDPG network mainly consists of the BAN network and the BCN network. It replaces the original Actor network and the original Critic network with Bayesian Neural Network (BNN), namely Bayesian Actor Network (BAN) and Bayesian Critic Network (BCN). The established BAN and BCN networks are both four-layer static feedforward networks, including three hidden layers and one output layer. Each hidden layer consists of a fully connected layer and an activation function. It also includes a target Actor network and a target Critic network. The structure of the target Actor network and the target Critic network remains unchanged, that is, they are the same as the original Actor network and the original Critic network, but the updates are slower to solve the problem of unstable Q-value estimation.

[0072] Step S32: Design of hidden layers in the BAN network;

[0073] The hidden layers of the BAN network use the ReLU activation function, denoted as: Among them, W i s is the Bayesian weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i This is the bias vector of the i-th hidden layer. For the first hidden layer, s1 is the input data x of the BAN network, specifically represented as (1, state_dim). This is the identifier of the tensor dimension, representing the dimension of the number of samples in a single sample, so the first part is 1, and state_dim is the dimension of the state space. For subsequent hidden layers, s1... i This is the output of the previous hidden layer.

[0074] W is the Bayesian weight matrix of the i-th hidden layer. i With bias vector b i All follow a Gaussian prior distribution with zero mean and fixed variance, defined as: ; where N(⋅,⋅) represents a Gaussian distribution, and σ0=0.1 is a hyperparameter used to regulate the intensity of prior uncertainty.

[0075] The mathematical expression for the ReLU activation function is: Where x is the input data, the intuitive interpretation of the ReLU activation function is: when the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, the ReLU activation function directly outputs 0. This design aims to introduce nonlinearity and enhance the expressive power of the BAN network.

[0076] Step S33: Design of the output layer of the BAN network;

[0077] For the final output layer of the BAN network, since no activation function is used, the mean μ of the output action is instead output. a and variance σ a 2 To adapt to probabilistic obstacle avoidance, the specific mathematical calculation formula is as follows:

[0078] ;

[0079] ;

[0080] Among them, W final It is the Bayesian weight matrix of the action mean output layer; b final W is the bias vector of the action mean output layer; σ It is the Bayesian weight matrix of the action variance output layer; b σ `output3` is the bias vector of the action variance output layer; `output3` is the output of the last hidden layer; the `Softplus` function is used to ensure the variance σ. a 2 Greater than zero is defined as: .

[0081] The final action sampling is performed by compressing the mean to [-1, 1] using the tanh function to adapt it to the continuous action space, and then sampling action A from the Gaussian distribution, as follows: The mathematical expression for the tanh function is: Ultimately, the BAN network outputs a probabilistic action distribution based on the input state, rather than a single deterministic action.

[0082] Step S34: Design of hidden layers in the BCN network;

[0083] The hidden layers of the BCN network use the ReLU activation function, expressed as: Among them, W i s is the linear weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i s1 is the bias vector of the i-th hidden layer. For the first hidden layer, s1 is the input data x of the BCN network, specifically represented as (state_dim×n, action_dim×n), where n is the batch size, state_dim is the state space dimension, and action_dim is the action space dimension; for subsequent hidden layers, s1... i This is the output of the previous hidden layer.

[0084] The linear weight matrix W of the i-th hidden layer i With bias vector b i All follow a Gaussian prior distribution with zero mean and fixed variance, defined as ; where N(⋅,⋅) represents a Gaussian distribution, and σ0=0.15 is a hyperparameter used to adjust the intensity of prior uncertainty.

[0085] The mathematical expression for the ReLU activation function is: Where x is the input data, the intuitive interpretation of the ReLU activation function is: when the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, the ReLU activation function directly outputs 0. This design aims to introduce nonlinearity and enhance the expressive power of the BCN network.

[0086] Step S35: Design of the output layer of the BCN network;

[0087] For the final output layer of the BCN network, since no activation function is used, the mean μ of the output Q-value is instead calculated. q and variance σ q 2 The specific mathematical formula for assessing the uncertainty of action value is as follows:

[0088] ;

[0089] ;

[0090] Among them, W final It is the Bayesian weight matrix of the Q-mean output layer, b final W is the bias vector of the Q-mean output layer; σ,q It is the Bayesian weight matrix of the Q-value variance output layer; b σ,q This is the bias vector of the output layer, representing the Q-value variance; out3 is the output of the last hidden layer; the Softplus function is used to ensure the variance σ. q 2 Greater than zero is defined as: .

[0091] To obtain a specific Q-value for loss calculation, sampling is required from this parameterized Gaussian distribution. This sampling process is a core part of the BCN network, and the specific sampling formula is as follows: .

[0092] Based on the input global state and global action, the BCN network first outputs the mean μ of the Q-value distribution. q and variance σ q 2 Then, a Q-value sample is obtained through a sampling formula. This process not only evaluates the magnitude of the action's value (based on the mean μ) q (This can be reflected in the data) and can also quantify the degree of uncertainty in this assessment (by variance σ). q 2 (This is reflected in the fact that) the smaller the variance, the more reliable the BCN network's judgment on the value of the current action, which is crucial for making robust decisions in uncertain environments.

[0093] Step S4: Design a comprehensive heuristic reward function, including a distance reward function, a risk penalty function, a boundary penalty function, and a path reward function;

[0094] like Figure 3 As shown, the specific implementation process is as follows:

[0095] Step S41: Design a comprehensive heuristic reward function;

[0096] To incentivize the robot to move towards the target point, rewards should be given for movement towards the target and penalties for movement away from the target point. Considering safety in dynamic obstacle avoidance, dangerous behaviors of the robot should be penalized; that is, an obstacle range should be set, and the robot should be penalized for approaching obstacles within this range, with the penalty increasing the closer it gets to the obstacle. To prevent the robot from colliding with map boundaries during movement, a range should also be set for the map boundaries; when the robot enters this range, the closer it gets to the boundary, the greater the penalty. To improve the robot's movement efficiency, a maximum path length should be set, and the shorter the path to the target point, the greater the reward. When the robot reaches the target point, the dynamic obstacle avoidance task is considered successful, and a large reward should be given to encourage the robot to remember the experience upon reaching the target point. To prevent the robot from moving too fast or too slow during movement, abnormal speeds should be penalized. When the robot collides with the map boundary during movement, the dynamic obstacle avoidance task is considered a failure, and a strong penalty should be given. When the robot exceeds the maximum path length during movement, the dynamic obstacle avoidance task is considered a failure, and a strong penalty should be given. When the robot collides with an obstacle during its movement, it is determined that the dynamic obstacle avoidance task has failed and a strong penalty is imposed.

[0097] Therefore, the comprehensive heuristic reward function is defined as: ; where r d The distance reward / penalty function, r, represents the relative motion between the robot and the target point. p The function r represents the penalty function for danger. b It is a penalty function for areas near the map boundary, r s It is the reward function for path length, r o It is the reward for the robot reaching the target point, r v The penalty for abnormal speed, r e It is the penalty for the robot colliding with the map boundary, r g It is the penalty for the robot exceeding the maximum path length, r c This is the penalty for the robot colliding with an obstacle. Where r... o r v r e r g and r c Both are relatively large constants (the range of values ​​is 20 to 100), the difference is r. o It should be much larger than other constants so that the robot can remember the experience of reaching the target point.

[0098] Step S42: Design of distance reward / penalty function;

[0099] The distance reward / penalty function is defined as follows:

[0100] ;

[0101] Where λ is the distance reward / penalty factor, d n-1 d is the distance between the robot and the target point at the previous moment. n This represents the robot's current distance from the target point, where n is the actual time point during training. When n=1, it indicates the very beginning of training, and there is no distance from the previous time point, so distance-based rewards / penalties are not calculated at this stage, and r is assigned... d Initial value 0; when n > 1, when (d n-1 -d n When (d) is positive, it means the robot is getting closer to the target point, and the distance reward function will give a positive reward; when (d) is positive, it means the robot is getting closer to the target point. n-1 -d n When the distance is negative, it means that the robot is moving away from the target point. The distance reward function will give a negative reward, i.e., a penalty, to encourage the robot to move closer to the target point.

[0102] Step S43: Design of the hazard penalty function;

[0103] The definition of the hazard penalty function is as follows:

[0104] ;

[0105] Where β is the danger penalty factor and is negative, k is the penalty value and is a positive integer, d is the distance between the robot and the nearest obstacle, and r x This is the radius of the nearest obstacle area. When the robot does not enter any obstacle area, the hazard penalty value is zero; when the robot enters an obstacle area, the closer the robot is to the obstacle, the greater the hazard penalty value will be, in order to train the robot to avoid and stay away from obstacles.

[0106] Step S44: Design of boundary penalty function;

[0107] The boundary penalty function is defined as follows:

[0108] ;

[0109] Where α is the boundary penalty parameter, d b It is the distance from the mobile robot to the nearest boundary (top, bottom, left, right) of the map, y b This refers to the boundary penalty range. When the robot does not enter the boundary penalty range, the boundary penalty value is zero; once the robot enters the boundary penalty range, it will be penalized, and the closer the robot is to the map boundary, the larger the boundary penalty value. The specific boundary penalty value is determined by the boundary penalty parameter α.

[0110] Step S45: Design of the path reward function;

[0111] The path reward function is defined as follows:

[0112] ;

[0113] Here, γ is the arrival flag; its value is 1 when the robot successfully reaches the target point, and 0 if it fails to do so. j is the path reward parameter and is a constant (ranging from 300 to 400). s is the actual path length traveled by the robot from the starting point to the target point. Therefore, the robot does not receive this path reward if it fails the dynamic obstacle avoidance task; it only receives it upon successful completion. Furthermore, the shorter the path traveled, the higher the path reward value.

[0114] Step S5: Perform dynamic obstacle avoidance task;

[0115] Step S51: Set up the state space;

[0116] The state space is defined as: Where x and y are the robot's positions, v is the robot's velocity, r is the robot's size, and x... o1 x o2 …and y o1 y o2 …is the real-time position of the obstacle, v o1 v o2 … is the velocity of the obstacle, r o1 r o2 … is the size of the obstacle, x t y t It is the location of the target point, d x d y d represents the relative position of the robot and the target point, and d is the real-time distance from the robot to the target point. Although the robot only considers the nearest obstacle when performing obstacle avoidance, the state space still contains information about other obstacles, making it easier to select which obstacle is closest to the robot.

[0117] Step S52: Layered priority experience replay mechanism;

[0118] The hierarchical priority experience replay mechanism is implemented by calculating hierarchical experience priority, sampling probability and importance weight, and constructing three sub-experience replay pools.

[0119] Among them, the three sub-experience replay pools are the key experience pool N. key Normal experience pool N normal and negative experience pool N neg The total experience pool is then represented as: Key Experience Pool N keyCapacity allocation of 30%, storage of critical obstacle avoidance scenario experience; ordinary experience pool N normal Capacity utilization rate of 60%, storage experience in non-critical and stable scenarios; negative experience pool N neg The storage capacity is 10%, used to store experience from failed scenarios. A critical distance hyperparameter (1m) is also set; experiences where the robot successfully avoids obstacles when the distance between it and the obstacle is less than the critical distance hyperparameter are stored in the critical experience pool N. key The experience of robot collisions or timeouts is stored in the negative experience pool N. neg The remaining experience is stored in the regular experience pool N. normal .

[0120] Step S53: Calculate the priority of stratified experience;

[0121] The specific formula for calculating the priority of stratified experience is as follows: Among them, W pool As the pool type weight, when the experience belongs to the critical experience pool N key Normal experience pool N normal and negative experience pool N neg At that time, the pool type weights are 2.0, 0.5, and 5.0, respectively, to differentiate and amplify the priority of high-value experiences; u is a small constant (the constant ranges from 0.0001 to 0.001) to avoid an experience priority of 0, which would prevent the experience from ever being sampled; an experience refers to a tuple containing environmental state information, a certain action taken by the robot, the reward obtained, and the result of the journey, and this tuple is called an experience; k is the experience priority scaling factor, which is responsible for controlling the degree of influence of experience priority. When k is 0, it degenerates into uniform sampling, and when k is 1, it samples entirely according to the prediction error priority; j i Let represent the prediction error of the i-th experience. The larger the prediction error, the more important the experience is, because the goal of deep reinforcement learning is to make the model's prediction value infinitely close to the true value. Experiences with large prediction errors are precisely the direct manifestation of the model's knowledge blind spots. These experiences can bring new and key information to the model, helping the model to quickly correct erroneous cognitions, and are more valuable for learning than experiences with small prediction errors.

[0122] The specific formula for calculating the prediction error is as follows: Q current This represents the output value of the BCN network, while Q... target This is the output value of the target Critic network, specifically represented as: ; r is the immediate reward obtained after performing action a (which can be calculated using the comprehensive heuristic reward function in step S41); f is the discount factor, representing the importance attached to future rewards. The larger the value of f, the more the robot focuses on long-term rewards; s' is the next state after performing action a, and a' is the action chosen by the target Actor network in state s'.

[0123] Step S54: Calculate the sampling probability of the stratified experience;

[0124] The specific formula for calculating the sampling probability of stratified experience is as follows: ; where p i It is the priority of the i-th experience; It is the sum of all experience priorities.

[0125] Step S55: Calculate the empirical importance weights;

[0126] The specific formula for calculating the importance weight of experience is as follows: Where N represents the total number of experiences in the buffer; P(i) represents the sampling probability of the i-th experience; max(w) is the maximum value of the importance weights of all experiences; n is a compensation factor that gradually decreases in intensity as the number of training steps increases from the initial value to 1. These experience importance weights are designed to correct the bias caused by oversampling of high-priority experiences.

[0127] Step S56: Dynamic obstacle avoidance training;

[0128] Pre-set an accuracy threshold; initialize parameters, including map size, target point location, robot initial position, speed, and size, obstacle initial position, speed, and size, and movement trajectory; place the robot in a simulation environment for dynamic obstacle avoidance training, enabling it to select the optimal action based on the state using a Bayesian MADDPG network, thereby achieving dynamic obstacle avoidance; when the robot's accuracy in completing the dynamic obstacle avoidance task exceeds the accuracy threshold, increase the number of obstacles in the environment; update network parameters every 50 training rounds; terminate training when the maximum number of training rounds is reached. The core basis for updating network parameters is as follows:

[0129] ;

[0130] Where L is the loss function, representing the weighted mean square error between the predicted Q-value and the target Q-value; B is the batch size for each training iteration; w i It is the importance weight of the i-th experience, j i L is the prediction error of the i-th experience; ultimately, the parameter updates of the entire network revolve around reducing L.

[0131] To verify the advantages of the Bayesian MADDPG algorithm of this invention compared with the traditional MADDPG algorithm, this invention designs a completion index, defined as follows:

[0132] ;

[0133] Where R represents completion level; D n D represents the distance from the starting point to the ending point. end This represents the distance between the robot's position at the end of the current round and the finish line. The closer the robot is to the target point, the higher the completion rate of this round. A completion rate of 100% indicates that the dynamic obstacle avoidance task has been successfully completed. Figure 6 This represents the traditional MADDPG algorithm in Figure 8 The training completion rate for each round in the given environment. Figure 7 This represents the Bayesian MADDPG algorithm of the present invention. Figure 8 The training completion rate in each round under the environment shows that although both algorithms can complete the task, the robot trained using the Bayesian MADDPG algorithm achieves 100% completion rate significantly more times. In other words, the success rate of this algorithm is significantly higher than that of the traditional MADDPG algorithm.

[0134] To verify the advantages of the hierarchical priority experience replay mechanism, this invention compared and verified it with ordinary experience sampling, using the total reward of each training round as the indicator. Both mechanisms still demonstrate advantages in... Figure 8 Training in an environment Figure 4 This represents the reward convergence graph under ordinary experience sampling. Figure 5 This represents the reward convergence graph under hierarchical priority experience sampling. It can be seen that ordinary experience sampling basically converges around 290 rounds, but the convergence is not ideal. In contrast, hierarchical priority experience sampling basically converges around 150 rounds and fully converges around 580 rounds. Therefore, it can be concluded that adopting the hierarchical priority experience replay mechanism can not only improve the convergence speed but also improve the convergence effect.

[0135] Meanwhile, the present invention Figure 8 The task difficulty was increased based on the original scenario, with the number of dynamic obstacles increased from 2 to 6. Even after training, the robot was still able to complete the task well. Figure 9 As shown; at the same time, based on the core invention points, the dynamic obstacle avoidance of dual-robot and triple-robot systems has been expanded, such as... Figure 10 and Figure 11 As shown, the trained robot can still complete the task well.

[0136] This invention discloses a dynamic obstacle avoidance method for indoor robots based on Bayesian MADDPG. The method trains a Bayesian MADDPG network using state information from both the robot and the environment, enabling it to learn to select appropriate action strategies based on acquired information, resulting in stronger adaptability and superior generalization ability. A comprehensive heuristic reward system is designed to balance path length, robot speed, and safety. The design of the hazard penalty function comprehensively considers the distance between the robot and obstacles, as well as the robot's movement, improving the safety of mobile robot tasks. A hierarchical priority experience replay mechanism is incorporated to increase the probability of learning effective experience and reduce ineffective training, thereby improving sample utilization, convergence speed, learning efficiency, and obstacle avoidance performance.

[0137] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A dynamic obstacle avoidance method for indoor robots based on Bayesian MADDPG, characterized in that, Includes the following steps: (1) Constructing the Bayesian MADDPG network structure: Replace the Actor and Critic networks in the traditional MADDPG network with Bayesian neural networks, and construct the Bayesian Actor network (BAN network) and the Bayesian Critic network (BCN network) respectively; the BAN network outputs the mean and variance of the action, supports probabilistic action sampling, and enhances the adaptability to uncertain environments; the BCN network outputs the mean and variance of the Q value, quantifying the uncertainty of the action value; (2) Design a comprehensive heuristic reward function, including a distance reward function, a risk penalty function, a boundary penalty function, and a path reward function; (3) A hierarchical priority experience replay mechanism is implemented by calculating hierarchical experience priority, hierarchical experience sampling probability, experience importance weight, and constructing an experience replay pool; the experience replay pool includes a key experience pool N. key Normal experience pool N normal and negative experience pool N neg The total experience pool is: Key Experience Pool N key Capacity allocation of 30%, storage of critical obstacle avoidance scenario experience; ordinary experience pool N normal Capacity utilization rate of 60%, storage experience in non-critical and stable scenarios; negative experience pool N neg 10% capacity utilization, experience in storage failure scenarios; (4) Pre-set the accuracy threshold, initialize the parameters, and place the robot for dynamic obstacle avoidance training so that it can use the Bayesian MADDPG network to select the optimal action according to the state to achieve dynamic obstacle avoidance; when the accuracy of the robot in completing the dynamic obstacle avoidance task is higher than the accuracy threshold, increase the number of obstacles in the environment; update the network parameters once every 50 training rounds; when the maximum number of training rounds is reached, end the training.

2. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 1, characterized in that, The BAN network comprises three hidden layers and one output layer. Each hidden layer consists of a fully connected layer and an activation function. The hidden layers of the BAN network use the ReLU activation function, denoted as: W i s is the Bayesian weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i It is the bias vector of the i-th hidden layer; the mean μ of the output action of the output layer of the BAN network. a and variance σ a 2 , respectively represented as: ; W final It is the Bayesian weight matrix of the action mean output layer; b final W is the bias vector of the action mean output layer; σ It is the Bayesian weight matrix of the action variance output layer; b σ `output3` is the bias vector of the action variance output layer; `output3` is the output of the last hidden layer; the `Softplus` function is used to ensure the variance σ. a 2 Greater than zero.

3. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 2, characterized in that, W is the Bayesian weight matrix of the i-th hidden layer. i With bias vector b i All follow a Gaussian prior distribution with zero mean and fixed variance, defined as: N(⋅,⋅) represents a Gaussian distribution, σ0=0.1 is a hyperparameter used to adjust the intensity of prior uncertainty; the ReLU activation function is expressed as: x is the input data. When the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, it directly outputs 0.

4. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 2, characterized in that, The final action sampling is performed by compressing the mean to [-1, 1] using the tanh function to adapt to the continuous action space, and then sampling action A from a Gaussian distribution. The tanh function is expressed as: Ultimately, the BAN network outputs a probabilistic action distribution based on the input state, rather than a single deterministic action.

5. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 1, characterized in that, The BCN network comprises three hidden layers and one output layer. Each hidden layer consists of a fully connected layer and an activation function. The hidden layers of the BCN network use the ReLU activation function, denoted as: W i s is the linear weight matrix of the i-th hidden layer; i b is the input of the i-th hidden layer; i It is the bias vector of the i-th hidden layer; the mean μ of the Q-value of the output layer of the BCN network. q and variance σ q 2 To assess the uncertainty of action value, they are expressed as follows: ; W final It is the Bayesian weight matrix of the Q-mean output layer, b final W is the bias vector of the Q-mean output layer; σ,q It is the Bayesian weight matrix of the Q-value variance output layer; b σ,q This is the bias vector of the output layer, representing the Q-value variance; out3 is the output of the last hidden layer; the Softplus function is used to ensure the variance σ. q 2 Greater than zero; the BCN network first outputs the mean μ of the Q-value distribution based on the input global state and global action. q and variance σ q 2 Then through the sampling formula A Q-value sample is obtained.

6. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 5, characterized in that, The linear weight matrix W of the i-th hidden layer i With bias vector b i All follow a Gaussian prior distribution with zero mean and fixed variance, defined as N(⋅,⋅) represents a Gaussian distribution, σ0=0.15 is a hyperparameter used to adjust the intensity of prior uncertainty; the ReLU activation function is expressed as: x is the input data. When the input data x is positive, the ReLU activation function directly outputs the data x; otherwise, it directly outputs 0.

7. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 1, characterized in that, The formula for calculating the hierarchical experience priority is as follows: ; W pool As the pool type weight, when the experience belongs to the critical experience pool N key Normal experience pool N normal and negative experience pool N neg At that time, the weights for this pool type are 2.0, 0.5, and 5.0, respectively, to differentiate and amplify the priority of high-value experiences; u is a small constant; experience is a tuple containing environmental state information, a certain action taken by the robot, the reward obtained, and the result of the journey; k is the experience priority scaling factor, which is responsible for controlling the degree of influence of experience priority. When k is 0, it degenerates into uniform sampling, and when k is 1, it samples entirely according to the prediction error priority; j i This represents the prediction error of the i-th experience. The larger the prediction error, the more important the experience.

8. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 7, characterized in that, The formula for calculating the prediction error is as follows: Q current Q represents the output value of the BCN network; target This is the output value of the target Critic network, expressed as: ; r is the immediate reward obtained after performing action a; f is the discount factor, representing the importance attached to future rewards. The larger the value of f, the more the robot focuses on long-term rewards; s' is the next state after performing action a, and a' is the action chosen by the target Actor network in state s'.

9. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 1, characterized in that, The formula for calculating the stratified empirical sampling probability is: ;p i It is the priority of the i-th experience; It is the sum of all experience priorities.

10. The indoor robot dynamic obstacle avoidance method based on Bayesian MADDPG according to claim 1, characterized in that, The formula for calculating the empirical importance weight is as follows: N represents the total number of experiences in the buffer; P(i) represents the sampling probability of the i-th experience; max(w) is the maximum value of the importance weights of all experiences; n is the compensation factor.

Images