Hierarchical AI navigation obstacle avoidance method for humanoid robot based on depth information

Abstract

The invention provides a hierarchical AI navigation obstacle avoidance method for a humanoid robot based on depth information, and the method comprises the steps: taking various information of the humanoid robot as input through an upper-layer reinforcement learning model, outputting a speed instruction, enabling the speed instruction to serve as one of the inputs of a lower-layer reinforcement learning model, outputting the torque of each joint through the lower-layer reinforcement learning model, and controlling the movement of the robot, the current speed and angular speed of the humanoid robot are estimated through the estimation network, fusion with humanoid robot body information is carried out during deployment, the real-time position of the humanoid robot is estimated, the navigation obstacle avoidance ability of the humanoid robot can be almost decoupled from the motion ability by setting a layered architecture, the network can also be trained separately, and the real-time position of the humanoid robot can be obtained. The lower-layer reinforcement learning model estimates the speed, the estimation precision is very high, the speed and the position of the fuselage can be estimated without external positioning, and the upper-layer reinforcement learning model adjusts the speed by analyzing a visual image to finally realize obstacle avoidance.

Description

Technical Field

[0001] The present invention relates to the technical field of humanoid robots, and in particular to a layered AI navigation and obstacle avoidance method for humanoid robots based on depth information. Background Art

[0002] In the field of humanoid robot technology, the current focus on leg movement capabilities is on walking, running, parkour and other fields. However, when robots are used to perform tasks indoors or outdoors in the future, the above movement capabilities must be combined with navigation and obstacle avoidance capabilities. There are still few related studies at present, so this invention is forward-looking. However, from the perspective of algorithms and software and hardware coordination, the realization of this capability still has many difficulties and limitations, which are mainly reflected in the following aspects:

[0003] Algorithm framework design: The emphasis of motion ability and obstacle avoidance ability is different. To organically combine the two abilities in a humanoid robot, a reasonable algorithm framework is needed to coordinate them. This is one of the difficulties.

[0004] Intelligence: Compared with other types of robots such as quadruped robots and sweeping robots, humanoid robots are more difficult to control and require higher intelligence in algorithms. How to improve the intelligence of robots is one of the key points in their development.

[0005] Algorithm stability: The robot’s navigation and motion capabilities cannot be completely decoupled. Therefore, how to maintain stable motion capabilities in a variety of navigation and obstacle avoidance scenarios, as well as how to maintain good obstacle avoidance performance in different forms of motion (such as walking or running) is another difficulty in development.

[0006] Cost control: Navigation and obstacle avoidance algorithms need to be implemented in combination with external sensors. A variety of sensors can be used, and the prices and principles of different sensors are completely different. Common sensors include lidar, RGB-D cameras, ultrasonic radars, etc. When selecting sensors for mass-produced products, cost constraints, stability of sensor data, and the difficulty of implementing the algorithm need to be considered.

[0007] The humanoid robots in the prior art generally have the following technical problems during operation:

[0008] Movement smoothness: Deep reinforcement learning is a widely recognized and adopted algorithm for the development of humanoid robot movement capabilities by companies and institutions. However, it is still insufficient in simulating the smoothness and robustness of human movement. Especially after giving the robot navigation and obstacle avoidance capabilities, higher requirements are placed on movement performance and smoothness. The existing gait control algorithms based on deep reinforcement learning are not stable enough.

[0009] Command following accuracy: The speed command following accuracy of humanoid robots in the existing technology needs to be improved and cannot meet the followability requirements of upper-level navigation commands, so it needs to be optimized.

[0010] Obstacle avoidance real-time performance: The robot's navigation and obstacle avoidance capabilities require high real-time performance, which places high demands on sensors and algorithms. In the field of humanoid robots, there are currently almost no relevant reference algorithms, so we need to develop and design obstacle avoidance real-time performance in a targeted manner.

[0011] Generalization: The existing humanoid robot motion control methods based on deep reinforcement learning have limited generalization in different scenarios, which may lead to performance degradation or error accumulation in practical applications.

[0012] Energy efficiency: The energy consumption of existing humanoid robot motion algorithms is relatively high, especially in navigation and obstacle avoidance tasks that require long-term operation. This problem will limit the practicality of the robot.

[0013] Virtual to real conversion: During the conversion from the virtual environment to the real world, the robot's motion control strategy may not be directly applicable and requires additional adjustments and optimization. Summary of the invention

[0014] In order to solve the above technical problems, the present invention proposes a hierarchical AI navigation and obstacle avoidance method for a humanoid robot based on depth information, so that the humanoid robot can achieve better navigation and obstacle avoidance effects when only a depth camera is used as a sensor.

[0015] To achieve the above object, the technical solution adopted by the present invention is:

[0016] A humanoid robot hierarchical AI navigation obstacle avoidance method based on depth information, characterized by: designing an upper reinforcement learning model and a lower reinforcement learning model, wherein the upper reinforcement learning model is used to take various information of the humanoid robot as input and output speed instructions, and the output speed instructions are used as the instruction input of the lower reinforcement learning model, and the lower reinforcement learning model outputs the torque of each joint for controlling the robot movement, and includes an estimator, an estimation network and a lower strategy network, which will estimate the current speed and angular velocity of the humanoid robot. After being fused with the humanoid robot body information, the real-time position of the humanoid robot can be estimated when deployed.

[0017] The specific steps include:

[0018] S1. Training of the underlying reinforcement learning model

[0019] The lower-level policy network and estimator are trained to obtain the policy model and the estimator model. The lower-level policy model is trained based on the AMP framework and a state estimator is introduced. The estimator model estimates the internal state and environmental state of the humanoid robot itself by using contrastive learning. The estimator includes the velocity estimation and implicit response of the humanoid robot, which are optimized through contrastive learning to approach the subsequent state of the humanoid robot.

[0020] The estimator training model based on contrastive learning is trained as follows:

[0021] Definition of positive and negative sample pairs:

[0022] In each iteration, the trajectories of each environment are collected as a batch of data. If o t+1 and t-H：t belong to the same trajectory, then they are a positive pair; otherwise, they are a negative pair.

[0023] Feature extraction:

[0024] Use the source encoder and the target encoder to transform the historical observations o t-H：t and subsequent observations o t+1 The feature vector z mapped to the high-dimensional space s and z t , z t is the hidden vector,

[0025] Normalization:

[0026] The eigenvectors are L2-normalized to ensure that they are located on the unit sphere of the high-dimensional space.

[0027] Prototype learning:

[0028] Use prototypes to predict the probability of cluster assignment to which a feature vector belongs. Prototypes are predefined points fixed in the feature space to help classification and contrastive learning.

[0029] Optimization goal:

[0030] The optimization goal is to maximize the prediction accuracy, that is, to maximize the log-likelihood sum of the matching probabilities of the positive sample pairs. The formula is as follows:

[0031]

[0032] in, and are the actual cluster assignments for the source and destination vectors, and is the predicted cluster assignment probability;

[0033] S2. Training of upper-level reinforcement learning model

[0034] S21. Phase 1: Core training phase based on simulation environment:

[0035] The PPO algorithm is used to train multiple networks.

[0036] The PPO algorithm is based on the Actor-Critic framework, where the Actor is the upper-layer policy network and the input is multi-frame observation information. Where H is the number of overlapping frames, H = 5, multi-frame observation information Contains the target point information g ht , target point information g ht It is expressed as the cosine and sin values ​​of the angle between the humanoid robot and the target point, and the ratio of the current distance to the target point and the initial task distance. The output is action a ht After training, the upper-layer strategy model Critic is obtained as an evaluation network, which is used to evaluate the actions generated by the current Actor.

[0037] Based on the above two networks, a terrain processing network is trained to extract features from terrain information. The input information of this network is the terrain height map within the range of 2m×3.2m in front of the humanoid robot, which is a series of two-dimensional discrete points. It is stretched into a one-dimensional vector m t After that, a terrain height map is formed.

[0038] The above three networks are all MLP networks.

[0039] By assuming the expected agent behavior and combining the environmental information, a reward function suitable for the above three networks is designed to evaluate the quality of each step of the agent and obtain a specific reward value r ht ,

[0040] Evaluate the output value of the network It is an estimate of the expected value of future cumulative rewards. By interacting with the current latest strategy network and the simulation environment, the true value of the current reward V can be obtained. t , the two can calculate the loss through the mean square error, and then the gradient update of the evaluation network can be realized.

[0041] The loss function of the action network is achieved by minimizing the difference between the old strategy and the new strategy. The specific formula is:

[0042]

[0043] in,

[0044] θ represents the parameters of the policy network.

[0045] ·r t(θ) is the probability ratio, which is calculated as the ratio of the probability of an action under the new strategy to the probability of an action under the old strategy, i.e.

[0046] ·A t is the estimated value of the advantage function, which means that in state s t Take action a t The difference between the expected return of the current strategy and the expected return of the current strategy

[0047] clip(x, 1-∈, 1+∈) is a clipping function that limits the value x to the range [1-∈, 1+∈], which is used to reduce the variance of the policy update and improve the stability of the algorithm.

[0048] ∈ is a hyperparameter that controls the extent of clipping and is usually set to a small value such as 0.1 or 0.2

[0049] Therefore, the final loss function is expressed as:

[0050]

[0051] S22, Phase 2: Replacement training phase:

[0052] Take the terrain height map m t The input MLP terrain processing network is replaced by a depth map d t The CNN terrain processing network is used as the input, and the height map m corresponding to the terrain around the humanoid robot body is collected through simulation. t , depth map d t and output y t , construct a data set and conduct supervised learning training,

[0053] Use the depth map d t The CNN network used as input is used to fit the terrain height map processing network in the first stage for real machine deployment;

[0054] S3, the lower-level training model estimates the current speed and angular velocity of the humanoid robot, and after fusing them with the humanoid robot’s body information, estimates the real-time position of the humanoid robot when deployed;

[0055] S4. The upper-level strategy model obtained in S2 can directly obtain various information of the humanoid robot itself from the simulation environment during training. When deployed, it combines the position and angle estimates of the estimator in the lower-level network with its own IMU information to infer the state information, input it into the upper-level strategy network, and then output speed instructions to ultimately achieve obstacle avoidance.

[0056] As a preferred technical solution of the present invention: in step S21, the core training phase is completely performed in a simulation environment.

[0057] As a preferred technical solution of the present invention: the decision frequency of the upper reinforcement learning model is 3 Hz, the action frequency of the lower reinforcement learning model is 100 Hz, and the execution frequency of the PD controller is 1000 Hz.

[0058] In the above structure: The present invention realizes better navigation and obstacle avoidance effects of a humanoid robot using only a depth camera as a sensor by constructing a new algorithm architecture and a series of innovative technologies.

[0059] The present invention adopts a hierarchical training architecture, which includes an upper-layer reinforcement learning model and a lower-layer reinforcement learning model. The upper-layer reinforcement learning model is a vision-based reinforcement learning algorithm, which takes depth information and task information as input, and outputs speed instructions as the instruction input of the lower-layer reinforcement learning model. The upper-layer reinforcement learning model is a navigation and obstacle avoidance layer. At the same time, the lower-layer reinforcement learning model will estimate the current speed and angular velocity of the humanoid robot. After fusing with the humanoid robot's main body information, the real-time position of the robot can be estimated during deployment.

[0060] Therefore, the upper and lower layers are closely related and complement each other. Before the implementation of the present invention, the lower layer reinforcement learning model needs to be trained in advance. After the training is completed, the parameters of the network in the lower layer reinforcement learning model are fixed.

[0061] Lower-level reinforcement learning model: By optimizing the form and algorithm of the estimator and canceling the input of phase information, the humanoid robot can move with a more anthropomorphic gait and has very high stability. It can withstand a strong push and kick from an adult with a force of about 600N. Through the above optimization, the humanoid robot can achieve a more anthropomorphic gait and better command following.

[0062] Upper-level reinforcement learning model: The training is divided into two stages. The first stage is completed based on a simulation environment, taking the terrain height map and the humanoid machine's own information as input and speed as output. The second stage uses a CNN network with a depth map as input to fit the terrain height map processing network in the first stage for actual machine deployment.

[0063] The present invention sets a layered architecture so that the navigation and obstacle avoidance capabilities of the humanoid robot can be almost decoupled from the movement capabilities, and the network can be trained separately, which is convenient for problem analysis and optimization. At the same time, the upper and lower layers are connected. The lower layer reinforcement learning model estimates the speed with high estimation accuracy, so that the speed and position of the fuselage can be estimated without external positioning. The upper layer reinforcement learning model adjusts the speed by analyzing the visual image, and finally achieves obstacle avoidance and navigation.

[0064] Depth map fitting: Through a simple and efficient supervised learning method, the terrain height map input can be converted into the camera depth map input without modifying other networks in the framework, thus achieving migration from simulation to reality.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] This invention achieves better navigation and obstacle avoidance effects for humanoid robots by constructing a new algorithm architecture and a series of innovative technologies, using only a depth camera as a sensor. The details are as follows:

[0067] Pioneering layering algorithm:

[0068] The hierarchical training architecture adopted by the present invention can independently train the upper and lower layers at different times. At the same time, the better the instruction followability of the lower layer, the higher the accuracy of the speed estimation, the better the effect of the upper layer model, and the full play of the gait capability of the lower layer.

[0069] Higher movement stability:

[0070] The present invention has been optimized and modified so that the lower model can stand on the spot, switch seamlessly from standing to walking, and has very strong anti-interference ability.

[0071] Humanized gait:

[0072] Through imitation learning components and corresponding optimization, the robot's accuracy in imitating human movements has been significantly improved, making the robot's movements closer to natural human behavior, such as standing still when stationary.

[0073] Accurate obstacle avoidance effect:

[0074] The upper network in the present invention can accurately output instructions so that the robot can avoid obstacles based on the obstacle information perceived by the current depth map and react quickly.

[0075] Accurate navigation effect:

[0076] In the present invention, by performing speed estimation at the lower layer and fusing it with the actual information of the robot body, the current position of the robot can be accurately estimated to achieve the effect of fixed-point arrival.

[0077] Generalization ability:

[0078] Only block-shaped objects of different lengths, widths and heights were used as obstacles in the simulation environment. However, in actual tests, it was found that the robot could completely avoid people and other obstacles that did not appear in the simulation environment.

[0079] Through the above technical effects, the present invention promotes the development of humanoid robot technology, improves the mobility and obstacle avoidance capabilities of humanoid robots in future application scenarios, and provides possibilities for the realization of more application scenarios, especially in scenarios that require close interaction between humanoid robots and humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0082] like Figure 1 As shown, the present invention proposes a humanoid robot hierarchical AI navigation obstacle avoidance method based on depth information, which is characterized by: designing an upper-layer reinforcement learning model and a lower-layer reinforcement learning model, wherein the upper-layer reinforcement learning model is used to take various information of the humanoid robot as input and output speed instructions, and the output speed instructions are used as the instruction input of the lower-layer reinforcement learning model. The lower-layer reinforcement learning model outputs the torque of each joint for controlling the movement of the robot, and includes an estimator, an estimation network and a lower-layer strategy network, which will estimate the current speed and angular velocity of the humanoid robot. After being fused with the humanoid robot body information, the real-time position of the humanoid robot can be estimated during deployment.

[0083] The specific steps include:

[0084] S1. Training of the underlying reinforcement learning model

[0085] The lower-level policy network and estimator are trained to obtain the policy model and the estimator model. The lower-level policy model is trained based on the AMP framework and a state estimator is introduced. The estimator model estimates the internal state and environmental state of the humanoid robot itself by using contrastive learning. The estimator includes the velocity estimation and implicit response of the humanoid robot, which are optimized through contrastive learning to approach the subsequent state of the humanoid robot.

[0086] The estimator training model based on contrastive learning is trained as follows:

[0087] Definition of positive and negative sample pairs:

[0088] In each iteration, the trajectories of each environment are collected as a batch of data. If o t+1 and t-H：t belong to the same trajectory, then they are a positive pair; otherwise, they are a negative pair.

[0089] Feature extraction:

[0090] Use the source encoder and the target encoder to transform the historical observations ot-H：t and subsequent observations o t+1 The feature vector z mapped to the high-dimensional space s and z t , z t is the hidden vector,

[0091] Normalization:

[0092] The eigenvectors are L2-normalized to ensure that they are located on the unit sphere of the high-dimensional space.

[0093] Prototype learning:

[0094] Use prototypes to predict the probability of cluster assignment to which a feature vector belongs. Prototypes are predefined points fixed in the feature space to help classification and contrastive learning.

[0095] Optimization goal:

[0096] The optimization goal is to maximize the prediction accuracy, that is, to maximize the log-likelihood sum of the matching probabilities of the positive sample pairs. The formula is as follows:

[0097]

[0098] in, and are the actual cluster assignments for the source and destination vectors, and is the predicted cluster assignment probability;

[0099] S2. Training of upper-level reinforcement learning model

[0100] S21. Phase 1: Core training phase based on simulation environment:

[0101] The PPO algorithm is used to train multiple networks.

[0102] The PPO algorithm is based on the Actor-Critic framework, where the Actor is the upper-layer policy network and the input is multi-frame observation information. Where H is the number of overlapping frames, H = 5, multi-frame observation information Contains the target point information g ht , target point information g ht It is expressed as the cosine and sin values ​​of the angle between the humanoid robot and the target point, and the ratio of the current distance to the target point and the initial task distance. The output is action a ht , after training, we get the upper strategy model

[0103] Critic is an evaluation network, which is used to evaluate the actions of the current Actor.

[0104] Based on the above two networks, a terrain processing network is trained to extract features from terrain information. The input information of this network is the terrain height map within the range of 2m×3.2m in front of the humanoid robot, which is a series of two-dimensional discrete points. After stretching it into a one-dimensional vector mt, a terrain height map is formed.

[0105] The above three networks are all MLP networks.

[0106] By assuming the expected agent behavior and combining the environmental information, a reward function suitable for the above three networks is designed to evaluate the quality of each step of the agent and obtain a specific reward value r ht ,

[0107] Evaluate the output value of the network It is an estimate of the expected value of future cumulative rewards. By interacting with the current latest strategy network and the simulation environment, the true value of the current reward V can be obtained. t , the two can calculate the loss through the mean square error, and then the gradient update of the evaluation network can be realized.

[0108] The loss function of the action network is achieved by minimizing the difference between the old strategy and the new strategy. The specific formula is:

[0109]

[0110] in,

[0111] θ represents the parameters of the policy network.

[0112] ·r t (θ) is the probability ratio, calculated as the ratio of the probability of an action under the new strategy to the probability of an action under the old strategy, i.e.

[0113] ·A t is the estimated value of the advantage function, which means that in state s t Take action a t The difference between the expected return of the current strategy and the expected return of the current strategy

[0114] clip(x, 1-∈, 1+∈) is a clipping function that limits the value x to the range [1-∈, 1+∈], which is used to reduce the variance of the policy update and improve the stability of the algorithm.

[0115] ∈ is a hyperparameter that controls the extent of clipping and is usually set to a small value such as 0.1 or 0.2

[0116] Therefore, the final loss function is expressed as:

[0117]

[0118] S22, Phase 2: Replacement training phase:

[0119] Take the terrain height map m t The input MLP terrain processing network is replaced by a depth map d t The CNN terrain processing network is used as the input, and the height map m corresponding to the terrain around the humanoid robot body is collected through simulation. t , depth map d t and output y t , construct a data set and conduct supervised learning training,

[0120] Use the depth map d t The CNN network used as input is used to fit the terrain height map processing network in the first stage for real machine deployment;

[0121] S3, the lower-level training model estimates the current speed and angular velocity of the humanoid robot, and after fusing them with the humanoid robot’s body information, estimates the real-time position of the humanoid robot when deployed;

[0122] S4. The upper-level strategy model obtained in S2 can directly obtain various information of the humanoid robot from the simulation environment during training. When deployed, it combines the position and angle estimation of the estimator in the lower-level network with its own IMU information to infer the state information, input it into the upper-level strategy network, and then output the speed command to finally achieve obstacle avoidance.

[0123] In step S21, the core training phase is completely performed in a simulation environment.

[0124] The decision frequency of the upper reinforcement learning model is 3 Hz, the action frequency of the lower reinforcement learning model is 100 Hz, and the execution frequency of the PD controller is 1000 Hz.

[0125] The present invention constructs a new algorithm architecture and a series of innovative technologies to enable a humanoid robot to achieve better navigation and obstacle avoidance effects using only a depth camera as a sensor.

[0126] The present invention adopts a hierarchical training architecture, which includes an upper-layer reinforcement learning model and a lower-layer reinforcement learning model. The upper-layer reinforcement learning model is a vision-based reinforcement learning algorithm, which takes depth information and task information as input, and outputs speed instructions as the instruction input of the lower-layer reinforcement learning model. The upper-layer reinforcement learning model is a navigation and obstacle avoidance layer. At the same time, the lower-layer reinforcement learning model will estimate the current speed and angular velocity of the humanoid robot. After fusing with the humanoid robot's main body information, the real-time position of the robot can be estimated during deployment.

[0127] Therefore, the upper and lower layers are closely related and complement each other. Before the implementation of the present invention, the lower layer reinforcement learning model needs to be trained in advance. After the training is completed, the parameters of the network in the lower layer reinforcement learning model are fixed.

[0128] Lower-level reinforcement learning model: By optimizing the form and algorithm of the estimator and canceling the input of phase information, the humanoid robot can move with a more anthropomorphic gait and has very high stability. It can withstand a strong push and kick from an adult with a force of about 200N. Through the above optimization, the humanoid robot can achieve a more anthropomorphic gait and better command following ability.

[0129] Upper-level reinforcement learning model: The training is divided into two stages. The first stage is completed based on a simulation environment, taking the terrain height map and the humanoid machine's own information as input and speed as output. The second stage uses a CNN network with a depth map as input to fit the terrain height map processing network in the first stage for actual machine deployment.

[0130] The present invention sets a layered architecture so that the navigation and obstacle avoidance capabilities of the humanoid robot can be almost decoupled from the movement capabilities, and the network can be trained separately, which is convenient for problem analysis and optimization. At the same time, the upper and lower layers are connected. The lower layer reinforcement learning model estimates the speed with high estimation accuracy, so that the speed and position of the fuselage can be estimated without external positioning. The upper layer reinforcement learning model adjusts the speed by analyzing the visual image, and finally achieves obstacle avoidance and navigation.

[0131] Depth map fitting: Through a simple and efficient supervised learning method, the terrain height map input can be converted into the camera depth map input without modifying other networks in the framework, thus achieving migration from simulation to reality.

[0132] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A humanoid robot hierarchical AI navigation and obstacle avoidance method based on depth information, characterized in that: An upper-level reinforcement learning model and a lower-level reinforcement learning model are designed. The upper-level reinforcement learning model is used to take various information of the humanoid robot as input and output speed instructions. The output speed instructions are used as the instruction input of the lower-level reinforcement learning model. The lower-level reinforcement learning model outputs the position of each joint, which is subsequently converted into torque through a PD controller to control the movement of the robot. It also includes an estimator, an estimation network, and a lower-level policy network. It will estimate the current speed and angular velocity of the humanoid robot. After fusing with the humanoid robot's main body information, the real-time position of the humanoid robot can be estimated during deployment. The specific steps include: S1. Training of the underlying reinforcement learning model The lower-level policy network and the estimator are trained to obtain the lower-level policy model and the estimator model. The lower-level policy model is trained based on the AMP framework, and a state estimator is introduced. The estimator model estimates the internal state and environmental state of the humanoid robot itself by using a contrastive learning method. The estimator includes the speed estimation and implicit response of the humanoid robot, which are optimized through contrastive learning to approach the subsequent state of the humanoid robot. The estimator training model based on contrastive learning is trained as follows: Definition of positive and negative sample pairs: In each iteration, the trajectories of each environment are collected as a batch of data. If o t+1 and t-H：t belong to the same trajectory, then they are a positive pair; otherwise, they are a negative pair. Feature extraction: Use the source encoder and the target encoder to transform the historical observations o t-H：t and subsequent observations o t+1 The feature vector z mapped to the high-dimensional space s and z t , z t is the hidden vector, Normalization: The eigenvectors are L2-normalized to ensure that they are located on the unit sphere of the high-dimensional space. Prototype learning: Use prototypes to predict the probability of cluster assignment to which a feature vector belongs. Prototypes are predefined points fixed in the feature space to help classification and contrastive learning. Optimization goal: The optimization goal is to maximize the prediction accuracy, that is, to maximize the log-likelihood sum of the matching probabilities of the positive sample pairs. The formula is as follows: in, and are the actual cluster assignments for the source and destination vectors, and is the predicted cluster assignment probability; S2. Training of upper-level reinforcement learning model S21. Phase 1: Core training phase based on simulation environment: The PPO algorithm is used to train multiple networks. The PPO algorithm is based on the Actor-Critic framework, where the Actor is the upper-layer policy network and the input is multi-frame observation information. Where H is the number of overlapping frames, H = 5, multi-frame observation information The target point information g is included ht , target point information g ht It is expressed as the cosine and sin values ​​of the angle between the humanoid robot and the target point, and the ratio of the current distance to the target point and the initial task distance. The output is action a ht , after training, the upper strategy model is obtained; Critic is an evaluation network, which is used to evaluate the actions of the current Actor. Based on the above two networks, a terrain processing network is trained to extract features from terrain information. The input information of this network is the terrain height map within the range of 2m×3.2m in front of the humanoid robot, which is a series of two-dimensional discrete points. It is stretched into a one-dimensional vector m t After that, a terrain height map is formed. The above three networks are all MLP networks. By assuming the expected agent behavior and combining the environmental information, a reward function suitable for the above three networks is designed to evaluate the quality of each step of the agent and obtain a specific reward value r ht , Evaluate the output value of the network It is an estimate of the expected value of future cumulative rewards. By interacting with the current latest strategy network and the simulation environment, the true value of the current reward V can be obtained. t , the two can calculate the loss through the mean square error, and then the gradient update of the evaluation network can be realized. The loss function of the action network is achieved by minimizing the difference between the old strategy and the new strategy. The specific formula is: in, θ represents the parameters of the policy network. ·r t (θ) is the probability ratio, calculated as the ratio of the probability of an action under the new policy to the probability of an action under the old policy, Right now ·A t is the estimated value of the advantage function, which means that in state s t Take action a t The difference between the expected return of the current strategy and the expected return of the current strategy clip(x, 1-∈, 1+∈) is a clipping function that limits the value x to the range [1-∈, 1+∈], which is used to reduce the variance of the policy update and improve the stability of the algorithm. ∈ is a hyperparameter that controls the extent of clipping and is usually set to a small value such as 0.1 or 0.2 Therefore, the final loss function is expressed as: S22, Phase 2: Replacement training phase: Take the terrain height map m t The input MLP terrain processing network is replaced by a depth map d t The CNN terrain processing network is used as the input, and the height map m corresponding to the terrain around the humanoid robot body is collected through simulation. t , depth map d t and output y t , construct a data set and conduct supervised learning training, Use the depth map d t The CNN network used as input is used to fit the terrain height map processing network in the first stage for real machine deployment; S3, the lower-level training model estimates the current speed and angular velocity of the humanoid robot, and after fusing them with the humanoid robot’s body information, estimates the real-time position of the humanoid robot when deployed; S4. The upper-level strategy model obtained in S2 can directly obtain various information about the humanoid robot from the simulation environment during training, but cannot obtain all the information during deployment. Therefore, it is necessary to combine the position and angle estimates of the estimator in the lower-level network with its own IMU information to infer the state information, input it into the upper-level strategy network, and then output the speed command to ultimately achieve obstacle avoidance.

2. According to claim 1, a humanoid robot hierarchical AI navigation obstacle avoidance method based on depth information is characterized by: In step S21, the core training phase is completely performed in a simulation environment.

3. The method for layered AI navigation and obstacle avoidance of a humanoid robot based on depth information according to claim 1, characterized in that: The decision frequency of the upper layer is 3 Hz, the action frequency of the lower layer is 100 Hz, and the execution frequency of the PD controller is 1000 Hz.

Images