A method and system for navigation control of a humanoid robot in a three-dimensional complex environment

Abstract

The application discloses a kind of three-dimensional complex environment-oriented humanoid robot navigation control method and system, it is related to humanoid robot autonomous navigation and depth reinforcement learning technical field, to solve the problems, such as three-dimensional navigation in prior art is difficult to consider global planning and local gait generation, landing point support evaluation is insufficient, decision interpretability is poor, complex terrain training convergence difficulty etc.The application constructs 2.5D local height map by depth camera, designs attention mechanism that fuses target orientation, robot motion state and terrain feature to generate weighted terrain encoding, constructs reinforcement learning framework containing attention guide reward, completes strategy learning based on TD3 algorithm and two-stage progressive training.The application significantly improves landing point selection accuracy and navigation success rate under complex terrain, enhances decision interpretability, and is suitable for humanoid robot autonomous navigation in unstructured environment.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot autonomous navigation and deep reinforcement learning technology, and in particular to a humanoid robot navigation control method and system for complex three-dimensional environments. Background Technology

[0002] Humanoid robots, with their humanoid design, can be directly adapted to human production and living scenarios without requiring large-scale modifications to existing infrastructure. They possess far superior mobility in unstructured terrain compared to traditional wheeled robots, making them a core intelligent equipment platform for advanced autonomous tasks in complex environments. However, in practical engineering applications, the high-frequency periodic interference introduced by the bipedal gait of humanoid robots easily leads to significant localization drift and mapping distortion in traditional Simultaneous Localization and Mapping (SLAM) algorithms, severely restricting the robustness and consistency of system state estimation. Furthermore, achieving real-time, collision-free motion planning and gait decision-making in complex 3D scenes remains a core challenge that urgently needs to be overcome. These technological bottlenecks significantly limit the autonomous operation capabilities of humanoid robots and their successful implementation in various scenarios.

[0003] In recent years, deep reinforcement learning has been widely applied in the field of robot navigation due to its powerful processing capabilities for complex sequential decision-making problems. Existing research includes: some scholars have proposed interactive navigation methods for quadruped robots based on large language models, using primitive tree structures to complete high-level task planning, but these do not consider the dynamic envelope of the swinging legs and the support constraints of footholds unique to bipedal robots; other studies have designed bipedal robot gait following controllers based on reinforcement learning, which can adaptively adjust stride length and stride height through local height fields to achieve stable walking on complex terrain, but their gait command generation relies on heuristic rules and lacks proactive assessment and refined modeling of foothold safety; and there are also vision-assisted teacher-student reinforcement learning frameworks that can perceive multiple types of terrain features through depth images, but they do not introduce attention mechanisms to achieve adaptive extraction of local terrain features, and cannot explicitly guide the robot to focus on safe and passable areas.

[0004] In summary, existing reinforcement learning-based methods for humanoid robot navigation and gait control still suffer from three major technical shortcomings: First, most methods simplify the robot to a point mass model, considering only planar motion without rigorously evaluating the size, flatness, and continuity of the support area. This leads to the failure of theoretically planned paths in actual execution due to infeasible landing points. Second, in 3D scenes containing steps, ramps, and gaps, existing methods lack proactive perception and refined modeling of local terrain safety, failing to fully integrate robot gait constraints and landing point support characteristics during path planning. Third, the reward function design of existing methods does not incorporate a guidance mechanism for attention-focusing behavior, resulting in a lack of interpretability in the decision-making process and difficulty in achieving stable end-to-end navigation in complex terrain. These shortcomings severely restrict the autonomous navigation capabilities and environmental adaptability of humanoid robots in 3D undulating environments. Summary of the Invention

[0005] To address these issues, embodiments of the present invention provide a navigation control method and system for humanoid robots in complex three-dimensional environments. This system solves the problems in existing technologies where humanoid robot navigation in three-dimensional environments struggles to balance global path planning and local gait generation, lacks refined assessment of the supportability of landing points, fails to fully utilize local terrain information to guide the selection of safe landing points, and lacks interpretability in the decision-making process, making it difficult to achieve stable and efficient end-to-end autonomous navigation in complex three-dimensional undulating terrain.

[0006] To address the aforementioned technical problems, this invention provides a navigation and control method for a humanoid robot in a complex three-dimensional environment, comprising the following steps: S1: Construct a complex 3D environment scene for humanoid robot navigation. The scene includes static obstacles, undulating terrain and impassable areas to provide a navigation task operating environment for the humanoid robot. S2: Real-time acquisition of terrain information of the surrounding environment through the depth camera mounted on the humanoid robot to construct a 2.5D local height map centered on the robot; S3: Use a convolutional neural network to extract features from the 2.5D local elevation map to obtain the deep structural features of the local terrain; S4: Based on the current motion state of the humanoid robot and the orientation information of the navigation target, construct the query vector of the attention mechanism; based on the deep terrain structure features and grid spatial location information obtained in step S3, construct the key vector and value vector of the attention mechanism. S5: The position, orientation and terrain flatness information of the navigation target are fused to calculate the attention score of each grid. After normalization, an attention weight map and weighted terrain coding features are generated. S6: Model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing ontological state and weighted terrain coding features, and define a continuous action space containing forward stride, lateral stride and yaw angle changes. S7: Design a multi-objective comprehensive reward function that includes attention-guided rewards to enhance the humanoid robot's behavior in selecting safe landing points; S8: Construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs. S9: Based on the dual-delay deep deterministic policy gradient algorithm, the parameters of the policy network and the dual-value network are updated by combining the priority experience replay mechanism. S10: A two-stage progressive training strategy is adopted to train the network, and the trained strategy network is deployed to the humanoid robot platform to achieve real-time autonomous navigation in a complex three-dimensional environment.

[0007] Preferably, in step S2, the method for constructing a 2.5D local height map centered on the robot is as follows: The depth images captured by the depth camera are converted into 3D point clouds in a local coordinate system centered on the robot. The 3D point clouds are then projected onto a horizontal plane, and each grid cell stores the height value of that location relative to the robot's standing plane. For grid cells without valid depth data, neighboring grid cells are interpolated to fill the gap, resulting in a 2.5D local height map covering a preset range around the robot.

[0008] Preferably, in step S3, the convolutional neural network is a three-layer convolutional structure, specifically configured as follows: The first layer is a 16-channel 3×3 convolutional layer, the second layer is a 32-channel 3×3 convolutional layer, and the third layer is a 64-channel 3×3 convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to extract the structural features of the terrain's edges, gaps, and passable areas.

[0009] Preferably, in step S4, the method for constructing the query vector of the attention mechanism is as follows: Extract the orientation vector of the navigation target relative to the robot Normalized distance from robot to target Robot swinging leg logo Relative position of feet The 6-dimensional initial query vector is concatenated and then mapped through a fully connected layer to obtain the query features of the attention mechanism. .

[0010] Preferably, in step S5, the method for calculating the attention score of each grid is as follows: Calculate the position bias term based on the Gaussian kernel respectively. Orientation bias term based on target orientation consistency Landing point flatness score based on grid neighborhood height variance And obstacle penalty items for areas with height differences exceeding the threshold. ; The attention score for each grid is obtained by fusing the above items according to preset weights. The formula for calculating the attention score is: ; The attention scores are normalized using Softmax with a learnable temperature coefficient to obtain the attention weights of each grid, thus forming an attention weight map.

[0011] Preferably, in step S7, the multi-objective comprehensive reward function consists of five parts: task completion reward, distance progress reward, angle correction reward, attention guidance reward, and step penalty. The attention-guided reward is set based on the attention weight map generated in step S5. When the attention weight of the grid corresponding to the actual landing point selected by the robot is higher than the preset threshold, a positive reward is given; otherwise, a negative penalty is given. The formula for calculating the total reward is: ; In the formula, Rewards for completing the task. As a reward for progress, Rewards for angle correction. The reward is for attention-guided actions, and -0.01 is the penalty for each step taken.

[0012] Preferably, in step S8, the policy network concatenates the attention weight map with the target direction vector and inputs it into a three-layer fully connected network. The three-layer fully connected network consists of a 256-dimensional ReLU layer, a 128-dimensional ReLU layer, and a 4-dimensional Tanh layer, and the output is scaled to obtain the navigation control action of the humanoid robot. The dual-value network employs two Q-networks with identical structures and independent initialization. Each Q-network first extracts environmental features through an attention-based terrain encoder, then concatenates the extracted features with the current action and inputs them into a three-layer fully connected network. The three-layer fully connected network consists of a 256-dimensional ReLU layer, a 128-dimensional ReLU layer, and a 1-dimensional linear output layer, which outputs the Q-value estimate of the state-action pair.

[0013] Preferably, in step S9, the method for updating the network parameters of the policy network and the dual-value network based on the dual-delay deep deterministic policy gradient algorithm combined with the priority experience replay mechanism is as follows: Experience tuples generated during robot navigation Stored to a priority experience replay buffer, the experience tuple containing the current state. ,action Instant rewards Next state End of round marker ; Each time a batch of experience is sampled from the buffer, two value networks are updated first. The update objective of the value networks is to minimize the mean square error between the predicted Q value and the target value. The formula for calculating the target value is: ; In the formula, For the target Q value, As a discount factor, To obtain the minimum value of the two Q-network outputs, For the first A target value network, with parameters as follows: , For the first The objective policy network is in state The action of outputting below, The target policy noise; The policy network employs a delayed update strategy, performing one policy network update after every two value network updates. The optimization objective of the policy network is to maximize the output of the first value network, i.e.: ; In the formula, Let the loss function be the policy network. For the experience replay buffer Mid-sampled state Seeking expectations, For the first value network, the parameters are: , For the current policy network in state The action to be output; Both the policy network and the target network corresponding to the value network use a soft update method to update parameters. The soft update formula is: ; ; In the formula, For the first Parameters of a target value network For the first The parameters of the current value network, For the parameters of the target policy network, These are the parameters of the current policy network. This is the soft update coefficient. This is an assignment operation, indicating that the target network parameters are updated gradually.

[0014] Preferably, in step S10, the method for training the network using a two-stage progressive training strategy is as follows: The first stage is training on flat terrain without obstacles. A flat map with no height undulations and no obstacles is used. The training is conducted for a preset number of rounds so that the robot can master the basic navigation ability to walk towards the target. The second stage is training on complex 3D terrain. The system switches to a complex 3D environment map that includes static obstacles, undulating terrain, and impassable areas. Training continues until the network converges, enabling the robot to master the ability to select safe landing points and adapt to complex terrain.

[0015] This invention also provides a humanoid robot navigation control system for complex three-dimensional environments. This system is used to implement the aforementioned humanoid robot navigation control method for complex three-dimensional environments, specifically including: A 3D complex navigation scene construction module is used to build a 3D complex environment scene for humanoid robot navigation. The scene includes static obstacles, undulating terrain and impassable areas, providing a navigation task operation environment for humanoid robots. The 2.5D local height map construction module is used to collect terrain information of the surrounding environment in real time through the depth camera on the humanoid robot and construct a 2.5D local height map centered on the robot. The terrain deep structure feature extraction module is used to extract features from the 2.5D local height map using a convolutional neural network to obtain the deep structure features of the local terrain. The attention mechanism vector construction module is used to construct the query vector of the attention mechanism based on the current motion state of the humanoid robot and the orientation information of the navigation target; and to construct the key vector and value vector of the attention mechanism based on the deep terrain structure features and grid spatial location information obtained by the deep terrain structure feature extraction module. The adaptive attention terrain coding module is used to fuse the position, orientation and terrain flatness information of the navigation target, calculate the attention score of each grid, and generate an attention weight map and weighted terrain coding features after normalization. The navigation task Markov decision modeling module is used to model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing the ontological state and weighted terrain coding features, and define a continuous action space containing forward stride, lateral stride and yaw angle changes. The attention-guided multi-objective reward function design module is used to design a multi-objective comprehensive reward function that includes attention-guided rewards, in order to enhance the humanoid robot's behavior in selecting safe landing points; A dual-network architecture deep reinforcement learning network construction module is used to construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs. The dual-delay deep deterministic policy gradient update module is used to update the parameters of the policy network and the dual-value network based on the dual-delay deep deterministic policy gradient algorithm and combined with the priority experience replay mechanism. The two-stage progressive training and navigation deployment module is used to train the network using a two-stage progressive training strategy, and then deploy the trained strategy network to the humanoid robot platform to achieve real-time autonomous navigation in complex three-dimensional environments.

[0016] As can be seen from the above technical solutions, this invention application has the following beneficial effects: (1) This invention achieves adaptive focusing on safe and footholding areas by integrating a multi-dimensional attention-based terrain encoder that incorporates target orientation, robot motion state, and terrain flatness, along with an attention-guided reward mechanism. This addresses the core shortcomings of existing technologies, such as insufficient assessment of foothold support and the lack of interpretability in black-box navigation decisions. Experimental results show that this design achieves a foothold accuracy of over 90% in complex scenarios such as obstacle avoidance, balance beams, and beam jumps, representing a maximum improvement of 48.5% compared to the comparison method. Furthermore, the quantifiable and visualized attention weight map enables interpretability of the navigation decision-making process, facilitating algorithm optimization and troubleshooting for engineering implementation.

[0017] (2) This invention adopts a two-stage progressive training strategy of "basic ability learning for flat terrain - refined adaptation for complex terrain", combined with a priority experience replay mechanism and a double-delay deep deterministic policy gradient (TD3) algorithm, which solves the industry pain points of low efficiency, difficulty in convergence, and easy overfitting of direct training samples in reinforcement learning in complex 3D environments. This design enables the model to achieve a navigation success rate of over 80% after about 3000 training rounds, which is more than 30% faster than the scheme without two-stage training. At the same time, it significantly reduces policy fluctuations during training and significantly improves the robustness and generalization of the final navigation strategy.

[0018] (3) This invention overcomes the shortcomings of existing technologies that simplify robots into point mass models. Based on a 2.5D local height map, it constructs a high-dimensional state space that integrates the robot's gait state and terrain features. It designs a continuous action space that matches the bipedal motion characteristics and a multi-objective comprehensive reward function, realizing end-to-end collaborative optimization of global path navigation and local foothold gait decision-making. This framework can seamlessly adapt to various unstructured 3D scenarios such as obstacle avoidance, narrow beam walking, and gap crossing, without relying on manually designed heuristic gait rules, greatly improving the environmental adaptability and autonomous operation capability of humanoid robots in complex life and industrial scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a navigation and control method for a humanoid robot in a complex three-dimensional environment provided by the present invention; Figure 2 This is a schematic diagram of the attention mechanism guidance in this invention, wherein (a) is a traversable gap, (b) is a jump beam, (c) is an obstacle, and (d) is a balancing beam; Figure 3 This is a diagram of the DRL navigation learning framework that incorporates attention in this invention; Figure 4 This is a schematic diagram of the first-stage training reward curve in this invention; Figure 5 This is a schematic diagram of the second-stage training reward curve in this invention; Figure 6 This is a comparison chart of the learning efficiency of different methods in this invention; Figure 7 A diagram comparing landing accuracy and attention focus in obstacle avoidance scenarios; Figure 8 This is a diagram comparing the accuracy of landing point and the degree of attention focus in a balanced beam scenario. Figure 9 This diagram illustrates the comparison between the accuracy of landing points and the degree of attention focus in a jump scene. Figure 10 This diagram illustrates the comparison between landing accuracy and attention focus in scenarios where gaps can be crossed. Figure 11 This is a block diagram of a humanoid robot navigation and control system for complex three-dimensional environments provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: This embodiment discloses a navigation control method for humanoid robots in complex 3D environments. Its core is to achieve adaptive focusing on safe landing areas through a terrain encoder incorporating an attention mechanism. It combines a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with a two-stage progressive training strategy to address the problems of low landing point selection accuracy, difficult training convergence, and poor decision interpretability when humanoid robots navigate in complex 3D terrain. The complete method flow of this embodiment is as follows: Figure 1 As shown, the attention mechanism guides the logic as follows: Figure 2 As shown, the deep reinforcement learning (DRL) navigation learning framework that incorporates attention is as follows: Figure 3 As shown.

[0022] This embodiment constructs a 12m×12m three-dimensional navigation environment in the ROS Gazebo simulation platform, deploys a Kuavo humanoid robot as the controlled object, and includes static obstacles and impassable areas in the scene. The robot needs to safely navigate from a random starting point to a designated target point. The method specifically includes the following steps: In step S1, a three-dimensional complex environment scene for humanoid robot navigation is constructed. The scene includes static obstacles, undulating terrain and impassable areas, providing a navigation task operating environment for the humanoid robot.

[0023] Specifically, this step provides a standardized navigation task execution and verification environment for the humanoid robot. The constructed 12m×12m 3D scene includes static obstacles, undulating terrain, and impassable areas, covering four typical navigation scenarios. Each scenario presents different challenges to the robot's foothold selection and motion stability. The specific scenario configuration is as follows: 1. Obstacle Avoidance Scenario: Six rectangular obstacles are randomly distributed within the scenario. The dimensions of each obstacle are as follows: A (1.2m×0.5m×0.5m), B (0.6m×1.0m×0.5m), C (0.7m×0.5m×0.5m), D (0.5m×1.0m×0.5m), E (0.6m×0.5m×0.5m), and F (0.5m×1.4m×0.5m). The total area occupied by the obstacles is... The obstacle density in the scene is approximately 1.65%. 2. Balanced beam scenario: A single narrow beam, 2.0m long and 0.3m wide, is set up in the scenario. The beam surface is 0.2m above the ground. The two sides of the beam are empty, impassable areas. The robot is required to walk stably on the narrow beam. 3. Beam Jumping Scenario: The scenario consists of three beams, each 3.0m long and 0.3m wide, forming a passageway. There is a 0.2m gap between adjacent beams, and the height difference between the beam surface and the ground is greater than 0.2m. The robot is required to continuously jump over the gaps to complete the passage. 4. Crossable Gap Scenario: A 0.2m wide and 0.2m deep ditch is set in the center of the scene, and the robot is required to cross the ditch directly to complete the navigation.

[0024] In step S2, the humanoid robot uses a depth camera to collect terrain information of the surrounding environment in real time to construct a 2.5D local height map centered on the robot.

[0025] Specifically, the robot uses a depth camera mounted on its head to collect depth and color images of the surrounding environment in real time. The collected depth images are first converted into three-dimensional point clouds in a local coordinate system centered on the robot. Then, the three-dimensional point clouds are projected onto a horizontal plane. Each grid cell stores the height value of that position relative to the robot's standing plane, forming a 2.5D local height map. For grid cells without valid depth data, interpolation with neighboring grid cells is used to fill the gaps, ensuring the integrity of the map.

[0026] In this embodiment, the final 2.5D local height map is 30×30 pixels in size and has a resolution of 0.05 meters per pixel. It can cover an area of ​​1.5m×1.5m around the robot, providing basic data for subsequent terrain feature extraction and attention encoding.

[0027] In step S3, a convolutional neural network is used to extract features from the 2.5D local height map to obtain the deep structural features of the local terrain.

[0028] Specifically, this step employs a three-layer convolutional neural network architecture to extract hierarchical features from the 2.5D local height map. The convolutional layers are configured as follows: the first layer is a 16-channel 3×3 convolutional layer, the second layer is a 32-channel 3×3 convolutional layer, and the third layer is a 64-channel 3×3 convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. This convolutional neural network learns and outputs low-level structural features such as edges, gaps, and passable areas in the local map, providing basic feature input for subsequent attention mechanism encoding.

[0029] In step S4, a query vector for the attention mechanism is constructed based on the current motion state of the humanoid robot and the orientation information of the navigation target; and a key vector and a value vector for the attention mechanism are constructed based on the deep terrain structure features and grid spatial location information obtained in step S3.

[0030] Specifically, the query vector for the attention mechanism is constructed based on the current motion state of the humanoid robot and the orientation information of the navigation target. The specific process is as follows: Let the direction vector of the target direction relative to the robot be... The normalized distance from the robot to the target is The robot's swinging leg logo is The relative position of the two feet is The above parameters are concatenated to obtain a 6-dimensional initial query vector, which is then mapped to query features through a fully connected layer. Complete the construction of the attention query vector.

[0031] Furthermore, based on the deep terrain structure features and grid spatial location information obtained in step S3, the key vector and value vector of the attention mechanism are constructed. The specific process is as follows: The 64-dimensional terrain features output from the convolutional layer are concatenated with the 2D relative coordinates of the raster, and then projected through independent fully connected layers to obtain the key features. AND value characteristics Complete the construction of attention key vectors and value vectors.

[0032] In step S5, the position, orientation and terrain flatness information of the navigation target are fused to calculate the attention score of each grid. After normalization, an attention weight map and weighted terrain coding features are generated.

[0033] Specifically, this step integrates goal orientation and terrain safety, and fuses multi-dimensional information to calculate the attention score for each grid cell in the local map. The specific calculation process is as follows: First, we introduce a position bias term based on a Gaussian kernel, calculated as follows: ; in, Grid points in a local map Relative to the robot's local coordinates, This is the local coordinate of the target point relative to the robot, and it is used to strengthen the weight of the region close to the target point.

[0034] Simultaneously, a direction bias term based on target direction consistency is introduced, calculated as follows: ; This item measures the consistency between the orientation of grid points in a local map and the target orientation, and strengthens the weight of areas facing the target orientation.

[0035] In addition, regarding terrain safety, the flatness score of the landing point is calculated using the following formula: ; This score is obtained by taking the variance of the height values ​​of the 3×3 neighborhood around each grid point in the local map. The smaller the variance, the flatter the area is, and the more suitable it is as a landing point. This item is used to strengthen the weight of flat and suitable landing areas.

[0036] In summary, an obstacle penalty item is also introduced. A penalty of -20 is applied to areas where the height difference exceeds the robot's maximum step height. The final attention score calculation formula for each grid is as follows: ; Furthermore, the attention scores calculated above are subjected to Softmax normalization with a temperature coefficient to obtain the attention weight of each grid point in the local map. The calculation formula is as follows: ; Among them, temperature coefficient Set as a learnable parameter with an initial value of 1.0, and adjust adaptively during training.

[0037] The attention weights of all grid points form a 30×30 attention weight map. This weight reflects the importance of each location in the local map in the current decision-making process. It can display high weights for locations facing or approaching the target point and for landing areas in the local map, guiding the robot to the correct location. The attention weight is then weighted and fused with the deep terrain structure features extracted in step S3 to finally generate weighted terrain coding features.

[0038] S6: Model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing ontological state and weighted terrain coding features, and define a continuous motion space containing forward stride, lateral stride and yaw angle changes.

[0039] Specifically, we first perform Markov decision process modeling: The navigation problem of humanoid robots in a 3D undulating environment is modeled as a standard Markov decision process. The robot needs to safely navigate from the starting point to a designated target point in a 2.5D map containing static obstacles and impassable areas. At each decision moment... The robot acquires its current state and outputs an action. The environment provides immediate feedback and rewards the robot, which then completes the state transition until the navigation task is completed or fails.

[0040] Secondly, a high-dimensional state space is constructed: the constructed high-dimensional state description is composed of ontology states and local maps with attention weights, with a total dimension of 913, and its specific composition is as follows: Local map portion: 30×30 pixels, resolution 0.05 m / pixel, covering an area of ​​1.5 m × 1.5 m around the robot, corresponding to 900-dimensional features; The ontology perception component is a 13-dimensional vector used to characterize the robot's own kinematics and task state, specifically including its relative position to the target. and Target direction vector as well as Euclidean distance to the target ; Robot facing the feature , and Robot-target position orientation error characteristics and ,in Current swing leg icon Where 0 represents the left leg swing and 1 represents the right leg swing; and the relative position of the feet, that is, the position of the swinging leg relative to the supporting leg. .

[0041] Finally, define the continuous action space: define the robot's action output as a three-dimensional continuous vector. The three dimensions represent the changes in forward stride, lateral stride, and yaw angle, respectively, and this action directly affects the robot's swing leg. At each decision step, the robot determines the direction of the swing leg based on the current swing leg marker. Calculate the target landing point position, and update the foot position and torso pose after the action is performed to complete the state transition.

[0042] S7: Design a multi-objective comprehensive reward function that includes attention-guided rewards to enhance the humanoid robot's behavior in selecting safe landing points.

[0043] Specifically, the multi-objective comprehensive reward function designed in this step comprehensively considers task completion, navigation efficiency, motion stability, and landing point safety. This allows the robot to pursue task completion while also ensuring safe and natural movement, explicitly reinforcing the humanoid robot's behavior in selecting safe landing points. The reward function consists of five parts: task completion reward, distance progress reward, angle correction reward, attention guidance reward, and step penalty. The specific definitions of each part are as follows: 1. Task completion reward: As the basic guide for navigation tasks, when the robot successfully reaches the target point (i.e. the distance between the robot and the target point is less than 0.1 meters), a positive reward of +100 is given; when the robot's foot is on an obstacle, or when more than 1 / 3 of its foot is in the gap causing it to miss, a penalty of -80 is given and the current round ends in failure, thus strengthening the robot's awareness of safe navigation.

[0044] 2. Distance Progress Reward: Used to guide the robot to continuously move closer to the target. The distance between the robot and the target at the current moment. The distance from the previous moment. The distance progress reward is defined as the reduction in distance per step: ; in, The initial distance between the robot and the target point is used for normalization, and double the penalty is imposed on the robot's behavior of moving away from the target.

[0045] 3. Angle Correction Reward: Used to assist the robot in maintaining the correct orientation towards the target. Let the current robot orientation unit vector be... The unit vector pointing from the robot to the target is Then the included angle between the two Angle reward is defined as: ; in, The orientation angle of the previous decision step is used to give positive rewards to the robot for correcting its orientation and aligning with the target.

[0046] 4. Attention-Guided Rewards: Based on the attention weight map generated in step S5 This setting reflects the importance of each location in the local map in the current decision, for the grid position corresponding to the actual selected landing point. The attention-guided reward is defined as: ; This reward can explicitly guide robots to choose safe areas with high attention weight, while improving the interpretability of the decision-making process.

[0047] 5. Step Penalty: This is used to encourage the robot to complete the task with the fewest possible steps. For each step, a small penalty of -0.01 is given, so that the robot tends to choose the more efficient path while ensuring safety.

[0048] Based on the above, the formula for calculating the total reward at each decision step is as follows: ; in, The reward for completing the mission is +100 upon reaching the finish line or -80 penalty upon mission failure; otherwise, it is 0. As a reward for progress, Rewards for angle correction. The reward is for attention-guided actions, and -0.01 is the penalty for each step taken.

[0049] S8: Construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs.

[0050] Specifically, a policy network is first constructed: the policy network is used to output the navigation control actions of the humanoid robot, and the network input is the attention weight map generated in step S5. The network, consisting of three fully connected layers, incorporates features from the concatenation of the target direction vector. Specifically, the first layer is a 256-dimensional ReLU layer, the second a 128-dimensional ReLU layer, and the third a 4-dimensional Tanh layer. The final output, multiplied by the maximum motion amplitude, yields the actual robot navigation control command. Explicit injection of the target direction vector ensures the network can directly perceive the target's location, preventing target information from attenuating during feature encoding.

[0051] Then, a dual-value network is constructed: the dual-value network is used to evaluate the value of state-action pairs. It adopts a dual-Q network structure, with two value networks initialized independently but with identical structures, which can effectively suppress the overestimation of value during reinforcement learning training. Each value network first extracts environmental features through an attention-based terrain encoder, then concatenates the extracted features with the current action, and inputs them into a three-layer fully connected network. The specific configuration is as follows: the first layer is a 256-dimensional ReLU layer, the second layer is a 128-dimensional ReLU layer, and the third layer is a 1-dimensional linear output layer, which finally outputs the Q-value estimate of the current state-action pair.

[0052] Finally, network initialization settings: all network weights are initialized using orthogonal initialization or Xavier initialization; the last layer of the policy network is initialized with a bias of 0, the weights are initialized orthogonally and the gain is set to 1.0, so that the network's initial output tends to be small-amplitude, which is convenient for environment exploration in the early stage of training.

[0053] S9: Based on the dual-delay deep deterministic policy gradient algorithm, the parameters of the policy network and the dual-value network are updated by combining the priority experience replay mechanism.

[0054] Specifically, this step deeply integrates the attention mechanism with the TD3 algorithm to form an interpretable end-to-end navigation learning system. The specific parameter update process is as follows: At each decision-making moment, the policy network outputs action commands based on the encoded features, driving the robot to perform a stepping action. After the action is executed, the environment provides feedback on the next moment's state and an immediate reward; this includes the current state. ,action ,award Next state and the end-of-round marker experience tuples The samples are stored in the priority experience replay buffer. Priority experience replay assigns priority to samples based on the size of the time-series difference error, enabling the model to learn from experiences with larger prediction errors more frequently and improving sample utilization efficiency.

[0055] During training, after sampling a batch of experience from the buffer, the two value networks are first updated. The update objective of the value networks is to minimize the mean square error between the predicted Q-value and the target value, where the target value is calculated using the following formula: ; In the formula, For the target Q value, As a discount factor, To obtain the minimum value of the two Q-network outputs, For the first A target value network, with parameters as follows: , For the first The objective policy network is in state The action of outputting below, This is the target policy noise, used to improve the robustness of the policy.

[0056] The policy network employs a delayed update strategy, performing one policy network update after every two value network updates. The optimization objective of the policy network is to maximize the output of the first value network, and its loss function is defined as: ; The policy network parameters are updated using the gradient ascent method, where... Let the loss function be the policy network. For the experience replay buffer Mid-sampled state Seeking expectations, For the first value network, the parameters are: , For the current policy network in state The action to output.

[0057] Both the policy network and the target network corresponding to the value network use a soft update method to update parameters. The soft update formula is: ; ; In the formula, For the first Parameters of a target value network For the first The parameters of the current value network, For the parameters of the target policy network, These are the parameters of the current policy network. In this embodiment, the soft update coefficient is used. , This is an assignment operation, indicating that the target network parameters are updated gradually.

[0058] In step S10, a two-stage progressive training strategy is used to train the network, and the trained strategy network is deployed to the humanoid robot platform to achieve real-time autonomous navigation in a complex three-dimensional environment.

[0059] Specifically, this step employs a two-stage progressive training strategy, enabling the robot to first master basic navigation capabilities in a simple environment, and then gradually transition to complex environments for refined learning. This addresses the problems of low efficiency and difficulty in convergence when directly training samples in complex 3D terrain. The specific training process is as follows: Phase 1: Training on flat terrain without obstacles: This phase uses a completely flat map without any height variations or obstacles. The training rounds are 500. The training objective is to enable the robot to learn the basic ability to walk towards the target, understand the guiding role of distance and angle rewards, and complete the learning of basic navigation capabilities. Phase 2: Training on undulating terrain with obstacles: This phase switches to the complex 3D map with height changes and obstacles built in step S1, and continues training until a total of 5000 rounds. The training goal is to enable the robot to learn to choose safe landing points, adapt to undulating terrain, and maintain navigation efficiency in complex environments, thus completing the refined learning of navigation capabilities in complex scenarios.

[0060] Furthermore, the trained and converged policy network is deployed on a physical humanoid robot platform. Based on the real-time perceived local height map and the robot's motion state, the policy network outputs navigation action commands in real time, driving the robot to complete real-time autonomous navigation tasks in a three-dimensional undulating environment.

[0061] This embodiment verifies the effectiveness of the method of the present invention through training process monitoring and ablation experiments. The relevant results are as follows: 1. Changes in reward during training: The reward curve for the first stage of training is as follows: Figure 4 As shown, due to the need to accumulate effective samples for priority experience replay, the reward is low in the initial stage. After 50 rounds, the robot begins to learn the basic stepping strategy, and the reward value increases significantly; the reward curve for the second stage of training is shown in the figure. Figure 5 As shown, in the early stages of training (500-2000 rounds), the reward fluctuates significantly due to the increased complexity of the environment. As training progresses (2000-4000 rounds), the model gradually learns effective safe navigation strategies, the reward curve continues to rise, and the focus of attention steadily increases.

[0062] 2. Ablation Experiment Results: To verify the effectiveness of each core module, three sets of comparative experiments were set up: the complete method (the method in this embodiment), training without the first stage of flat terrain, and encoding map features using a regular CNN without the attention terrain encoder. The learning efficiency of different methods is compared as follows: Figure 6As shown, the complete method achieves an average navigation success rate of over 80% for the four terrain types after approximately 3000 rounds, while the two ablation methods require over 4000 rounds to reach a similar level, and their final stability performance is significantly lower than that of the complete method.

[0063] 3. Multi-scenario performance test results: The comparison results of landing point accuracy and attention focus of different methods in four typical navigation scenarios are as follows: Figures 7 to 10 As shown, the complete method achieves the highest landing accuracy and attention focus across all terrains: (1) In obstacle avoidance scenarios, the accuracy of landing point reached 95.6±1.8%, and the focus of attention reached 93.8±2.1%; (2) In the scenario of balanced beams, the accuracy of landing point reached 92.8±2.3%, and the focus of attention reached 91.3±2.5%; (3) In the jumping scene, the accuracy of the landing point reached 90.1±2.0%, and the focus of attention reached 88.7±2.8%; (4) In scenarios where gaps can be crossed, the accuracy of landing points reached 96.2±1.2%, and the focus of attention reached 94.5±1.7%.

[0064] In the comparative experiment, after removing the attention encoder, the accuracy of the landing point in each scene decreased significantly, with the accuracy in the balanced beam scene being only 44.3±2.1%; after removing the two-stage training, the model's attention focus was significantly reduced, with the focus in the balanced beam scene being only 67.4±4.8%.

[0065] The experimental results above demonstrate that the method proposed in this embodiment, by integrating attention mechanisms and deep reinforcement learning, can effectively guide the robot to focus on the most navigational value area in the local height map, generating an interpretable attention weight map. This enables the robot to achieve stable walking and efficient navigation in complex 3D terrains such as obstacle avoidance, balance beams, jumping beams, and traversable gaps, verifying the synergistic effect of attention mechanisms and two-stage training strategies in improving the robot's terrain adaptability.

[0066] Example 2: like Figure 11 As shown, this invention provides a humanoid robot navigation and control system for complex three-dimensional environments. This system is used to implement the humanoid robot navigation and control method for complex three-dimensional environments described in Embodiment 1 above, specifically including: A 3D complex navigation scene construction module is used to build a 3D complex environment scene for humanoid robot navigation. The scene includes static obstacles, undulating terrain and impassable areas, providing a navigation task operation environment for humanoid robots. The 2.5D local height map construction module is used to collect terrain information of the surrounding environment in real time through the depth camera on the humanoid robot and construct a 2.5D local height map centered on the robot. The terrain deep structure feature extraction module is used to extract features from the 2.5D local height map using a convolutional neural network to obtain the deep structure features of the local terrain. The attention mechanism vector construction module is used to construct the query vector of the attention mechanism based on the current motion state of the humanoid robot and the orientation information of the navigation target; and to construct the key vector and value vector of the attention mechanism based on the deep terrain structure features and grid spatial location information obtained by the deep terrain structure feature extraction module. The adaptive attention terrain coding module is used to fuse the position, orientation and terrain flatness information of the navigation target, calculate the attention score of each grid, and generate an attention weight map and weighted terrain coding features after normalization. The navigation task Markov decision modeling module is used to model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing the ontological state and weighted terrain coding features, and define a continuous action space containing forward stride, lateral stride and yaw angle changes. The attention-guided multi-objective reward function design module is used to design a multi-objective comprehensive reward function that includes attention-guided rewards, in order to enhance the humanoid robot's behavior in selecting safe landing points; A dual-network architecture deep reinforcement learning network construction module is used to construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs. The dual-delay deep deterministic policy gradient update module is used to update the parameters of the policy network and the dual-value network based on the dual-delay deep deterministic policy gradient algorithm and combined with the priority experience replay mechanism. The two-stage progressive training and navigation deployment module is used to train the network using a two-stage progressive training strategy, and then deploy the trained strategy network to the humanoid robot platform to achieve real-time autonomous navigation in complex three-dimensional environments.

[0067] This embodiment provides a humanoid robot navigation control system for a three-dimensional complex environment, used to implement the aforementioned humanoid robot navigation control method for a three-dimensional complex environment. Therefore, the specific implementation of the humanoid robot navigation control system for a three-dimensional complex environment can be found in the previous embodiment section of the humanoid robot navigation control method for a three-dimensional complex environment. To avoid redundancy, it will not be repeated here.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A navigation and control method for a humanoid robot in a complex three-dimensional environment, characterized in that, Includes the following steps: S1: Construct a complex 3D environment scene for humanoid robot navigation. The scene includes static obstacles, undulating terrain and impassable areas to provide a navigation task operating environment for the humanoid robot. S2: Real-time acquisition of terrain information of the surrounding environment through the depth camera mounted on the humanoid robot to construct a 2.5D local height map centered on the robot; S3: Use a convolutional neural network to extract features from the 2.5D local elevation map to obtain the deep structural features of the local terrain; S4: Based on the current motion state of the humanoid robot and the orientation information of the navigation target, construct the query vector of the attention mechanism; based on the deep terrain structure features and grid spatial location information obtained in step S3, construct the key vector and value vector of the attention mechanism. S5: The position, orientation and terrain flatness information of the navigation target are fused to calculate the attention score of each grid. After normalization, an attention weight map and weighted terrain coding features are generated. S6: Model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing ontological state and weighted terrain coding features, and define a continuous action space containing forward stride, lateral stride and yaw angle changes. S7: Design a multi-objective comprehensive reward function that includes attention-guided rewards to enhance the humanoid robot's behavior in selecting safe landing points; S8: Construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs. S9: Based on the dual-delay deep deterministic policy gradient algorithm, the parameters of the policy network and the dual-value network are updated by combining the priority experience replay mechanism. S10: A two-stage progressive training strategy is adopted to train the network, and the trained strategy network is deployed to the humanoid robot platform to achieve real-time autonomous navigation in a complex three-dimensional environment.

2. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S2, the method for constructing a 2.5D local height map centered on the robot is as follows: The depth images captured by the depth camera are converted into 3D point clouds in a local coordinate system centered on the robot. The 3D point clouds are then projected onto a horizontal plane, and each grid cell stores the height value of that location relative to the robot's standing plane. For grid cells without valid depth data, neighboring grid cells are interpolated to fill the gap, resulting in a 2.5D local height map covering a preset range around the robot.

3. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S3, the convolutional neural network is a three-layer convolutional structure, specifically configured as follows: The first layer is a 16-channel 3×3 convolutional layer, the second layer is a 32-channel 3×3 convolutional layer, and the third layer is a 64-channel 3×3 convolutional layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to extract the structural features of the terrain's edges, gaps, and passable areas.

4. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S4, the method for constructing the query vector for the attention mechanism is as follows: Extract the orientation vector of the navigation target relative to the robot Normalized distance from robot to target Robot swinging leg logo Relative position of feet The 6-dimensional initial query vector is concatenated and then mapped through a fully connected layer to obtain the query features of the attention mechanism. .

5. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S5, the method for calculating the attention score of each grid is as follows: Calculate the position bias term based on the Gaussian kernel respectively. Orientation bias term based on target orientation consistency Landing point flatness score based on grid neighborhood height variance And obstacle penalty items for areas with height differences exceeding the threshold. ; The attention score for each grid is obtained by fusing the above items according to preset weights. The formula for calculating the attention score is: ； The attention scores are normalized using Softmax with a learnable temperature coefficient to obtain the attention weights of each grid, thus forming an attention weight map.

6. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S7, the multi-objective comprehensive reward function consists of five parts: task completion reward, distance progress reward, angle correction reward, attention guidance reward, and step penalty. The attention-guided reward is set based on the attention weight map generated in step S5. When the attention weight of the grid corresponding to the actual landing point selected by the robot is higher than the preset threshold, a positive reward is given; otherwise, a negative penalty is given. The formula for calculating the total reward is: ； In the formula, Rewards for completing the task. As a reward for progress, Rewards for angle correction. The reward is for attention-guided actions, and -0.01 is the penalty for each step taken.

7. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S8, the policy network concatenates the attention weight map with the target direction vector and inputs it into a three-layer fully connected network. The three-layer fully connected network consists of a 256-dimensional ReLU layer, a 128-dimensional ReLU layer, and a 4-dimensional Tanh layer. The output is scaled to obtain the navigation control action of the humanoid robot. The dual-value network employs two Q-networks with identical structures and independent initialization. Each Q-network first extracts environmental features through an attention-based terrain encoder, then concatenates the extracted features with the current action and inputs them into a three-layer fully connected network. The three-layer fully connected network consists of a 256-dimensional ReLU layer, a 128-dimensional ReLU layer, and a 1-dimensional linear output layer, which outputs the Q-value estimate of the state-action pair.

8. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S9, the method for updating the network parameters of the policy network and the dual-value network based on the dual-delay deep deterministic policy gradient algorithm combined with the priority experience replay mechanism is as follows: Experience tuples generated during robot navigation Stored to a priority experience replay buffer, the experience tuple containing the current state. ,action Instant rewards Next state End of round marker ; Each time a batch of experience is sampled from the buffer, two value networks are updated first. The update objective of the value networks is to minimize the mean square error between the predicted Q value and the target value. The formula for calculating the target value is: ； In the formula, For the target Q value, As a discount factor, To obtain the minimum value of the two Q-network outputs, For the first A target value network, with parameters as follows: , For the first The objective policy network is in state The action of outputting below, The target policy noise; The policy network employs a delayed update strategy, performing one policy network update after every two value network updates. The optimization objective of the policy network is to maximize the output of the first value network, i.e.: ； In the formula, Let the loss function be the policy network. For the experience replay buffer Mid-sampled state Seeking expectations, For the first value network, the parameters are: , For the current policy network in state The action to be output; Both the policy network and the target network corresponding to the value network use a soft update method to update parameters. The soft update formula is: ； ； In the formula, For the first Parameters of a target value network For the first The parameters of the current value network, For the parameters of the target policy network, These are the parameters of the current policy network. This is the soft update coefficient. This is an assignment operation, indicating that the target network parameters are updated gradually.

9. The navigation and control method for humanoid robots in complex three-dimensional environments according to claim 1, characterized in that, In step S10, the method for training the network using a two-stage progressive training strategy is as follows: The first stage is training on flat terrain without obstacles. A flat map with no height undulations and no obstacles is used. The training is conducted for a preset number of rounds so that the robot can master the basic navigation ability to walk towards the target. The second stage is training on complex 3D terrain. The system switches to a complex 3D environment map that includes static obstacles, undulating terrain, and impassable areas. Training continues until the network converges, enabling the robot to master the ability to select safe landing points and adapt to complex terrain.

10. A navigation and control system for a humanoid robot in a complex three-dimensional environment, characterized in that, The system is used to implement the humanoid robot navigation and control method for three-dimensional complex environments as described in any one of claims 1 to 9, specifically including: A 3D complex navigation scene construction module is used to build a 3D complex environment scene for humanoid robot navigation. The scene includes static obstacles, undulating terrain and impassable areas, providing a navigation task operation environment for humanoid robots. The 5D local height map construction module is used to collect terrain information of the surrounding environment in real time through the depth camera on the humanoid robot and construct a 2.5D local height map centered on the robot. The terrain deep structure feature extraction module is used to extract features from the 2.5D local height map using a convolutional neural network to obtain the deep structure features of the local terrain. The attention mechanism vector construction module is used to construct the query vector of the attention mechanism based on the current motion state of the humanoid robot and the orientation information of the navigation target; and to construct the key vector and value vector of the attention mechanism based on the deep terrain structure features and grid spatial location information obtained by the deep terrain structure feature extraction module. The adaptive attention terrain coding module is used to fuse the position, orientation and terrain flatness information of the navigation target, calculate the attention score of each grid, and generate an attention weight map and weighted terrain coding features after normalization. The navigation task Markov decision modeling module is used to model the three-dimensional environment navigation task of humanoid robots as a Markov decision process, construct a high-dimensional state space containing the ontological state and weighted terrain coding features, and define a continuous action space containing forward stride, lateral stride and yaw angle changes. The attention-guided multi-objective reward function design module is used to design a multi-objective comprehensive reward function that includes attention-guided rewards, in order to enhance the humanoid robot's behavior in selecting safe landing points; A dual-network architecture deep reinforcement learning network construction module is used to construct a policy network and a dual-value network based on deep reinforcement learning. The policy network is used to output navigation control actions, and the dual-value network is used to evaluate the value of state-action pairs. The dual-delay deep deterministic policy gradient update module is used to update the parameters of the policy network and the dual-value network based on the dual-delay deep deterministic policy gradient algorithm and combined with the priority experience replay mechanism. The two-stage progressive training and navigation deployment module is used to train the network using a two-stage progressive training strategy, and then deploy the trained strategy network to the humanoid robot platform to achieve real-time autonomous navigation in complex three-dimensional environments.

Images