A robot motion control method based on stable blind state and deep reinforcement learning fusion

Abstract

The application relates to a robot motion control method based on stable blind state and deep reinforcement learning fusion, and belongs to the technical field of robot control. An integrated architecture composed of a Soft Actor-Critic (SAC) reinforcement learning strategy and a model-driven stable step algorithm is constructed, so that the problem that a robot is difficult to stably climb obstacles such as stairs in an unstructured environment with a failed visual sensor is solved. In the method, the SAC algorithm is used in the upper layer to receive observation states containing the robot centroid position, foot end position and sensor force feedback, and dynamically output action instructions containing plane displacement and leg lifting height; the stable step algorithm is used in the lower layer to execute the action, a polygon center is solved by using a robot kinematics model to adjust the centroid, and the detection of unknown terrain and the guarantee of physical balance are realized in combination with the foot end force feedback. In simulation training, a carefully designed reward function is used to guide the strategy to complete the task in a small number of steps, so that the robot can adaptively optimize gait parameters according to the proprioceptive feedback, realize a 93% climbing success rate on a 10cm high stair and good generalization ability on variable height terrain, and effectively balance the stability of classical control and the flexibility of learning algorithm.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, specifically relating to a robot motion control method based on the fusion of ontology perception and deep reinforcement learning. Background Technology

[0002] With the rapid development of robotics technology, robots, due to their unique discrete ground-touching motion, have demonstrated enormous application potential in unstructured scenarios such as disaster relief, field exploration, and complex industrial inspection. Compared to traditional wheeled or tracked robots, robots can traverse obstacles, climb stairs, and move on uneven ground, giving them a significant advantage in replacing humans in performing dangerous tasks. However, stable motion control of robots in complex and changing environments remains a core technological challenge in this field.

[0003] Currently, robot navigation and obstacle avoidance heavily rely on visual sensors. While visual systems can provide rich environmental information, they are easily limited by environmental conditions in practical applications. In disaster sites filled with darkness, thick smoke, fog, or dust, as well as in poorly lit underground mines or caves, visual sensors often become unreliable or even completely fail due to degraded image quality. Furthermore, visual sensors face risks such as image blurring, occlusion, and physical damage caused by high-speed movement, making robot systems that overly rely on visual feedback lack sufficient robustness and survivability in extreme environments.

[0004] To address the challenge of visual degradation, "blind walking" control methods based on proprioceptive feedback have attracted widespread attention. This approach primarily relies on tactile feedback from joint and force sensors to perceive terrain features. Early research mainly focused on classical kinematic models, ensuring balance through static stability analysis and pre-defined gait patterns. While achieving high accuracy, this approach lacked adaptability to unknown and complex terrain. In recent years, deep reinforcement learning has made breakthroughs in robotics. Through large-scale simulation training, robots can exhibit extremely strong balance recovery capabilities. However, when dealing with structured obstacles requiring precise landing point planning, such as stairs, pure reinforcement learning methods often struggle to converge, and the trained strategies have limited generalization performance when facing obstacles of varying heights.

[0005] Therefore, the industry urgently needs a new control scheme that can provide robust stability using classical control models while leveraging the optimization and adaptive capabilities of deep reinforcement learning in dynamic environments. By combining the robustness of stable blind-state algorithms with the flexibility of learning algorithms, robots can accurately adjust gait parameters through physical contact feedback even without visual guidance, achieving efficient and autonomous movement in complex unstructured terrain. Summary of the Invention

[0006] The purpose of this invention is to provide a robot motion control method based on the fusion of stable blind state and deep reinforcement learning, to solve the problem that robots struggle to stably traverse obstacles such as stairs, steps, and thresholds using only visual information when visual sensors fail, lighting is insufficient, smoke or dust obscures the path, or environmental information is incomplete. This method combines a reinforcement learning strategy based on the SoftActor-Critic algorithm with a model-driven stable stepping algorithm, enabling the robot to adaptively adjust gait parameters and maintain static stability during walking by utilizing feedback from the center of mass position, foot position, and foot force sensors, without relying on visual input.

[0007] To achieve the above objectives, this invention provides a robot motion control method based on the fusion of stable blind state and deep reinforcement learning, comprising the following steps: S1. Establish an integrated control system, which includes an upper-layer reinforcement learning strategy and a lower-layer stable stepping algorithm; wherein, the upper-layer reinforcement learning strategy is based on the Soft Actor-Critic algorithm. A maximum entropy reinforcement learning framework is used to balance reward maximization and policy exploration, thereby improving the robustness of the policy in continuous control tasks. Its objective function is... for: in, The temperature coefficient is used to adjust the entropy term. The degree of impact on the policy. The observed states received by the SAC policy. This includes data on the robot's center of mass, foot position, and vertical reaction force at the foot after maximum normalization. The action instructions output by the strategy. Including the identifier of the target moving leg Planar displacement in the world coordinate system and the height of the leg lift .

[0008] S2. Construct a reinforcement learning model in a physical simulation environment. The reinforcement learning model includes a policy network for generating random action policies and a critic network for evaluating state values. Through interactive training between the policy network and the critic network, the robot gradually learns how to choose actions in obstacle terrain.

[0009] S3. Obtain the robot's observation state information, which includes the robot's center of mass position, the position of each foot in the world coordinate system, and the vertical reaction force data measured by the foot force sensors. Further, the foot force sensor data is processed using a maximum normalization method to scale the original force feedback data to a preset value range while preserving the relative magnitude relationships between the force feedbacks from each foot.

[0010] S4. The observed state information is input into the policy network, which outputs action commands. These action commands include the identifier of the target moving leg, the planar displacement of the target moving leg in the horizontal plane, and the lifting height of the target moving leg. Through these action commands, the upper-layer reinforcement learning policy can dynamically adjust the robot's next swing leg selection, landing position, and leg lifting amplitude under different terrain conditions.

[0011] S5. Execute the stable stepping algorithm, calculate the center of the supporting polygon formed by the supporting legs according to the action command, and adjust the robot's center of mass position according to the center of the supporting polygon to ensure that the robot meets the quasi-static stability condition at any moment in the motion cycle; then, generate the foot motion trajectory according to the target position of the swing leg, and use inverse kinematics to solve the joint angles. Furthermore, the motion trajectory of the swing leg is determined by a parabolic function. By discretizing the trajectory points and performing inverse kinematics calculations in real time, the foot motion remains smooth and the acceleration is limited; during the swing leg movement, the environmental contact state is continuously monitored by the foot force sensor. When the detected force exceeds a preset threshold, the swing leg movement is immediately stopped to achieve physical detection of obstacles or the ground.

[0012] S6. A comprehensive reward function is constructed based on the rewards for successful stair climbing, foot displacement, and fall penalties. The SAC algorithm is used to continuously optimize the policy network parameters until the robot achieves stable climbing on stair terrain without relying on visual input. Furthermore, the reward for successful stair climbing is inversely proportional to the total number of steps required to complete the task, to encourage the robot to use fewer steps to overcome stair obstacles. During the training phase, evaluation is conducted on stair terrain at different heights, enabling the policy network to generalize to unseen stair sizes within a certain range.

[0013] In a further preferred embodiment, in step S5, the stable stepping algorithm adopts a quasi-static gait paradigm to ensure that the robot is in a static equilibrium state at any moment of the motion cycle, and can remain stable even if the motion suddenly stops.

[0014] In a further preferred embodiment, in step S5, the motion trajectory of the swing leg is determined by a parabolic function. By discretizing the trajectory points and performing inverse kinematics calculations in real time, the movement of the foot is ensured to be smooth and the acceleration is limited.

[0015] In a further preferred embodiment, in step S6, the reward for successfully climbing the stairs is inversely proportional to the total number of steps required to complete the task, so as to encourage the robot to efficiently pass through the staircase obstacle with the fewest steps.

[0016] In a further preferred embodiment, the force sensor data in the observation state is processed using the maximum normalization method to scale the original values ​​to the range of [-1, 1] while maintaining the relative proportions between the values ​​of each sensor.

[0017] In a further preferred embodiment, during the training phase, the policy network learns to generalize to unseen stair sizes within a certain range by evaluating the staircase terrain at different heights.

[0018] Compared with the prior art, the present invention has the following beneficial effects: First, this invention combines an upper-layer deep reinforcement learning strategy with a lower-layer model-driven stable stepping algorithm, enabling the robot to achieve stable motion control by relying on proprioceptive information and foot force feedback even in environments where visual sensors fail or visual information is unreliable.

[0019] Secondly, the present invention uses a stable stepping algorithm to calculate the center of the supporting polygon and adjust the position of the centroid, so that the robot meets the quasi-static stability condition before moving the swing leg, thereby reducing the risk of the robot tipping over when climbing steps, stairs or crossing obstacles.

[0020] Third, the present invention uses a foot force sensor to determine the contact state between the swing leg and the terrain in real time. When the foot detection force exceeds a preset threshold, the swing leg movement is stopped immediately, so that the robot can complete the physical detection of the height of unknown obstacles or the ground contact position without vision.

[0021] Fourth, this invention optimizes the target moving leg, planar displacement, and leg lifting height by integrating a reward function-guided strategy network, enabling the robot to improve its efficiency in traversing stair obstacles while maintaining stability, and enhancing its adaptability to terrain with different height steps.

[0022] Fifth, this invention is applicable to robot motion control scenarios such as disaster relief, underground inspection, industrial plant inspection, warehousing and logistics, passage through complex building environments, and other scenarios where vision is limited or terrain is uncertain. Attached Figure Description

[0023] Figure 1 The flowchart below shows the robot motion control method based on the fusion of stable blind state and deep reinforcement learning proposed in this application. Figure 2 This is a schematic diagram of the control system of this application; Detailed Implementation Example 1 like Figures 1-2 As shown, this embodiment provides a robot motion control method based on the fusion of stable blind state and deep reinforcement learning. This method is applicable to tasks such as stair climbing, step crossing, and obstacle passage for quadruped robots under conditions of no visual feedback or unreliable visual feedback. The robot includes a body, leg drive mechanisms, joint actuators, foot force sensors, and a controller; the controller is used to receive the robot's body state and foot force feedback, and output corresponding joint control commands.

[0024] S1. Establish an integrated control system.

[0025] In this embodiment, the integrated control system includes an upper-layer reinforcement learning strategy and a lower-layer stabilizing stepping algorithm. The upper-layer reinforcement learning strategy employs the Soft Actor-Critic algorithm, which outputs action commands based on the robot's current observed state; the lower-layer stabilizing stepping algorithm executes these action commands based on the robot's kinematic model and maintains the robot's static stability during execution.

[0026] Specifically, the transformation relationships between the robot's body coordinate system, world coordinate system, and coordinates of each foot are first established, and a kinematic model of the robot's legs is then created. This kinematic model is used to inversely calculate the corresponding leg joint angles based on the target foot position, providing a computational basis for subsequent swing leg trajectory tracking and center of mass adjustment.

[0027] S2. Construct reinforcement learning models in a physical simulation environment.

[0028] In this embodiment, a physical simulation platform can be used to construct the robot training environment. The simulation environment includes stairs, steps, or other obstacle terrain, and the robot's body mass, joint limits, ground friction coefficient, foot contact parameters, and sensor sampling parameters are set. The reinforcement learning model includes a policy network and a critic network. The policy network generates action instructions based on the input state, while the critic network evaluates the value of the current state and the corresponding action.

[0029] During training, the robot repeatedly performs climbing or crossing tasks in a simulated environment. The policy network updates parameters based on the results of each task, enabling it to gradually learn to select appropriate moving legs, planar displacement, and leg lifting height under different terrain conditions.

[0030] S3. Obtain the robot's observation status information.

[0031] Within each decision cycle, the controller acquires the robot's center of mass position, the position of each foot in the world coordinate system, and the vertical reaction force data measured by the force sensors at each foot. To avoid the impact of data with different dimensions on the training of the policy network, the foot force sensor data is processed using a maximum normalization method. That is, the preset maximum contact force or the maximum contact force within the current sampling window is used as the normalization benchmark to scale the original force value to a preset range while preserving the relative proportions between the contact forces at each foot.

[0032] Based on the above observations, the policy network can determine whether the robot is currently close to an obstacle, which leg is more suitable for movement, whether the foot has already touched the ground, and whether the robot is currently in a stable posture.

[0033] S4. Input the observed state into the policy network and output action instructions.

[0034] After receiving the observation status, the policy network outputs a motion command that includes the target moving leg identifier, planar displacement, and leg lift height. The target moving leg identifier is used to determine the leg that will be the swing leg at the next moment; the planar displacement is used to determine the distance the swing leg moves in the horizontal plane relative to its current position; and the leg lift height is used to determine the highest point of the foot when the swing leg crosses a step, threshold, or other obstacle.

[0035] For example, when the robot's forelegs are close to the edge of a step and the foot force feedback indicates a change in height in front, the policy network can output a larger leg lift height and a shorter horizontal displacement to avoid the foot hitting the step surface; when the robot is on a flat surface, the policy network can output a lower leg lift height and a smoother forward displacement to improve motion efficiency.

[0036] S5. Execute the stable step algorithm.

[0037] Upon receiving the action command, the lower-level stabilizing stepping algorithm first determines the current set of supporting legs and calculates the supporting polygon based on the foot position of the supporting legs. Subsequently, the algorithm calculates the center of the supporting polygon and controls the robot's center of mass to move towards the center of the supporting polygon or its safe neighborhood, so that the robot maintains a quasi-static stable state before the swing leg is lifted.

[0038] Once the center of mass is adjusted, the swing leg moves along a preset parabolic trajectory. This parabolic trajectory has its endpoints at the starting and target foot positions of the swing leg, and its highest point is determined by the leg lift height output by the strategy network. After the trajectory is generated, it is discretized into multiple target foot points, and inverse kinematics is performed on each target point to obtain the corresponding joint angle control command.

[0039] During the swing leg's descent, the foot force sensor continuously detects the vertical reaction force. When the detected foot force exceeds a preset threshold, the controller determines that the swing leg has contacted the ground or obstacle surface and immediately stops the swing leg's downward movement, thus completing the physical detection of terrain height and contact position without visual conditions.

[0040] S6. Construct a comprehensive reward function and optimize the policy network parameters.

[0041] During the training phase, a comprehensive reward function was constructed based on successful stair climbing rewards, foot displacement rewards, and fall penalties. The successful stair climbing reward is inversely proportional to the total number of steps required for the robot to complete the task, encouraging the robot to complete the task with fewer steps. The foot displacement reward encourages the robot to make effective movements in the target direction. The fall penalty suppresses behaviors such as excessive tilting of the robot's posture, deviation of the center of gravity from the support range, or excessive lateral displacement.

[0042] Through the aforementioned reward function, the SAC algorithm continuously optimizes the policy network parameters, enabling the policy network to gradually learn to output reasonable actions under different stair heights, obstacle sizes, and contact feedback conditions. After training, the optimized policy network is used for actual robot control, allowing the robot to complete stable climbing on stair terrain without relying on visual input.

[0043] Example 2: Application in Disaster Relief Scenarios In another embodiment, this method can be applied to disaster relief robots. For example, in fire scenes, earthquake ruins, smoky environments, or underground spaces, visual sensors may be unable to provide reliable images due to smoke, insufficient lighting, or obstruction. In such cases, the robot can disable visual navigation or reduce the weight of visual information, utilizing only the center of mass position, foot position, and foot force sensor data to execute this method.

[0044] In this scenario, the policy network determines the presence of steps, rubble, or collapsed edges ahead based on foot force feedback. The lower-level stabilizing stepping algorithm adjusts the robot's center of mass before each swing leg movement, ensuring it lies within the safe area of ​​the supporting polygon. When the swing leg touches rubble, a stair edge, or other obstacles, the foot force sensor detects that the contact force exceeds a threshold. The controller immediately stops the swing leg from pressing down and uses that contact point as the new stable support point. This allows the robot to reduce the risk of tipping over and improve its obstacle-crossing ability in visually limited rescue environments.

[0045] Example 3: Application in Industrial Inspection Scenarios In another embodiment, this method can be applied to industrial inspection scenarios such as factories, warehouses, pipe corridors, underground mines, or substations. These scenarios often have thresholds, cables, steps, protruding pipes, and uneven ground, and some areas suffer from insufficient lighting, dust obstruction, or visual glare interference.

[0046] When performing inspection tasks, the robot can dynamically adjust its gait based on foot contact feedback using this method. For lower thresholds or pipeline protrusions, the policy network can output a smaller leg lift height and a shorter planar displacement; for higher steps or stairs, the policy network can output a larger leg lift height, and a stable stepping algorithm controls the centroid to move to the vicinity of the support polygon center before performing the swing leg movement. This embodiment can improve the stability and passage efficiency of the inspection robot in complex industrial terrain.

[0047] Example 4: Training and generalization application under staircase terrain of different heights In another embodiment, to improve the policy network's adaptability to unknown stair dimensions, stair terrain with different heights, widths, and friction coefficients can be set during the training phase. For example, low-height, medium-height, and high-height steps can be set respectively, and the step height, ground friction coefficient, robot initial posture, and sensor noise can be randomly changed during training.

[0048] Through the training method described above, the policy network can learn the correspondence between changes in foot force feedback and terrain height, and maintain a certain generalization ability on unseen step sizes. In actual deployment, when the robot encounters steps or obstacles that are not entirely consistent with the training environment, it can still correct the landing point of the swing leg through foot contact feedback, and maintain the body balance through a stable stepping algorithm, thereby improving the applicability of this method in real-world environments.

Claims

1. A robot motion control method based on the fusion of stable blind state and deep reinforcement learning, characterized in that, Includes the following steps: S1. Establish an integrated control system, which includes an upper-layer reinforcement learning strategy and a lower-layer stable stepping algorithm; wherein, the upper-layer reinforcement learning strategy is based on the Soft Actor-Critic algorithm. A maximum entropy reinforcement learning framework is used to balance reward maximization and policy exploration, thereby improving the robustness of the policy in continuous control tasks; its objective function is... for: in, The temperature coefficient is used to adjust the entropy term. The degree of impact on the policy; the observed states received by the SAC policy. This includes data on the robot's center of mass, foot position, and vertical reaction force at the foot after maximum normalization. Action instructions output by the strategy Including the identifier of the target moving leg Planar displacement in the world coordinate system and the height of the leg lift ; S2. Construct a reinforcement learning model in a physical simulation environment. The reinforcement learning model includes a policy network for generating random action policies and a critic network for evaluating state values. Through interactive training between the policy network and the critic network, the robot gradually learns how to choose actions in obstacle terrain. S3. Obtain the robot's observation state information, which includes the robot's center of mass position, the position of each foot in the world coordinate system, and the vertical reaction force data measured by the foot force sensor; further, the foot force sensor data is processed using the maximum normalization method to scale the original force feedback data to a preset value range while retaining the relative magnitude relationship between each foot force feedback. S4. Input the observation state information into the policy network, and the policy network outputs action instructions. The action instructions include the identifier of the target moving leg, the planar displacement of the target moving leg in the horizontal plane, and the lifting height of the target moving leg. Through the above action instructions, the upper-layer reinforcement learning policy can dynamically adjust the robot's next swing leg selection, landing position, and lifting amplitude under different terrain conditions. S5. Execute the stable stepping algorithm, calculate the center of the supporting polygon formed by the supporting legs according to the action command, and adjust the position of the robot's center of mass according to the center of the supporting polygon to ensure that the robot meets the quasi-static stability condition at any time in the motion cycle; then, generate the foot motion trajectory according to the target position of the swing leg, and use inverse kinematics to solve the joint angle; further, the motion trajectory of the swing leg is determined by a parabolic function, and by discretizing the trajectory points and performing inverse kinematics calculation in real time, the foot motion is kept smooth and the acceleration is limited; during the swing leg movement, the environmental contact state is continuously monitored by the foot force sensor, and when the detected force exceeds the preset threshold, the swing leg movement is stopped immediately to achieve physical detection of obstacles or the ground; S6. A comprehensive reward function is constructed based on the reward for successful stair climbing, the reward for foot displacement, and the penalty for falling. The SAC algorithm is used to continuously optimize the policy network parameters until the robot can achieve stable climbing on the stair terrain without relying on visual input. Furthermore, the reward for successful stair climbing is inversely proportional to the total number of steps required to complete the task, so as to encourage the robot to use fewer steps to pass through the stair obstacle. During the training phase, by evaluating on stair terrain at different heights, the policy network can generalize to unseen stair sizes within a certain range.

2. The method as described in claim 1, characterized in that: In step S5, the stable stepping algorithm adopts a quasi-static gait paradigm to ensure that the robot is in a static equilibrium state at any moment of the motion cycle, and can remain stable even if the motion suddenly stops.

3. The method as described in claim 1, characterized in that: In step S5, the trajectory of the swing leg is determined by a parabolic function. By discretizing the trajectory points and performing inverse kinematics calculations in real time, the movement of the foot is ensured to be smooth and the acceleration is limited.

4. The method as described in claim 1, characterized in that: In step S6, the reward for successfully climbing the stairs is inversely proportional to the total number of steps required to complete the task, in order to encourage the robot to efficiently pass through the staircase obstacle with the fewest steps possible.

5. The method as described in claim 1, characterized in that: The force sensor data in the observation state is processed using the maximum normalization method, which scales the original values ​​to the range of [-1, 1] while maintaining the relative proportions between the values ​​of each sensor.

6. The method as described in claim 1, characterized in that: During the training phase, the policy network learns to generalize to unseen stair sizes within a certain range by evaluating it on stair terrain at different heights.

Images