A complex terrain adaptive motion control method and system for a two-wheeled foot robot

Abstract

The present invention discloses a complex terrain adaptive motion control method and system for a two-wheeled legged robot, and relates to the field of robot motion control technology. The method comprises the steps of: constructing a complex terrain model including a robot model, obstacles, and environmental constraints; constructing a motion decision controller, and using a reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models, wherein the asymmetric training strategy introduces risk constraints during updating to constrain the behavior of the strategy; performing policy gradient backpropagation based on the training results, and optimizing the asymmetric training strategy using an adaptive learning rate adjustment method based on performance feedback. The present invention solves the problem of efficient, robust, and adaptive motion control of a two-wheeled legged robot in a variety of complex terrain environments by designing an asymmetric training strategy and a segmented training mechanism in the reinforcement learning process.

Description

Technical Field

[0001] The present invention relates to the technical field of robot motion control, and in particular to a complex terrain adaptive motion control method and system for a two-wheeled foot robot. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As the application scenarios of mobile robots continue to expand, achieving efficient, stable, and robust motion control on unknown or complex terrain has become a research hotspot in academia and industry. Existing wheeled robots, legged robots, and wheel-legged robots often use classic traditional model predictive control (MPC), sliding mode control, or empirical trajectory planning methods. These methods achieve good performance in structured, flat environments, but often struggle to balance real-time performance and stability in unstructured, dynamically changing environments, or environments with diverse terrain features (such as loose gravel, gullies, steps, and slopes), resulting in control failures and excessive energy consumption.

[0004] In recent years, reinforcement learning has garnered widespread attention for its adaptive decision-making capabilities in high-dimensional, nonlinear systems. Reinforcement learning-based motion control methods, through the collaborative training of a policy network and an evaluation network, can learn end-to-end control strategies in simulated environments. However, when faced with diverse and complex terrains, existing reinforcement learning methods fail to fully consider the deep fusion of multi-source sensor data and risk constraints, resulting in insufficient robustness when transferring learned strategies to real-world environments. Furthermore, their training mechanisms are unable to adapt to unstructured, complex scenarios. Therefore, how to use reinforcement learning to achieve efficient adaptive control of two-wheeled, legged robots in complex terrains has become a pressing technical challenge. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a complex terrain adaptive motion control method and system for a two-wheeled legged robot. By designing an asymmetric training strategy and a segmented training mechanism in the reinforcement learning process, the problem of efficient, robust and adaptive motion control of the two-wheeled legged robot in a variety of complex terrain environments is solved.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A first aspect of the present invention provides a complex terrain adaptive motion control method for a two-wheeled legged robot, comprising the following steps:

[0008] Construct a robot model based on the posture parameters and motion parameters of the two-wheeled foot robot;

[0009] Build complex terrain models that include robot models, obstacles, and environmental constraints;

[0010] A motion decision controller is constructed and trained using a reinforcement learning model using an asymmetric training strategy for multi-objective collaborative optimization of the motion decision controller for different complex terrain models. The asymmetric training strategy introduces risk constraints during the update process to constrain the strategy's behavior.

[0011] Policy gradient backpropagation is performed based on the training results, and an adaptive learning rate adjustment method based on performance feedback is used to optimize the asymmetric training strategy, thereby achieving adaptive motion control of the two-wheeled legged robot.

[0012] Furthermore, the motion decision controller includes a multi-source sensor data fusion module and an action decision network module. The multi-source sensor data fusion module fuses the data collected from the two-wheeled leg robot with the environmental terrain information, generates a feature vector and transmits it to the action decision network module, performs real-time prediction of the terrain environment, and generates corresponding action decisions.

[0013] Furthermore, the reinforcement learning model includes a policy network and an evaluation network. The policy network is used to output the robot's actions based on the input of the two-wheeled legged robot's current state observation. The evaluation network is responsible for evaluating the value of the robot's actions and states, and outputting the state-action value function.

[0014] Furthermore, an asymmetric training strategy is adopted, and multiple two-wheeled and legged robots are used for training in a parallel learning and collaborative decision-making manner.

[0015] Furthermore, the complete reward function of reinforcement learning is:

[0016] ,

[0017] in, is the main reward function, is the velocity tracking reward function, is a highly stable reward function, is the touchdown penalty function, is the motion stability reward function, is the energy efficiency reward function, is the reward function for special terrain, A weight parameter specialized for structured step terrain.

[0018] Furthermore, the asymmetric training strategy uses samples to estimate the policy gradient during the update process, uses the clipping objective function to update the policy, and introduces the KL divergence to control the amplitude of the policy update.

[0019] Furthermore, multi-objective collaborative optimization obtains the globally optimal decision information by setting multiple optimization objectives and performing weighted combination with different weights. The global loss function expression based on multi-objective collaborative optimization design is as follows:

[0020] ;

[0021] in, is the global loss function, is the main loss of the strategy, is the risk cost penalty term, is the policy value function loss, is the risk value loss, is the entropy regularization term.

[0022] A second aspect of the present invention provides a complex terrain adaptive motion control system for a two-wheeled legged robot, comprising:

[0023] a parameter acquisition module configured to construct a robot model according to the posture parameters and motion parameters of the two-wheeled foot robot;

[0024] a model building module configured to build a complex terrain model including a robot model, obstacles, and environmental constraints;

[0025] A reinforcement learning training module is configured to build a motion decision controller. It uses a reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models. The asymmetric training strategy introduces risk constraints during updates to constrain the strategy's behavior.

[0026] The backpropagation optimization module is configured to perform policy gradient backpropagation based on the training results, and uses an adaptive learning rate adjustment method based on performance feedback to optimize the asymmetric training strategy, thereby realizing adaptive motion control of the two-wheeled leg robot.

[0027] The third aspect of the present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps in the complex terrain adaptive motion control method of a two-wheeled legged robot as described in the first aspect of the present invention.

[0028] A fourth aspect of the present invention provides a computer device, comprising:

[0029] a processor adapted to execute a computer program;

[0030] A computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for adaptive motion control of a two-wheeled legged robot on complex terrain as described in the first aspect of the present invention is implemented.

[0031] One or more of the above technical solutions have the following beneficial effects:

[0032] This invention discloses a complex-terrain adaptive motion control method and system for a two-wheeled legged robot. By constructing a complex terrain model and a motion decision controller, and using a reinforcement learning model for multi-objective collaborative optimization training, this method achieves efficient and stable motion control for the two-wheeled legged robot on complex terrain, demonstrating significant technical benefits. This method addresses the problem of efficient, robust, and adaptive motion control for two-wheeled legged robots in various complex terrain environments.

[0033] The motion decision controller of this invention further enhances the robot's motion control capabilities. The multi-source sensor data fusion module effectively integrates various data collected by the two-wheeled robot with information about the surrounding terrain, generating feature vectors that are then fed into the motion decision network module. This process enables real-time prediction of the terrain environment and generates corresponding motion decisions, enabling the robot to quickly and appropriately adjust its motion based on real-time environmental changes, enhancing its dynamic response and mobility in complex terrain.

[0034] The reinforcement learning-based control approach of this invention enables the robot to find the optimal motion strategy in complex terrain through continuous learning and optimization, thereby improving movement efficiency and stability. Its asymmetric training strategy, which constrains policy behavior by introducing risk constraints and employs parallel learning and collaborative decision-making among multiple two-wheeled robots, further enhances the effectiveness and efficiency of training. During training, the policy gradient is estimated using samples, the policy is updated using a clipped objective function, and the KL divergence is introduced to control the magnitude of policy updates. These technical measures effectively ensure the stability and convergence of the training process, enabling the robot to quickly adapt and make optimal decisions in complex and changing terrain environments.

[0035] The multi-objective collaborative optimization training of the present invention realizes the generation of globally optimal decision information, and by setting different weights for weighted combination, the robot can comprehensively consider various factors in complex terrain and find the optimal motion path and action strategy, thereby realizing efficient and stable motion control of the two-wheeled leg robot on complex terrain, providing reliable technical support for the application of the two-wheeled leg robot in complex environments, and has broad application prospects and important practical significance.

[0036] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flow chart of the complex terrain adaptive motion control method of the two-wheeled leg robot in Example 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of the two-wheeled leg robot continuously crossing a structured step terrain in Example 1 of the present invention;

[0040] Figure 3 This is a schematic diagram of a two-wheeled legged robot crossing a ravine terrain in Example 1 of the present invention;

[0041] Figure 4 This is a diagram showing the process of the two-wheeled leg robot continuously crossing a structured step terrain in Example 1 of the present invention;

[0042] Figure 5 This is a diagram of the process of the two-wheeled and legged robot continuously descending stairs in Example 1 of the present invention;

[0043] Figure 6 This is a diagram of the process of the two-wheeled leg robot crossing the ravine ground in Example 1 of the present invention;

[0044] Figure 7 This is a diagram of the movement process of the two-wheeled leg robot on rough terrain in Example 1 of the present invention. DETAILED DESCRIPTION

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations;

[0047] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] Example 1:

[0049] The first embodiment of the present invention provides a complex terrain adaptive motion control method for a two-wheeled foot robot, such as Figure 1 As shown, the following steps are included:

[0050] Step 1: Construct a robot model based on the posture parameters and motion parameters of the two-wheeled foot robot.

[0051] In this embodiment, a force sensor and an inertial measurement unit are provided on the two-wheeled leg robot to collect real-time posture parameters and motion parameters.

[0052] Step 2: Build a complex terrain model that includes the robot model, obstacles, and environmental constraints.

[0053] In this embodiment, obstacles are those caused by complex terrain, such as structured steps, slopes, wavy terrain, rugged terrain, gullies, and rough gravel terrain. Environmental constraints refer to the constraints imposed by the robot's interaction with the environment during motion. Examples include friction, contact, and collisions with the ground and obstacles, as well as pushes, kicks, and sudden weight applications to the robot.

[0054] Step 3: Construct a motion decision controller and use the reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models.

[0055] Step 3.1: Build a reinforcement learning algorithm model for the control of the two-wheeled and legged robot.

[0056] In this embodiment, the reinforcement learning model is a neural network model, consisting of a policy network and an evaluation network. The policy network outputs the robot's actions based on the input of the robot's current state observations. The evaluation network is responsible for evaluating the value of the robot's actions and states and outputting a state-action-value function. When building a reinforcement learning model, the state space, action space, decision strategy, and reward function in the reinforcement learning environment are set.

[0057] Step 3.2: Build the motion decision controller.

[0058] In this embodiment, the motion decision controller includes a multi-source sensor data fusion module and an action decision network module, wherein the multi-source sensor data fusion module fuses the data collected by the force sensor and inertial measurement unit on the two-wheeled leg robot with the environmental terrain information, generates a feature vector and transmits it to the action decision network module, and makes a real-time prediction of the terrain environment. Specifically, it speculates in real time the terrain environment that may be encountered, such as flat ground, rugged ground, gully terrain and structured step terrain, pre-trains special strategies for different terrains, dynamically adjusts the control parameters and reward weight distribution, and uses multi-dimensional course learning to perform dynamic training and parallel training to accelerate and improve training efficiency, and generates corresponding action decisions, thereby controlling the movement of the two-wheeled leg robot, such as Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 shown.

[0059] Step 3.3: Use the reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models.

[0060] Step 3.3.1: Design an asymmetric training strategy based on reinforcement learning.

[0061] In this embodiment, an asymmetric training strategy is used for training. Asymmetric training refers to inputting different observations into the policy network and evaluation network for training. In this embodiment, considering the acquisition of observations during the subsequent training strategy deployment process, the policy network only uses observation data that can be easily and accurately obtained by the real robot, while the evaluation network uses additional external environment data and observation data that cannot be obtained or is relatively difficult to obtain by the real robot. The above training method does not increase the computational complexity of the learning process, but also ensures the accuracy of the reinforcement learning process. It also improves the robustness of the strategy and facilitates the subsequent sim-to-sim and sim-to-real deployment processes.

[0062] In a specific embodiment, an asymmetric training strategy is adopted, different observations are input into the policy network and the evaluation network respectively, and then multiple two-wheeled robots are trained in a parallel learning and collaborative decision-making manner to establish a reinforcement learning reward function suitable for the two-wheeled robots. Considering that the two-wheeled robots use the data that they can accurately obtain during subsequent deployment without having to use the external environment data that cannot be effectively obtained and some difficult-to-obtain self-data, in order to ensure that the robots can fully utilize this information and smoothly learn a stable and efficient gait, the reinforcement learning reward function expression is designed as follows:

[0063] .

[0064] in, is the main reward function, is the velocity tracking reward function, including linear velocity and angular velocity tracking. is a highly stable reward function, is the touchdown penalty function, is the motion stability reward function, It is an energy efficiency reward function that specifically restricts joint position, velocity, and acceleration. It is a reward function for special terrains, such as steps, gullies, and other vertical terrains. A weight parameter specialized for structured step terrain.

[0065] More specifically:

[0066] The robot is encouraged to track the predetermined speed command and give full play to the maneuverability of the two-wheeled robot. The speed tracking reward function is as follows:

[0067] .

[0068] in, is the expected linear velocity, is the current linear velocity of the robot, is the expected angular velocity, is the current angular velocity of the robot, and is the scaling factor, and is the weight coefficient.

[0069] The two-wheeled legged robot is encouraged to maintain a predetermined height during movement to avoid unstable movement caused by the body height being too high or too low. The height stability reward function is as follows:

[0070] .

[0071] in, is the height from the foot end to the fuselage in the z direction, For the predetermined height, is the weight coefficient.

[0072] Penalize the base and calf of the two-wheeled robot touching the ground to prevent the robot from falling or other unstable behaviors or cheating on stability by touching the ground with the calf, and ensure the robot's most basic balance posture. The ground penalty function is as follows:

[0073] .

[0074] in, is the external contact force on the robot base, is the external contact force on the lower leg of the two-wheeled robot, is the weight coefficient.

[0075] Penalize the unstable behavior of the robot body to ensure that the robot can move on flat ground and remain stable when crossing structured step terrain or gully terrain, and penalize excessive joint angular velocity to ensure stability and smoothness during robot movement. The motion stability reward function is as follows:

[0076] .

[0077] in, is the projection of the gravity vector in the robot coordinate system, is the set gravity constant, is the acceleration calculated from the current and previous moment velocities, and is the weight coefficient.

[0078] Penalize the use of robot joint torque, encourage the robot to make full use of torque, and reduce energy loss. The energy efficiency reward function is as follows:

[0079] .

[0080] in, For the The torque of each joint, is the scaling factor, is the weight coefficient.

[0081] Since the two-legged robot has wheel joints at the ends of its feet and two legs, it is a major challenge to balance and stably cross structured step terrain and gully terrain. Therefore, a reward function is specialized for special terrain to encourage the robot to actively lift its legs and stably cross the terrain when encountering structured step terrain or gully terrain. The reward function for special terrain is as follows:

[0082] .

[0083] in, The contact condition for whether the wheels of the two-wheeled robot touch the step terrain, Indicates the wheel number, is the contact force on the wheels of the two-wheeled robot except the vertical direction, is the vertical contact force on the wheel, is the upward speed of the robot, is the upward acceleration of the robot, is the change in the distance from the wheel end to the base of the two-wheeled foot robot between the previous moment and the current moment, is the constraint value of the distance change, 、 and is the weight coefficient.

[0084] Step 3.3.2: The policy network generates actions based on observations, interacts with the environment to generate the first set of trajectories, and calculates rewards and advantages. Based on the actions generated by the policy network, the robot state is updated, the next state is calculated, and new observations are calculated as input to the policy for the next time step. The reward is calculated based on the current state.

[0085] Specifically, the evaluation network is responsible for evaluating the value of the current action and state, that is, the expected value of the cumulative reward that can be obtained in the future under the current state. The input includes the robot's own state observation information, the body linear speed, and the perceived external terrain information, and the output state value function , used to evaluate the value of the current action and state, that is, the expected value of the cumulative reward that can be obtained in the future under the current state. Set the main advantage function , which is used to measure the quality of the current action relative to the average strategy and update the strategy network based on feedback information. The main advantage function expression is as follows:

[0086] .

[0087] in, is the action value function, which means that in state Take action After the cumulative reward, Is the state value function, which means that in state The expected value of the cumulative reward under .

[0088] During actual calculation, the advantage function expression for each time step is as follows:

[0089] .

[0090] in, is the advantage function at time step t, is the action-value function at time step t, is the state value function at time step t.

[0091] The policy network generates actions based on the current state observations of the two-wheeled legged robot. The goal is to optimize the policy so that the generated actions can maximize the long-term cumulative reward. The expression of the policy objective is as follows:

[0092] .

[0093] in, For strategic goals, represents the expectation (average), To evaluate the advantage function of network computing, is the parameterized strategy of the policy network, which means that in the state Next select action probability.

[0094] The optimization goal of the evaluation network is to minimize the prediction error of the state value. The value function loss expression is as follows:

[0095] .

[0096] in, is the value function loss, represents the expectation (average) of the quantity at time step t, Cumulative reward for the goal, calculated by the reward and discount factor.

[0097] Step 3.3.3: In order to limit the amplitude of policy updates, the asymmetric training strategy uses samples to estimate the policy gradient during the update process, uses the clipping objective function to update the policy, and introduces the KL divergence to control the amplitude of the policy update, and in order to uniformly calculate the loss, that is, minimize the loss.

[0098] The strategy loss function expression is defined as follows:

[0099] .

[0100] in, is the policy loss function, It represents the expectation (average) of the data collected at time step t, is the strategy ratio, that is, the degree of deviation on the existing samples after the strategy is updated, For the "new" strategy network in Select action in status The probability of is the probability of the “old” policy network at the same state-action pair, are the parameters of the policy network, is the advantage function at time step t, is the cropping range, is the weight coefficient of KL regularization, which is dynamically adjusted according to the average KL divergence calculated in real time and the size of the target threshold. Measure the difference in policy distribution before and after the update, clip Indicates a cropping operation, i.e. Constrained to and between.

[0101] Step 3.3.4: The asymmetric training strategy introduces risk constraints during update to constrain the behavior of the strategy.

[0102] Specifically, when updating the strategy, risk constraints are introduced. By introducing risk-related value functions and loss functions, the behavior of the strategy is constrained, so that the strategy can meet the risk constraint threshold conditions while optimizing the reward, avoiding dangerous behaviors, excessive motor torque and energy consumption, and unstable actions. and risk cost penalty Introduced into the strategy optimization objective, the function expression is as follows:

[0103] .

[0104] in, is the time difference error of the risk constraint at time step t, is the risk value function at time step t, is the time difference error of the risk constraint at time step t+1, is the instantaneous risk value at time step t, is the risk value function at time step t+1, is the discount factor, is the GAE parameter, Represents the ending time step index of a complete sampling trajectory, The offset index.

[0105] Following the previous clipping idea, we get the risk proxy loss function, which is expressed as follows:

[0106] .

[0107] in, represents the proxy loss function.

[0108] For each time step, set a risk cost , which is used to reflect the degree to which the immediate risk exceeds the threshold. The risk cost penalty term expression is defined as follows:

[0109] .

[0110] in, is the penalty coefficient.

[0111] Similarly, we get the risk value loss:

[0112] .

[0113] in, is the risk value loss function.

[0114] Step 3.3.5: Multi-objective collaborative optimization sets multiple optimization objectives and sets different weights for weighted combination to obtain the global optimal decision information. The global loss function expression based on multi-objective collaborative optimization design is as follows:

[0115] In a specific implementation, multi-objective collaborative optimization expands the traditional single reward signal into a multimodal optimization objective, including policy task updates, evaluation network value optimization, and security and stability constraints related to task risks. A composite optimization objective is constructed and weighted combinations are performed using different weights to obtain the global optimal decision information. The global loss function expression is designed as follows:

[0116] .

[0117] in, is the global loss function, is the main loss of the strategy, is the risk cost penalty term, is the policy value function loss, is the loss of risk value, is the entropy regularization term.

[0118] Step 3.3.6: The training process dynamically and gradually increases the learning difficulty.

[0119] This embodiment uses multi-dimensional curriculum learning during the iterative training process, and dynamically adjusts the robot's motion instructions, observation noise, terrain difficulty, and external disturbances such as motor errors, random ground friction, and random external forces during the training process. In the initial stages of training, simple motion instructions, observation noise, and simple terrain are given. As the number of training sessions increases, the difficulty of the motion instructions and the complexity of the terrain are gradually increased, and the range of observation noise and the range of external disturbances are increased. At the same time, each time the environment is reset during training, the command range and the terrain difficulty of the robot are dynamically adjusted based on the robot's completion of the motion instructions and its movement performance under the terrain of the previous interaction process.

[0120] Step 4: Perform policy gradient backpropagation based on the training results, and use the adaptive learning rate adjustment method based on performance feedback to optimize the asymmetric training strategy, thereby realizing adaptive motion control of the two-wheeled leg robot.

[0121] In a specific embodiment, the policy network samples actions according to the current state observation of the two-wheeled foot robot. , environment execution Return to new state , instant rewards and immediate risk value ,Will The experience buffer is stored and the target comprehensive return and comprehensive advantage estimate are calculated. The target comprehensive return includes the main reward function and the risk value function, and the comprehensive advantage estimate includes the main advantage function and the risk advantage function.

[0122] Calculate the individual losses and the global loss, perform policy gradient backpropagation, and update the parameters based on the learning rate to optimize the policy. Specifically, based on the calculated global loss, apply the chain rule from the output layer back to the input layer to obtain the gradient of each parameter. Then, update the parameters along the negative gradient direction to reduce the loss and achieve the goal of policy optimization.

[0123] However, traditional policy gradient algorithms use a fixed learning rate, and the step size along the gradient direction for each parameter update is a fixed value. In complex action situations, it is easy for the learning rate to be too large, exceeding the "optimal point" and causing oscillation or divergence, or too small a learning rate and slow convergence, resulting in low training efficiency. Therefore, an adaptive learning rate adjustment method based on performance feedback is used to dynamically adjust the learning rate through the KL divergence to balance the exploration phase in the early stage of training and the convergence phase in the later stage of training. The expression for dynamically adjusting the learning rate is as follows:

[0124] .

[0125] in, represents the learning rate before the kth round of update, k is the training round, is the average KL value calculated at the end of each epoch, is the scaling factor, and is the KL threshold.

[0126] Example 2:

[0127] A second embodiment of the present invention provides a complex terrain adaptive motion control system for a two-wheeled legged robot, comprising:

[0128] a parameter acquisition module configured to construct a robot model according to the posture parameters and motion parameters of the two-wheeled foot robot;

[0129] a model building module configured to build a complex terrain model including a robot model, obstacles, and environmental constraints;

[0130] A reinforcement learning training module is configured to build a motion decision controller. It uses a reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models. The asymmetric training strategy introduces risk constraints during updates to constrain the strategy's behavior.

[0131] The backpropagation optimization module is configured to perform policy gradient backpropagation based on the training results, and uses an adaptive learning rate adjustment method based on performance feedback to optimize the asymmetric training strategy, thereby realizing adaptive motion control of the two-wheeled leg robot.

[0132] Example 3:

[0133] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program, which is suitable for being loaded by a processor and executing the steps in the complex terrain adaptive motion control method of a two-wheeled legged robot as described in embodiment 1 of the present invention.

[0134] Example 4:

[0135] A fourth embodiment of the present invention provides a computer device, comprising:

[0136] a processor adapted to execute a computer program;

[0137] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps in the complex terrain adaptive motion control method of the two-wheeled leg robot as described in the first embodiment of the present invention are implemented.

[0138] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data processing device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0140] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technical object of a person skilled in the art that can be easily conceived of within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A complex terrain adaptive motion control method for a two-wheeled foot robot, characterized in that: The following steps are involved: Construct a robot model based on the posture parameters and motion parameters of the two-wheeled foot robot; Build complex terrain models that include robot models, obstacles, and environmental constraints; A motion decision controller is constructed and trained using a reinforcement learning model using an asymmetric training strategy for multi-objective collaborative optimization of the motion decision controller for different complex terrain models. The asymmetric training strategy introduces risk constraints during the update process to constrain the strategy's behavior. The complete reward function for reinforcement learning is: ， in, is the main reward function, is the velocity tracking reward function, is a highly stable reward function, is the touchdown penalty function, is the motion stability reward function, is the energy efficiency reward function, is the reward function for special terrain, is a weight parameter specialized for structured step terrain; Multi-objective collaborative optimization obtains the global optimal decision information by setting multiple optimization objectives and setting different weights for weighted combination. The global loss function expression based on multi-objective collaborative optimization design is as follows: ； in, is the global loss function, is the main loss of the strategy, is the risk cost penalty term, is the policy value function loss, is the risk value loss, is the entropy regularization term; The highly stable reward function is as follows: ， in, is the height from the foot end to the fuselage in the z direction, For the predetermined height, is the weight coefficient; The touchdown penalty function is as follows: ， in, is the external contact force on the robot base, is the external contact force on the lower leg of the two-wheeled foot robot, is the weight coefficient; The reward function for special terrain is as follows: ， in, The contact condition for whether the wheels of the two-wheeled robot touch the step terrain, Indicates the wheel number, is the contact force on the wheels of the two-wheeled robot except the vertical direction, is the vertical contact force on the wheel, is the upward speed of the robot, is the upward acceleration of the robot, is the change in the distance from the wheel end to the base of the two-wheeled foot robot between the previous moment and the current moment, is the constraint value of the distance change, 、 and is the weight coefficient; The risk cost penalty term is expressed as follows: ， in, is the penalty coefficient; set a risk cost at each time step ; Similarly, we get the risk value loss: ， in, is the risk value loss function; is the risk value function at time step t; Based on the training results, policy gradient backpropagation is performed, and an adaptive learning rate adjustment method based on performance feedback is used to optimize the asymmetric training strategy, thereby achieving adaptive motion control of the two-wheeled legged robot. During the update process, the asymmetric training strategy uses samples to estimate the policy gradient, uses the clipping objective function to update the policy, and introduces the KL divergence to control the amplitude of the policy update; The learning rate is dynamically adjusted by KL divergence to balance the exploration phase in the early stage of training and the convergence phase in the late stage of training. The expression for dynamically adjusting the learning rate is as follows: ， in, represents the learning rate before the kth round of update, k is the training round, is the average KL value calculated at the end of each epoch, Measure the difference in policy distribution before and after the update, It represents the expectation of the data collected at time step t, is the scaling factor, and is the KL threshold.

2. The complex terrain adaptive motion control method of a two-wheeled legged robot according to claim 1, characterized in that: The motion decision controller includes a multi-source sensor data fusion module and an action decision network module. The multi-source sensor data fusion module fuses the data collected from the two-wheeled leg robot with the environmental terrain information, generates a feature vector and transmits it to the action decision network module, performs real-time prediction of the terrain environment, and generates corresponding action decisions.

3. The complex terrain adaptive motion control method of a two-wheeled legged robot according to claim 1, characterized in that: The reinforcement learning model includes a policy network and an evaluation network. The policy network is used to output the robot's actions based on the input of the two-wheeled legged robot's current state observation. The evaluation network is responsible for evaluating the value of the robot's actions and states, and outputting the state-action-value function.

4. The complex terrain adaptive motion control method of a two-wheeled legged robot according to claim 1, characterized in that: An asymmetric training strategy is adopted, and multiple two-wheeled and legged robots are used for training in a parallel learning and collaborative decision-making manner.

5. A complex terrain adaptive motion control system for a two-wheeled foot robot, characterized in that: include: a parameter acquisition module configured to construct a robot model according to the posture parameters and motion parameters of the two-wheeled foot robot; a model building module configured to build a complex terrain model including a robot model, obstacles, and environmental constraints; A reinforcement learning training module is configured to build a motion decision controller. It uses a reinforcement learning model to perform multi-objective collaborative optimization training on the motion decision controller using an asymmetric training strategy for different complex terrain models. The asymmetric training strategy introduces risk constraints during updates to constrain the strategy's behavior. The complete reward function for reinforcement learning is: ， in, is the main reward function, is the velocity tracking reward function, is a highly stable reward function, is the touchdown penalty function, is the motion stability reward function, is the energy efficiency reward function, is the reward function for special terrain, is a weight parameter specialized for structured step terrain; Multi-objective collaborative optimization obtains the global optimal decision information by setting multiple optimization objectives and setting different weights for weighted combination. The global loss function expression based on multi-objective collaborative optimization design is as follows: ； in, is the global loss function, is the main loss of the strategy, is the risk cost penalty term, is the policy value function loss, is the risk value loss, is the entropy regularization term; The highly stable reward function is as follows: ， in, is the height from the foot end to the fuselage in the z direction, For the predetermined height, is the weight coefficient; The touchdown penalty function is as follows: ， in, is the external contact force on the robot base, is the external contact force on the lower leg of the two-wheeled foot robot, is the weight coefficient; The reward function for special terrain is as follows: ， in, The contact condition for whether the wheels of the two-wheeled robot touch the step terrain, Indicates the wheel number, is the contact force on the wheels of the two-wheeled robot except the vertical direction, is the vertical contact force on the wheel, is the upward speed of the robot, is the upward acceleration of the robot, is the change in the distance from the wheel end to the base of the two-wheeled foot robot between the previous moment and the current moment, is the constraint value of the distance change, 、 and is the weight coefficient; The risk cost penalty term is expressed as follows: ， in, is the penalty coefficient; set a risk cost at each time step ; Similarly, we get the risk value loss: ， in, is the risk value loss function; is the risk value function at time step t; A backpropagation optimization module is configured to perform policy gradient backpropagation based on the training results and optimize the asymmetric training strategy using an adaptive learning rate adjustment method based on performance feedback, thereby achieving adaptive motion control of the two-wheeled legged robot; During the update process, the asymmetric training strategy uses samples to estimate the policy gradient, uses the clipping objective function to update the policy, and introduces the KL divergence to control the amplitude of the policy update; The learning rate is dynamically adjusted by KL divergence to balance the exploration phase in the early stage of training and the convergence phase in the late stage of training. The expression for dynamically adjusting the learning rate is as follows: ， in, represents the learning rate before the kth round of update, k is the training round, is the average KL value calculated at the end of each epoch, Measure the difference in policy distribution before and after the update, It represents the expectation of the data collected at time step t, is the scaling factor, and is the KL threshold.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the complex terrain adaptive motion control method of a two-wheeled legged robot according to any one of claims 1 to 4.

7. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the complex terrain adaptive motion control method of the two-wheeled legged robot according to any one of claims 1 to 4.

Images