Method and device for controlling motion of a wheeled-legged unmanned platform in series

Abstract

The application relates to the technical field of platform motion control, and provides a series wheel-leg unmanned platform motion control method and device. A linear control strategy is provided through LQR, and a reinforcement learning algorithm is combined to gradually optimize the nonlinear compensation of the control strategy through three-stage segmented training, thereby effectively solving the problem that LQR cannot cope with the nonlinear situation when the unmanned platform posture changes. The control strategy is generated through LQR, and then reinforcement learning training is carried out, thereby avoiding the defect that a large number of trial and error are required in the early stage of reinforcement learning. In the reinforcement learning, a reward function is set, including a static stability reward, a dynamic balance reward and a safety constraint penalty, so that the static stability, dynamic compensation and safety constraint are considered, and the unmanned platform can consider stability and response speed.

Description

Technical Field

[0001] This application relates to the field of platform motion control technology, and in particular to a motion control method and device for a tandem wheeled legged unmanned platform. Background Technology

[0002] With the rapid development of various technologies, various unmanned platforms have emerged. Currently, the most widely used are wheeled robots and tracked robots. However, existing unmanned platforms are limited by their mechanical structure, and their mobility and stability in complex terrains remain significantly restricted. For example, wheeled robots often struggle to complete tasks when facing rugged terrain, obstacles, or soft ground due to limited speed or insufficient stability.

[0003] Research on wheel-legged balancing robots has emerged in recent years, with the Ascento and Ollie robots being representative examples. Currently, the mainstream wheel-legged configuration is the five-bar linkage. While this structure offers advantages in stability, its complex mechanical structure results in significant inertia, making it difficult to achieve rapid response and agile maneuverability.

[0004] Furthermore, in terms of control algorithms, the mainstream control algorithm is the LQR algorithm. However, due to the flexion and extension of the legs and the driving of the wheels, the dynamic model of the wheel-legged robot changes with the posture and cannot be accurately described by a single linear model. This invalidates the linear assumption of LQR and further leads to insufficient robustness of control. Summary of the Invention

[0005] Therefore, it is necessary to provide a motion control method and device for a tandem wheeled legged unmanned platform to address the aforementioned technical problems.

[0006] A motion control method for a tandem wheeled-legged unmanned platform, wherein the platform is composed of a hip joint, a knee joint and an ankle joint connected in series, and the platform is equipped with rollers; The method includes the following steps: Construct a simulation environment, and build a tandem wheeled-legged unmanned platform model and a terrain model within the simulation environment; Input the initial state of the tandem wheel-legged unmanned platform model into the simulation environment, and generate a control strategy through LQR; Using the control policy generated by LQR, the set state space, and the set action space as inputs, a policy network and a value network are constructed. A reinforcement learning algorithm is constructed to evaluate the value of the current state of the serial wheeled unmanned platform model using a value network. The policy network is iteratively optimized based on the evaluation results. A reward function is set and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty. The trained control strategy is coordinated with the tandem wheel-legged unmanned platform to achieve control of the tandem wheel-legged unmanned platform and monitor its actions and status in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy is used to control the tandem wheel-legged unmanned platform.

[0007] In one embodiment, the terrain model includes rough flat land, sloping terrain, and obstacle terrain.

[0008] In one embodiment, the policy network includes an input layer, a hidden layer, and an output layer; the input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and a He normal distribution; the hidden layer includes a first hidden layer and a second hidden layer, the first hidden layer containing 64 neurons and the second hidden layer containing 32 neurons, and a normalization layer is provided after each hidden layer; the output layer contains 4 neurons, and the output is limited to a preset range using the Tanh activation function.

[0009] In one embodiment, the value network includes an input layer, a hidden layer, and an output layer; the input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and the He normal distribution; the hidden layer includes a first hidden layer and a second hidden layer, the first hidden layer containing 64 neurons and the second hidden layer containing 32 neurons, and a normalization layer is provided after each hidden layer; the output layer contains 1 neuron and directly outputs the state value.

[0010] In one embodiment, the AdamW optimizer is used to iteratively optimize the policy network. The loss function consists of the pruning policy loss and the mean squared error value loss, with a weight ratio of 1:0.5 between the pruning policy loss and the mean squared error value loss.

[0011] In one embodiment, the set state space is a 16-dimensional vector, including physical constraints and LQR related terms. The physical constraints include fuselage attitude angle, fuselage angular velocity, joint position, joint velocity, terrain height, and ground friction coefficient. The LQR related terms include LQR deviation and LQR output torque. The defined motion space is a 4-dimensional vector, including the LQR output torque of the hip joint, the LQR output torque of the knee joint, the LQR output torque of the ankle joint, and the LQR output torque of the roller.

[0012] In one embodiment, the initial state of the tandem wheel-legged unmanned platform model is input into the simulation environment, and an initial control strategy is generated through LQR, including: Input the initial state of the tandem wheeled unmanned platform model into the simulation environment, including joint constraints, inertial parameters, and sensor configurations for the hip, knee, ankle, and roller joints. The real-time state of the tandem wheel-legged unmanned platform model is obtained through the simulation API, and the deviation between the real-time state and the ideal state is calculated. The deviation is input into the LQR control algorithm to obtain the torque values ​​that the joint motor and hub motor need to output, and a control strategy is generated.

[0013] In one embodiment, the reward function is: ; in, For the reward function; Static stable rewards; For dynamic balance of rewards; For safety constraints and penalties; Discount factor; The static stable reward is: ; in, The core weights for the attitude deviation component; The secondary weight for the joint velocity component; Penalty weights for load change items; This refers to LQR deviation; The first attenuation coefficient; This is the second attenuation coefficient; This represents the difference between the actual load quality and the reference load quality. For the velocity vectors of the hip, knee, and ankle joints; The dynamic balance reward is: ; in, The core weight of the fuselage angular velocity component; This represents the secondary weight of the inertial coupling component; The penalty weight for the total torque component; This is the third attenuation coefficient; This is the fourth attenuation coefficient; The fuselage angular velocity; This refers to the inertial coupling torque of joint motion; This is the superposition of the LQR torque and the reinforcement learning correction torque; The security constraint penalty is determined based on a preset state value.

[0014] In one embodiment, a value network is used to evaluate the value of the current state of the tandem wheeled-legged unmanned platform model. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes a static stable reward, a dynamic equilibrium reward, and a safety constraint penalty, including: In the first training phase, the weight ratio of reinforcement learning algorithm to LQR is fixed at 1:9. Training is conducted on rough flat ground with the platform unloaded until the platform stands upright without tilting, at which point the first control strategy is output. In the second training phase, the weight ratio of the reinforcement learning algorithm increases by 0.1 after each preset number of iterations, until the weight of the reinforcement learning algorithm increases to 0.5. Training is conducted on rough flat land and sloping terrain. The dynamic load is periodically changed according to each preset number of iterations until the pass rate on the sloping terrain reaches a preset value, at which point the second control strategy is output. The third training phase involves dynamically adjusting the weight ratios of the reinforcement learning algorithm. ; in, for The weights of the reinforcement learning algorithm at each time step; To reinforce the initial weights of the learning algorithm; This is a transition period; Training is conducted on rough flat ground and sloping terrain. Different actions are cyclically switched during training until there are no sudden changes in posture during action switching and the pass rate on sloping terrain reaches a preset value. The control strategy after training is then output.

[0015] A motion control device for a tandem wheeled-legged unmanned platform, wherein the platform is composed of a hip joint, a knee joint and an ankle joint connected in series, and the platform is equipped with rollers; The device includes: The simulation building module is used to build a simulation environment, and to build a tandem wheeled legged unmanned platform model and a terrain model within the simulation environment; The LQR module is used to input the initial state of the tandem wheel-legged unmanned platform model in the simulation environment and generate control strategies through LQR. The network construction module is used to construct policy networks and value networks using the control policy generated by LQR, the set state space, and the set action space as inputs. The reinforcement learning module is used to construct reinforcement learning algorithms to evaluate the value of the current state of the tandem wheeled unmanned platform model using a value network. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty. The platform control module is used to coordinate the trained control strategy with the tandem wheel-legged unmanned platform to control the tandem wheel-legged unmanned platform and monitor the actions and status of the tandem wheel-legged unmanned platform in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy will be used to control the tandem wheel-legged unmanned platform.

[0016] The aforementioned tandem wheeled-legged unmanned platform motion control method and device, through a tandem design, connects the hip, knee, and ankle joints sequentially in a series structure, reducing mechanical dead points and improving the unmanned platform's flexibility and adaptability in complex terrain. By providing a linear control strategy through LQR and combining it with reinforcement learning algorithms, the nonlinear compensation of the control strategy is gradually optimized through three-segment segmented training. This effectively solves the problem that LQR cannot handle nonlinear situations when the unmanned platform's attitude changes. Furthermore, by generating the control strategy through LQR and then conducting reinforcement learning training, the drawback of requiring extensive trial and error in the early stages of reinforcement learning is avoided. The reward function in reinforcement learning includes static stability rewards, dynamic balance rewards, and safety constraint penalties, thus balancing static stability, dynamic compensation, and safety constraints, enabling the unmanned platform to achieve both stability and response speed. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the motion control method for a tandem wheeled-legged unmanned platform in one embodiment. Figure 2 This is a schematic diagram of a tandem wheeled unmanned platform in one embodiment; Figure 3 This is a structural block diagram of a motion control device for a tandem wheeled unmanned platform in one embodiment.

[0018] Figure label: 1. Ankle joint; 2. Hip joint; 3. Knee joint. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] The high-fidelity de-identified data generation method based on vector space weighting provided in this application can be applied to, for example... Figure 2 The tandem wheeled unmanned platform shown consists of a hip joint 2, a knee joint 3, and an ankle joint 1 connected in series, and the platform is equipped with rollers.

[0021] In one embodiment, such as Figure 1 As shown, a motion control method for a tandem wheel-legged unmanned platform is provided, which is then applied to... Figure 2 Taking a tandem wheeled-legged unmanned platform as an example, the method includes the following steps: Step 201: Construct a simulation environment. In the simulation environment, construct a tandem wheeled legged unmanned platform model and a terrain model.

[0022] Step 202: Input the initial state of the tandem wheel-legged unmanned platform model into the simulation environment, and generate a control strategy through LQR.

[0023] Step 203: Using the control policy generated by LQR, the set state space, and the set action space as inputs, construct the policy network and the value network.

[0024] Step 204: Construct a reinforcement learning algorithm to evaluate the value of the current state of the tandem wheeled unmanned platform model using a value network. Iteratively optimize the policy network based on the evaluation results. Set a reward function and perform three-segment segmented training on the policy network by maximizing the reward function to optimize nonlinear compensation. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty.

[0025] Step 205: The trained control strategy is coordinated with the tandem wheel-legged unmanned platform to achieve control of the tandem wheel-legged unmanned platform and monitor the actions and status of the tandem wheel-legged unmanned platform in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy is used to control the tandem wheel-legged unmanned platform.

[0026] In the aforementioned motion control method for a tandem wheel-legged unmanned platform, a linear control strategy is provided through LQR (Limited Quality Reduction) and combined with reinforcement learning algorithms. The nonlinear compensation of the control strategy is gradually optimized through three-segment segmented training, thereby effectively solving the problem that LQR cannot cope with nonlinear situations when the attitude of the unmanned platform changes. Furthermore, by generating the control strategy through LQR and then conducting reinforcement learning training, the drawback of requiring a large number of trial and error in the early stages of reinforcement learning is avoided. In addition, the reward function set in reinforcement learning includes static stability reward, dynamic balance reward, and safety constraint penalty, thereby taking into account static stability, dynamic compensation, and safety constraints, enabling the unmanned platform to balance stability and response speed.

[0027] In one embodiment, the terrain model includes rough flat land, sloping terrain, and obstacle terrain.

[0028] In one embodiment, the state space is set as a 16-dimensional vector, including physical constraints and LQR related terms. The physical constraints include fuselage attitude angle, fuselage angular velocity, joint position, joint velocity, terrain height and ground friction coefficient. The LQR related terms include LQR deviation and LQR output torque. The motion space is defined as a 4-dimensional vector, including the LQR output torque of the hip joint, the LQR output torque of the knee joint, the LQR output torque of the ankle joint, and the LQR output torque of the roller.

[0029] In one embodiment, the policy network includes an input layer, a hidden layer, and an output layer; the input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and the He normal distribution; the hidden layer includes a first hidden layer and a second hidden layer, the first hidden layer contains 64 neurons, the second hidden layer contains 32 neurons, and a normalization layer is provided after each hidden layer; the output layer contains 4 neurons, and the output is limited to a preset range using the Tanh activation function.

[0030] In this embodiment, by employing the Tanh activation function to limit the output within a preset range, the bounded output characteristic of the Tanh activation function is utilized to achieve strict limitation on the output of the policy network without additional logic. Specifically, the Tanh activation function is used to limit the output within... , This provides the maximum torque for the joint, effectively compensating for LQR errors in complex terrain; it also ensures LQR maintains control dominance, preventing control logic confusion caused by excessive reinforcement learning output.

[0031] In one embodiment, the value network includes an input layer, a hidden layer, and an output layer; the input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and the He normal distribution; the hidden layer includes a first hidden layer and a second hidden layer, the first hidden layer contains 64 neurons, the second hidden layer contains 32 neurons, and a normalization layer is provided after each hidden layer; the output layer contains 1 neuron, which directly outputs the state value.

[0032] Understandably, value networks and policy networks are structurally similar, the difference being in the output layer. The value network's output layer contains only one neuron, directly outputting the state value without applying an activation function. Furthermore, each hidden layer is followed by a normalization layer to stabilize training, accelerate convergence, and mitigate gradient vanishing.

[0033] He's normal distribution, the core of which is a distribution with a mean of 0 and a standard deviation of 0. The sampling weights are derived from a normal distribution, where... The number of input neurons.

[0034] In this embodiment, the He normal distribution is adopted to keep the variance of activation values ​​of each layer stable during forward and backward propagation, effectively alleviate the gradient vanishing or exploding problem in deep networks, significantly accelerate model convergence and improve training stability.

[0035] In one embodiment, the AdamW optimizer is used to iteratively optimize the policy network. The loss function consists of the pruning policy loss and the mean squared error value loss, with a weight ratio of 1:0.5 between the pruning policy loss and the mean squared error value loss.

[0036] In this embodiment, the AdamW optimizer is used to iteratively optimize the policy network. The AdamW optimizer can combine adaptive learning rate to allocate appropriate learning rates to different parameters, making the parameters of the policy network and the value network smoother and avoiding training oscillations caused by inappropriate learning rates. By setting the weight ratio of the pruning policy loss to the mean squared error value loss to 1:0.5, it is ensured that the value network can learn sufficiently accurate state values, while avoiding update bias caused by excessively high value loss weights, which would lead to the policy over-reliance on temporarily inaccurate value estimates.

[0037] In one embodiment, the initial state of the tandem wheel-legged unmanned platform model is input into the simulation environment, and an initial control strategy is generated through LQR, including: Input the initial state of the tandem wheeled unmanned platform model into the simulation environment, including joint constraints, inertial parameters, and sensor configurations for the hip, knee, ankle, and roller joints. The real-time state of the tandem wheel-legged unmanned platform model is obtained through the simulation API, and the deviation between the real-time state and the ideal state is calculated. The deviation is input into the LQR control algorithm to obtain the torque values ​​that the joint motor and hub motor need to output, and a control strategy is generated.

[0038] In one embodiment, the reward function is: ; in, For the reward function; Static stable rewards; For dynamic balance of rewards; For safety constraints and penalties; Discount factor; The static stable reward is: ; in, The core weights for the attitude deviation component; The secondary weight for the joint velocity component; Penalty weights for load change items; This refers to LQR deviation; The first attenuation coefficient; This is the second attenuation coefficient; This represents the difference between the actual load quality and the reference load quality. For the velocity vectors of the hip, knee, and ankle joints; The dynamic balancing reward is: ; in, The core weight of the fuselage angular velocity component; This represents the secondary weight of the inertial coupling component; The penalty weight for the total torque component; This is the third attenuation coefficient; This is the fourth attenuation coefficient; The fuselage angular velocity; This refers to the inertial coupling torque of joint motion; This is the superposition of the LQR torque and the reinforcement learning correction torque; Safety constraint penalties are assigned values ​​based on preset states.

[0039] Specifically It contains column vectors of velocities for the hip, knee, and ankle joints.

[0040] Specifically, the safety constraint penalties are determined according to the following table: Table 1 Safety Constraint Penalty Value Table

[0041] It should be noted that The distance between the robot and the obstacle is calculated using the rayTest function in PyBullet.

[0042] In one embodiment, a value network is used to evaluate the value of the current state of the tandem wheeled-legged unmanned platform model. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes a static stable reward, a dynamic equilibrium reward, and a safety constraint penalty, including: In the first training phase, the weight ratio of reinforcement learning algorithm to LQR is fixed at 1:9. Training is conducted on rough flat ground with the platform unloaded until the platform stands upright without tilting, at which point the first control strategy is output. In the second training phase, the weight ratio of the reinforcement learning algorithm increases by 0.1 after each preset number of iterations, until the weight of the reinforcement learning algorithm increases to 0.5. Training is conducted on rough flat land and sloping terrain. The dynamic load is periodically changed according to each preset number of iterations until the pass rate on the sloping terrain reaches a preset value, at which point the second control strategy is output. The third training phase involves dynamically adjusting the weight ratios of the reinforcement learning algorithm. ; in, for The weights of the reinforcement learning algorithm at each time step; To reinforce the initial weights of the learning algorithm; This is a transition period; Training is conducted on rough flat ground and sloping terrain. Different actions are cyclically switched during training until there are no sudden changes in posture during action switching and the pass rate on sloping terrain reaches a preset value. The control strategy after training is then output.

[0043] In the first stage, the pruning factor in the policy network is set to 0.15 to limit the magnitude of each policy update, and the reward function is mainly static stable reward. In the value network, only the cumulative discounted reward of the static stable reward is used, so that the value network prioritizes learning the value of static states and reduces the training frequency to avoid the impact of value estimation fluctuations on the stability of the policy.

[0044] It should be noted that static stable rewards are "instant rewards" for a single time step; cumulative discounted rewards are the sum of all subsequent instant rewards multiplied by a discount factor, starting from the current time step. In essence, static stable rewards are a component of cumulative discounted rewards, which are the sum of multiple static stable rewards (after discounting by time step).

[0045] In the second stage, the weights of the reinforcement learning algorithm in the policy network are gradually increased to 0.5. Rough terrain, slopes, and dynamic loads are introduced to collect more complex trajectories. Static stable rewards and dynamic equilibrium rewards are used in the reward function, and the pruning factor is set to 0.2, allowing the policy network to update more significantly to learn nonlinear compensation. A correlation penalty between the reinforcement learning torque and the LQR output torque is added to the loss function to prevent the correction term from being completely unrelated to the LQR output. In the value network, static stable rewards and dynamic equilibrium rewards are integrated, and the training frequency is increased to improve the real-time performance of value estimation and better guide policy updates.

[0046] The relevance penalty in the second phase is: ; ; in, Penalty based on relevance; This is the penalty coefficient, with a value of 0.15; Output torque for LQR; To enhance the learning of torque.

[0047] In the third stage, when training different actions, state data during the switching process is collected and the reward function is weighted. The trajectory data during the switching stage is given twice the weight, so that the policy network learns the correction strategy during the switching first. In the value network, a smooth transition reward is added to the reward function. The smaller the change in body angle during the switching, the higher the reward.

[0048] The smooth transition reward is: ; in, Rewards for a smooth transition; This is the weighting coefficient, set to 0.4; The attenuation coefficient is set to 20; This represents the maximum fluctuation in the fuselage pitch angle during the switching process.

[0049] By adding a smooth transition reward to the reward function, the model is guided to exhibit less attitude fluctuation and a smoother process when switching operating conditions, thus avoiding the risk of sudden attitude changes or tipping over.

[0050] In this embodiment, the policy network learns and follows LQR in the first stage to avoid drastic fluctuations in the initial exploration phase; in the second stage, the policy network learns to compensate for nonlinear errors to adapt to disturbances in the process; and in the third stage, the policy network learns to smoothly transition between operating conditions, thereby adapting to sudden changes in state. Through three-stage segmented training, the weights of reinforcement learning are gradually increased, and in the third stage of training, the weights of reinforcement learning can be dynamically adjusted, enabling the unmanned platform to adaptively adjust according to operating conditions and ensuring no sudden changes in attitude during the adjustment process, thus successfully traversing sloping terrain.

[0051] In one embodiment, the three-segment segmented training is as follows: In the first training phase, the weight ratio of the reinforcement learning algorithm to LQR was fixed at 1:9. Training was conducted on a rough, flat surface with no load on the platform. The number of iterations ranged from 1 to 500, with a training duration of 1.5 hours. The target speed of the aircraft was 0.2 m / s. During 50 consecutive iterations... If the proportion of values ​​≤0.05 rad is ≥90% and there is no tipping, then convergence is determined, and the first control strategy is output.

[0052] In the second training phase, the weight ratio of the reinforcement learning algorithm increased by 0.1 after every 200 iterations, until the weight of the reinforcement learning algorithm increased to 0.5. Training was conducted on rough flat ground (height fluctuation of 0.01m~0.02m) and sloping terrain (5°~10° slope), with a dynamic load increased or decreased by 1kg every 100 iterations. If ω ≤ 0.08 N·m (inertial coupling cancellation rate ≥ 40%) and ω ≤ 0.6 rad / s when walking on a slope, then convergence is determined, and the second control strategy is output.

[0053] In the third training phase, the weight ratios of the reinforcement learning algorithm are dynamically adjusted: ; in, for The weights of the reinforcement learning algorithm at each time step; To reinforce the initial weights of the learning algorithm; This is a transition period; Training was conducted on rough flat ground and sloping terrain. During the training, the posture fluctuations were recorded in a cycle of "standing (5s) → walking (10s) → walking on a 10° slope (10s) → stopping (5s)". In 100 consecutive cycles, If the percentage of values ​​≤0.1rad is ≥95%, then convergence is determined, and the trained control strategy is output.

[0054] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0055] In one embodiment, such as Figure 3 As shown, a motion control device for a tandem wheeled-legged unmanned platform is provided. The platform consists of a hip joint, knee joint, and ankle joint connected in series. The platform is equipped with rollers. The device includes: Simulation building module 901 is used to build a simulation environment, and to build a tandem wheeled legged unmanned platform model and a terrain model in the simulation environment; The LQR module 902 is used to input the initial state of the tandem wheel-legged unmanned platform model in the simulation environment and generate a control strategy through LQR. Network construction module 903 is used to construct policy network and value network using the control policy generated by LQR, the set state space and the set action space as input; Reinforcement learning module 904 is used to construct reinforcement learning algorithms to evaluate the value of the current state of the tandem wheeled unmanned platform model using a value network. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty. The platform control module 905 is used to coordinate the trained control strategy with the tandem wheel-legged unmanned platform to control the tandem wheel-legged unmanned platform and monitor the actions and status of the tandem wheel-legged unmanned platform in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy is used to control the tandem wheel-legged unmanned platform.

[0056] Specific limitations regarding the motion control device for the tandem wheeled-legged unmanned platform can be found in the limitations of the motion control method for the tandem wheeled-legged unmanned platform described above, and will not be repeated here. Each module in the aforementioned motion control device for the tandem wheeled-legged unmanned platform can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A motion control method for a tandem wheeled-legged unmanned platform, characterized in that, For a tandem wheeled-legged unmanned platform, the platform is composed of a hip joint, a knee joint and an ankle joint connected in series, and the platform is equipped with rollers; The method includes the following steps: Construct a simulation environment, and build a tandem wheeled-legged unmanned platform model and a terrain model within the simulation environment; Input the initial state of the tandem wheel-legged unmanned platform model into the simulation environment, and generate a control strategy through LQR; Using the control policy generated by LQR, the set state space, and the set action space as inputs, a policy network and a value network are constructed. A reinforcement learning algorithm is constructed to evaluate the value of the current state of the serial wheeled unmanned platform model using a value network. The policy network is iteratively optimized based on the evaluation results. A reward function is set and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty. The trained control strategy is coordinated with the tandem wheel-legged unmanned platform to achieve control of the tandem wheel-legged unmanned platform and monitor its actions and status in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy is used to control the tandem wheel-legged unmanned platform.

2. The motion control method for a tandem wheel-legged unmanned platform according to claim 1, characterized in that, The terrain model includes rough flat land, sloping terrain, and obstacle terrain.

3. The motion control method for a tandem wheel-legged unmanned platform according to claim 1, characterized in that, The policy network includes an input layer, a hidden layer, and an output layer. The input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and the He normal distribution. The hidden layer includes a first hidden layer and a second hidden layer. The first hidden layer contains 64 neurons, and the second hidden layer contains 32 neurons. Each hidden layer is followed by a normalization layer. The output layer contains 4 neurons, and the output is limited to a preset range using the Tanh activation function.

4. The motion control method for a tandem wheel-legged unmanned platform according to claim 3, characterized in that, The value network comprises an input layer, a hidden layer, and an output layer. The input layer contains 16 neurons, and the weights are initialized using the ReLU activation function and the He normal distribution. The hidden layer comprises a first hidden layer and a second hidden layer, the first hidden layer containing 64 neurons and the second hidden layer containing 32 neurons, with a normalization layer following each hidden layer. The output layer contains 1 neuron and directly outputs the state value.

5. The motion control method for a tandem wheel-legged unmanned platform according to claim 4, characterized in that, The AdamW optimizer is used to iteratively optimize the policy network. The loss function consists of the pruning policy loss and the mean squared error value loss, with a weight ratio of 1:0.5 between the pruning policy loss and the mean squared error value loss.

6. The motion control method for a tandem wheel-legged unmanned platform according to claim 1, characterized in that, The set state space is a 16-dimensional vector, including physical constraints and LQR related terms. The physical constraints include fuselage attitude angle, fuselage angular velocity, joint position, joint velocity, terrain height and ground friction coefficient. The LQR related terms include LQR deviation and LQR output torque. The defined motion space is a 4-dimensional vector, including the LQR output torque of the hip joint, the LQR output torque of the knee joint, the LQR output torque of the ankle joint, and the LQR output torque of the roller.

7. The motion control method for a tandem wheeled-legged unmanned platform according to claim 1, characterized in that, Input the initial state of the tandem wheel-legged unmanned platform model into the simulation environment, and generate the initial control strategy through LQR, including: Input the initial state of the tandem wheeled unmanned platform model into the simulation environment, including joint constraints, inertial parameters, and sensor configurations for the hip, knee, ankle, and roller joints. The real-time state of the tandem wheel-legged unmanned platform model is obtained through the simulation API, and the deviation between the real-time state and the ideal state is calculated. The deviation is input into the LQR control algorithm to obtain the torque values ​​that the joint motor and hub motor need to output, and a control strategy is generated.

8. The motion control method for a tandem wheel-legged unmanned platform according to claim 1, characterized in that, The reward function is: in, For the reward function; Static stable rewards; For dynamic balance of rewards; For safety constraints and penalties; Discount factor; The static stable reward is: in, The core weights for the attitude deviation component; The secondary weight for the joint velocity component; Penalty weights for load change items; This refers to LQR deviation; The first attenuation coefficient; This is the second attenuation coefficient; This represents the difference between the actual load quality and the reference load quality. For the velocity vectors of the hip, knee, and ankle joints; The dynamic balance reward is: in, The core weight of the fuselage angular velocity component; This represents the secondary weight of the inertial coupling component; The penalty weight for the total torque component; This is the third attenuation coefficient; This is the fourth attenuation coefficient; The fuselage angular velocity; This refers to the inertial coupling torque of joint motion; This is the superposition of the LQR torque and the reinforcement learning correction torque; The security constraint penalty is determined based on a preset state value.

9. The motion control method for a tandem wheel-legged unmanned platform according to claim 2, characterized in that, A reinforcement learning algorithm is constructed to evaluate the value of the current state of a tandem wheeled-legged unmanned platform model using a value network. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes a static stable reward, a dynamic equilibrium reward, and a safety constraint penalty, including: In the first training phase, the weight ratio of reinforcement learning algorithm to LQR is fixed at 1:

9. Training is conducted on rough flat ground with the platform unloaded until the platform stands upright without tilting, at which point the first control strategy is output. In the second training phase, the weight ratio of the reinforcement learning algorithm increases by 0.1 after each preset number of iterations, until the weight of the reinforcement learning algorithm increases to 0.

5. Training is conducted on rough flat land and sloping terrain. The dynamic load is periodically changed according to each preset number of iterations until the pass rate on the sloping terrain reaches a preset value, at which point the second control strategy is output. The third training phase involves dynamically adjusting the weight ratios of the reinforcement learning algorithm. in, for Weights of the reinforcement learning algorithm at each time step; To reinforce the initial weights of the learning algorithm; This is a transition period; Training is conducted on rough flat ground and sloping terrain. Different actions are cyclically switched during training until there are no sudden changes in posture during action switching and the pass rate on sloping terrain reaches a preset value. The control strategy after training is then output.

10. A series-type wheeled-legged unmanned platform motion control device, characterized in that, For a tandem wheeled-legged unmanned platform, the platform is composed of a hip joint, a knee joint and an ankle joint connected in series, and the platform is equipped with rollers; The device includes: The simulation building module is used to build a simulation environment, and to build a tandem wheeled legged unmanned platform model and a terrain model within the simulation environment; The LQR module is used to input the initial state of the tandem wheel-legged unmanned platform model in the simulation environment and generate control strategies through LQR. The network construction module is used to construct policy networks and value networks using the control policy generated by LQR, the set state space, and the set action space as inputs. The reinforcement learning module is used to construct reinforcement learning algorithms to evaluate the value of the current state of the tandem wheeled unmanned platform model using a value network. Based on the evaluation results, the policy network is iteratively optimized. A reward function is set, and the policy network is trained in three segments to optimize nonlinear compensation by maximizing the reward function. The reward function includes static stable reward, dynamic balance reward, and safety constraint penalty. The platform control module is used to coordinate the trained control strategy with the tandem wheel-legged unmanned platform to control the tandem wheel-legged unmanned platform and monitor the actions and status of the tandem wheel-legged unmanned platform in real time. If the actions and status of the tandem wheel-legged unmanned platform exceed the preset values, the initial control strategy will be used to control the tandem wheel-legged unmanned platform.

Images