Amphibious quadruped robot motion mode switching method based on reinforcement learning

Abstract

The invention discloses an amphibious quadruped robot motion mode switching method based on reinforcement learning, and belongs to the field of robot motion control. Aiming at the motion control challenge of the robot in an amphibious environment, a self-adaptive propulsion control frame is provided, and a propeller and a leg motion system of the quadruped robot are deeply fused. Through hierarchical reinforcement learning, a low-level controller for land walking and water swimming is trained, an advanced skill coordination network is designed, historical data is used for deducing environment characteristics and predicting future states, and dynamic switching of motion modes is achieved. According to the method, the motion robustness of the robot under unknown interference is remarkably improved, the method can adapt to diversified scenes without additional fine adjustment, and the method has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of robot motion control, and particularly relates to a motion control method for an amphibious quadruped robot based on hierarchical reinforcement learning and thruster-leg collaborative control, which is suitable for water-land alternation and low-gravity environment. BACKGROUND

[0002] With the increasing maturity of robot control technology, leg-foot type robots exhibit significant advantages in complex terrain adaptability due to their discrete contact point motion characteristics. Although quadruped robots can dynamically adjust the landing points according to the terrain, they still face severe challenges in coordinating multiple actuators (such as leg-foot joints and thrusters) in high-dimensional action space. Traditional six-legged amphibious robots (such as CPG control models) have underwater stability, but their complex mechanical structure leads to poor land mobility and high energy consumption, making it difficult to adapt to the dynamic switching requirements of water-land environments.

[0003] Especially in water-land transition, low-gravity or buoyancy environments, robots often have limited motion capabilities due to foot contact failure. In existing technologies, the dynamic coupling modeling of thrusters and leg feet is difficult, and the underwater environment perception information is limited, resulting in a lack of environmental adaptive decision-making ability for robots. Although reinforcement learning has made progress in single robot path planning, there are still obvious deficiencies in multi-modal motion collaborative control (such as real-time switching between swimming and walking) and environmental reasoning, and an intelligent decision-making comprehensive solution is urgently needed. SUMMARY

[0004] The application proposes a thruster-based quadruped robot control method based on hierarchical reinforcement learning, which realizes efficient motion control of robots in land and underwater environments for amphibious quadruped robots with thrusters integrated into the quadruped robot structure combined with hierarchical reinforcement learning algorithm.

[0005] The technical solution adopted by the application to achieve the above purpose is: a reinforcement learning-based amphibious quadruped robot motion mode switching method, comprising the following steps:

[0006] Constructing two bottom controllers for controlling the robot actuators;

[0007] Predicting environmental features and future robot states through historical trajectory data, and dynamically weighting the outputs of the two bottom controllers to realize motion mode switching.

[0008] The construction of the two bottom controllers for controlling the robot actuators includes the following steps:

[0009] The first bottom controller is used for learning land walking skills, receives body state information and expected movement speed instructions from the robot, and obtains movement speed instructions for controlling leg joints through a reinforcement learning algorithm; the body state information includes joint angles, joint speeds, foot end ground contact states, torso postures and angular velocities; the movement speed instructions include leg joint speeds;

[0010] The second bottom controller is used for learning underwater swimming skills, receives robot body state information and expected underwater movement speed instructions, and obtains movement speed instructions for controlling the propeller and leg joints through a reinforcement learning algorithm, including propeller thrust size and leg joint speed; the underwater movement speed instructions include propeller thrust size and leg joint speed.

[0011] The prediction of environmental features and future body states through historical trajectory data includes the following steps:

[0012] 1) Obtain a historical body state information sequence X = [O t-k ,O t-k+1 ,...,O t ], wherein O t represents the state of the robot at time t, k represents a discrete time, and t represents time; N is the batch size, which is used to represent the batch number of body state information sequences, C is the number of attention channels, and L is the sequence length, and a 1D convolutional neural network is used to extract local spatiotemporal features:

[0013] 2) Calculate the average pooling in the dimension:

[0014] 3) Then obtain the channel attention weight through a two-layer fully connected network to highlight the most relevant feature channels:

[0015] att c = σ (W2·δ (W1·avg))

[0016] Wherein att c represents the channel attention, W1 and W2 are learnable weights, δ is the ReLU activation, and σ is the sigmoid function;

[0017] 4) Obtain features: X channel = Xe att c ;

[0018] Wherein e represents element multiplication, and att c is propagated in the time dimension;

[0019] 5) A 1D convolutional neural network with a learnable dilation rate di a number of 1D convolutional layers, and the feature channel of the ith 1D convolutional layer is denoted as att i :

[0020] att i = Conv1d(X channel , dilation=d i )

[0021] wherein Conv1d represents a 1D CNN convolution kernel, dilation represents the dilation rate; i represents the 1D convolutional layer number;

[0022] 6) Integrate features of different time scales through a time attention mechanism:

[0023] att total = concat([att1,...,att M ])

[0024] X attended = X channel e att total

[0025] wherein att total represents the concatenated features, concat represents concatenation, and attn M represents the Mth channel attention;

[0026] 7) The features are summarized into a hidden state h:

[0027] 8) Process h through a decoder to predict the environment information: e, s' = Decoder(h);

[0028] e represents the environmental features, including buoyancy and altitude, and s' represents the body state information of the robot at the subsequent time step, i.e., the future body state.

[0029] The dynamic weighting control of the outputs of the two bottom controllers includes the following steps:

[0030] According to the hidden state h in the predicted environment information, combined with the current body state information O t The SAC network is used to generate the weights a1 and a2 of different skills, and the composite control instructions are sent to the robot leg joints and thrusters, as follows:

[0031] For the first bottom controller output, the motion speed instruction for controlling the leg joint is weighted by a1 and output to the leg joint; for the second bottom controller output, the motion speed instruction for controlling the thruster and leg joint is weighted by a2 and output to the thruster and leg joint.

[0032] Switching the motion mode of the robot according to the environmental characteristics, comprising the following steps:

[0033] The SAC network judges the ground environment according to the environmental characteristics e, and the weight a1 of the first bottom controller is greater than a2;

[0034] The SAC network judges the underwater environment according to the environmental characteristics e, and the weight a2 of the second bottom controller is greater than a1;

[0035] The SAC network judges the water-land transition area according to the environmental characteristics e, and the weights of the first bottom controller and the second bottom controller are obtained through the SAC network.

[0036] The motion mode switching system of the amphibious quadruped robot based on reinforcement learning comprises:

[0037] The bottom controller unit is used to build two bottom controllers for controlling the robot actuators;

[0038] The environmental state prediction unit is used to predict the environmental characteristics and the future body state through historical trajectory data, and to realize the switching of the motion mode by dynamically weighting the outputs of the two bottom controllers.

[0039] The environmental state prediction unit comprises:

[0040] The neural network unit is used to predict the environmental characteristics and the future body state through historical trajectory data;

[0041] The SAC network is used to control the outputs of the two bottom controllers by dynamic weighting, comprising: according to the hidden state h in the predicted environmental information, combining the current body state information O t The SAC network generates the weights a1 and a2 of different skills, and sends the composite control instructions to the robot leg joints and the thruster, so as to switch the motion mode of the robot according to the environmental characteristics.

[0042] The present application has the following advantages and benefits:

[0043] 1. The present application trains an integrated thruster quadruped robot through reinforcement learning, so that it can realize speed tracking in land and underwater environments, and significantly improves the motion efficiency and adaptability of the robot in amphibious environments.

[0044] 2. The present application proposes a skill coordination network, which analyzes the historical trajectory of the robot to infer the environmental background, and adaptively adjusts the weights of the land walking and underwater swimming controllers, so as to realize seamless switching between different motion modes. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the workflow diagram of the method of the present application. DETAILED DESCRIPTION

[0046] The application will be further described in detail below with reference to the accompanying drawings and examples.

[0047] The application relates to a thrust leg amphibious robot control method based on hierarchical reinforcement learning and application. The method uses a hierarchical reinforcement learning framework to train the robot to realize autonomous coordinated control of land walking and underwater swimming dual-mode motion, and can switch the motion mode in real time and smoothly according to the change of the environment. The amphibious quadruped robot is prior art, and the robot body is a quadruped structure, a plurality of vector thrusters (such as small bypass ratio thrusters) are installed on the trunk, and the thrusters can provide thrusts in different directions and strengths.

[0048] In the implementation of the application, two independent bottom layer reinforcement learning controllers are first trained. The first bottom layer controller learns land walking skills: the controller receives body state information (such as joint angle, joint speed, foot end ground contact state, trunk posture and angular velocity) and expected motion speed instructions (such as leg joint speed) from the robot, and trains the robot in a simulation environment through a reinforcement learning algorithm (such as PPO, SAC, etc.) to enable the robot to stably and efficiently track the given speed command on various land terrains. The second bottom layer controller learns underwater swimming skills: the controller also receives robot body state information and expected underwater motion speed instructions (including thruster thrust size and leg joint speed), but the state space and reward function design needs to consider the fluid dynamics characteristics of water (such as resistance, buoyancy) and the synergistic effect of thrusters and legs in water. The training goal is to enable the robot to track the given speed underwater and achieve efficient propulsion.

[0049] After the completion of the bottom layer controller training, the key step of the present application is to train a high layer skill coordination network. The high layer network continuously receives the historical state sequence information of the robot (including but not limited to the recent joint state, torso state, foot bottom contact force / state, thruster state, and the tracking error of the expected speed and actual speed within a historical time window). The network analyzes these historical trajectory information, infers the current environment state or mode of the robot (for example, whether it is underwater, whether it is in the shallow transition area, whether it is in the low gravity simulation environment, whether the foot end can reliably contact the ground, the environmental buoyancy condition, etc.). Based on this environment inference, the high layer network dynamically calculates the fusion weight of the output of the two bottom layer controllers (walking and swimming) to generate the final composite control instruction sent to the joints and thrusters of the robot. This process realizes an adaptive control strategy: when the environmental conditions are mainly suitable for walking (such as dry ground), the weight of the walking controller dominates; when in underwater environment, the weight of the swimming controller dominates; and in ambiguous environments such as water-land transition areas, the network can intelligently fuse the outputs of the two to achieve smooth mode switching and efficient propulsion.

[0050] The application of the present application is to enable a legged robot equipped with thrusters (especially a quadruped robot) to achieve efficient and robust all-terrain movement in complex and dynamically changing amphibious environments and simulated low gravity environments (such as marine exploration, tidal zone operations, lunar / mars surface surveying, etc.). The robot can autonomously decide the main motion mode or the fusion ratio of multiple modes to be taken according to the sensed self-state and environmental feedback (partially observable or difficult to accurately model), using the trained hierarchical controller, thereby realizing the transition from land to underwater or vice versa, and coping with challenging scenarios such as sudden disappearance of the ground under the feet (such as stepping into a deep pit or water body). This technology solves the problem of traditional robots being difficult to move flexibly in water-land interface and unstable support environments, significantly improving the environmental adaptability and task completion ability of the robot.

[0051] Next, the implementation steps of the present application will be described in detail in conjunction with the specific process:

[0052] As shown in Figure 1 , a motion mode switching method for amphibious quadruped robots based on reinforcement learning is used for amphibious robot control, and the main steps are as follows:

[0053] Step 1: Build an adaptive amphibious skill coordination network to predict environmental features and future body states through historical trajectory data, dynamically weight the output of low-level controllers, and achieve adaptive motion control in water-land environments.

[0054] Step 2: Promote its real-time adjustment of behavior in challenging scenarios such as amphibious to complete speed and trajectory tracking.

[0055] The output of a trained low-level network can be weighted based on environmental information to facilitate movement under amphibious conditions. By utilizing historical trajectories to predict environmental features and future body states, the robot executes movements based on the weighted output of the low-level controller.

[0056] To infer current environmental characteristics from the robot's historical information and predict the robot's next physical state, the historical ontological state information sequence X = [O t-k O t-k+1 ,...,O t ], where O t This represents the state of the robot at time t. Let N represent the batch size, C represent the number of channels (state features), and L represent the sequence length. A 1D convolutional neural network (1DCNN) is used to extract local spatiotemporal features.

[0057]

[0058] Average pooling across the computational dimension:

[0059]

[0060] This produces Then, a two-layer fully connected network is used to derive channel attention weights to highlight the most relevant feature channels:

[0061] att c =σ(W2·δ(W1·avg))

[0062] Where W1 and W2 are learnable weights, δ is ReLU activation, and σ is the sigmoid function, the features are obtained as follows:

[0063] X channel =Xe att c

[0064] Where 'e' represents element-wise multiplication, propagated over time by 'att'. c To further model time dependence, a learnable inflation rate d is used. i Multiple 1D convolutional layers are used. These dilation rates are treated as trainable parameters, initialized at [1,2,4], and optimized via backpropagation during training. This method enables the convolutional kernels to adaptively adjust their temporal receptive fields, thereby improving prediction accuracy in dynamic environments.

[0065] attn i =Conv1d(X channel dilation=d i )

[0066] Conv1d denotes a 1D CNN convolution kernel, and dilation denotes dilation rate.

[0067] The temporal attention mechanism integrates features of different time scales and gives more weight to time steps.

[0068] X attended = X channel e concat([attn1,...,attn M ])

[0069] The features are summarized into a hidden state h:

[0070]

[0071] The decoder, implemented as a multi-layer perceptron, processes h to predict environment information:

[0072] e,s′=Decoder(h)

[0073] In this framework, e represents environmental features, including variables such as buoyancy and height, while s' represents the robot's pose information at subsequent time steps. This approach enables the policy to effectively represent the robot and its environment using trajectory data.

[0074] After this prediction process, the Soft Actor-Critic (SAC) network uses the hidden representation h to generate corresponding weights for different skills in combination with the current observation.

[0075] The above examples are used to explain and illustrate the present application, but not to limit the present application, any modifications and changes made to the present application within the spirit and protection scope of the claims of the present application, fall within the protection scope of the present application.

Claims

1. A method for switching motion modes of an amphibious quadruped robot based on reinforcement learning, characterized in that, The method comprises the following steps: constructing two bottom controllers for controlling the robot actuators; predicting the environmental features and the future body state through historical trajectory data, and dynamically weighting the outputs of the two bottom controllers to realize the switching of the motion mode.

2. The method of claim 1, wherein, The constructing of the two bottom controllers for controlling the robot actuators comprises the following steps: The first bottom controller is used for learning the land walking skill, receives the body state information and the expected motion speed instruction from the robot, and obtains the motion speed instruction for controlling the leg joints through a reinforcement learning algorithm; the body state information comprises the joint angle, the joint speed, the foot end ground contact state, the trunk posture and the angular velocity; the motion speed instruction comprises the speed of the leg joints; The second bottom controller is used for learning the underwater swimming skill, receives the body state information and the expected underwater motion speed instruction, and obtains the motion speed instruction for controlling the propeller and the leg joints through a reinforcement learning algorithm, which comprises the thrust of the propeller and the speed of the leg joints; the underwater motion speed instruction comprises the thrust of the propeller and the speed of the leg joints.

3. The method of claim 1, wherein, The predicting of the environmental features and the future body state through historical trajectory data comprises the following steps: 1) Obtain a sequence of historical ontology state information X = [O t-k ,O t-k+1 ,...,O t ], wherein O t represents the state of the robot at time t, k represents a discrete time, and t represents time; N is the batch size, used to represent the number of batches of ontology state information, C is the number of attention channels, and L is the sequence length, and a 1D convolutional neural network is used to extract local spatiotemporal features: 2) Average pooling over the computed dimensions: 3) Then a two-layer fully connected network is used to obtain the channel attention weight to highlight the most relevant feature channel: att c = σ(W2 · δ(W1 · avg)) where att c denotes channel attention, W1and W2are learnable weights, δ is a ReLU activation, and σ is a sigmoid function; 4) GetFeature: X channel = Xe att c ; where e denotes element-wise multiplication, and att propagates in the time dimension c ; 5) using a plurality of 1D convolution layers with a learnable dilation rate d i of the i-th 1D convolution layer att i : att i = Conv1d(X channel , dilation=d i ) wherein, Conv1d represents a 1D CNN convolution kernel, dilation represents the dilation rate; i represents the 1D convolution layer number; 6) The features of different time scales are integrated through a time attention mechanism: att total = concat([att1,...,att M ]) X attended = X channel e att total wherein att total denotes the concatenated features, concat denotes the concatenation, attn M denotes the M-th channel attention; 7) The features are summarized into a hidden state h: 8) The environment information is predicted by processing h through a decoder: e, s' = Decoder(h); e represents the environmental features, including the buoyancy and the height, and s' represents the body state information of the robot at the subsequent time step, i.e. the future body state.

4. The method of claim 1, wherein, The dynamically weighting of the outputs of the two bottom controllers comprises the following steps: According to the hidden state h in the predicted environment information, combined with the current ontology state information O t The SAC network is used to generate the weights a1 and a2 of different skills, and the composite control instructions of the robot leg joints and the thruster are sent, as follows: The motion speed instruction for controlling the leg joints output by the first bottom controller is weighted by a1 and then output to the leg joints; the motion speed instruction for controlling the propeller and the leg joints output by the second bottom controller is weighted by a2 and then output to the propeller and the leg joints.

5. The method of claim 1 or 4, wherein, The switching of the motion mode of the robot according to the environmental features comprises the following steps: When the SAC network judges that the environment is the ground environment according to the environmental features e, the weight a1 of the first bottom controller is greater than a2; When the SAC network judges that the environment is the underwater environment according to the environmental features e, the weight a2 of the second bottom controller is greater than a1; When the SAC network judges that the environment is the water-land transition area according to the environmental features e, the weights of the first bottom controller and the second bottom controller are obtained through the SAC network.

6. The motion mode switching system for an amphibious quadruped robot based on reinforcement learning, characterized in that, It comprises: a bottom controller unit for constructing two bottom controllers for controlling the robot actuators; an environmental state prediction unit for predicting the environmental features and the future body state through historical trajectory data, and dynamically weighting the outputs of the two bottom controllers to realize the switching of the motion mode.

7. The reinforcement learning based amphibious quadruped robot motion mode switching system according to claim 6, wherein, The environmental state prediction unit comprises: a neural network unit for predicting the environmental features and the future body state through historical trajectory data; The SAC network is used to control the output of two bottom controllers by dynamic weighting, comprising: combining the current ontology state information O according to the hidden state h in the predicted environment information t The SAC network is used to generate the weights a1 and a2 of different skills, and the composite control instructions of the robot leg joints and thrusters are sent to switch the motion mode of the robot according to the environmental characteristics.

Images