Four-legged robot control method and system based on transformer reinforcement learning architecture

Abstract

The application discloses a quadruped robot control method and system based on a Transformer reinforcement learning architecture. The method is related to the technical field of robot intelligent control. The method realizes real-time acquisition of the body state and environmental information of a quadruped robot through a multi-modal sensor. The information at the current and historical time is constructed into a time sequence state sequence. Feature embedding and time position coding are performed on the sequence to form an input token sequence. The token sequence is input into a policy network pre-trained through reinforcement learning, with a Transformer encoder as the core. The network utilizes a self-attention mechanism to capture long-range global dependency relationships between states and outputs continuous action instructions at the joint level. Finally, the corresponding movements of the robot are executed through a bottom controller to drive each joint of the robot. The system comprises corresponding sensing, computing and executing modules. The application effectively solves the gradient disappearance and memory decay problems of traditional recurrent neural networks in long sequence decision-making, and significantly improves the motion intelligence, robustness and energy efficiency of the quadruped robot in complex and unstructured environments.

Description

Technical Field

[0001] This invention relates to the field of robot control and artificial intelligence, specifically to a quadruped robot control method and system based on the Transformer reinforcement learning architecture. Background Technology

[0002] As is well known, quadruped robots mimic the locomotion of quadruped animals in nature. Compared to wheeled or tracked robots, they have a stronger ability to adapt to terrain and have broad application prospects in complex scenarios such as disaster relief, field exploration, and logistics distribution. The core technological challenge lies in achieving stable, agile, energy-efficient, and autonomous motion control.

[0003] Existing control schemes are mainly divided into two categories: model-based control and learning-based control. Model-based control, such as model predictive control (MPC), relies on accurate robot dynamics models and environment models. These methods perform well when the model is accurate and the environment is known, but they are very sensitive to model errors, external disturbances, and unknown environments, and lack robustness. Learning-based control, especially deep reinforcement learning (DRL), allows robots to autonomously learn strategies through interaction with the environment, without the need for accurate prior models, and shows great potential in handling uncertainties and complex environments. However, most current mainstream DRL policy networks use multilayer perceptrons (MLPs) or recurrent neural networks (RNNs / LSTMs). MLPs cannot effectively utilize historical information, and their decisions lack foresight. Although RNNs / LSTMs can handle sequences, they suffer from the vanishing / exploding gradient problem, and their serial computation method makes it difficult to capture the dependencies between distant states in long sequences, limiting their performance in quadrupedal locomotion tasks that require long-range planning. Summary of the Invention

[0004] The purpose of this invention is to provide a quadruped robot control method and system based on the Transformer reinforcement learning architecture, so as to solve the problems of poor robustness of model-based methods and the difficulty of traditional neural networks in effectively capturing long-range temporal dependencies in learning-based methods, thereby improving the motion performance of quadruped robots in complex environments.

[0005] The technical solution adopted in this invention is: a quadruped robot control method based on the Transformer reinforcement learning architecture, comprising the following steps:

[0006] S1. The sensor module acquires the body state information and environmental perception information of the quadruped robot in real time.

[0007] S2. Construct a temporal state sequence by combining the ontology state information and environmental perception information acquired at the current moment and the previous N consecutive historical moments.

[0008] S3. Feature embedding is performed on each frame of state data in the temporal state sequence obtained in step S2, and position encoding is introduced to inject temporal information, which together form the input token sequence.

[0009] S4. Input the input token sequence obtained in step S3 into the pre-trained Transformer reinforcement learning policy model; the Transformer reinforcement learning policy model uses its internal self-attention mechanism as the core to calculate the global dependency relationship between all elements in the sequence and output the target motion vector of each joint of the quadruped robot at the current moment.

[0010] S5. Convert the target motion vector obtained in step S4 into motor drive signals, and control each joint of the quadruped robot to perform corresponding movements through the underlying controller.

[0011] As a preferred embodiment, in step S1, the body state information includes at least the three-dimensional attitude, angular velocity, and linear acceleration of the fuselage from the inertial measurement unit, as well as the joint angles and joint velocities from the joint encoder; the environmental perception information includes compressed visual feature vectors obtained by feature extraction from RGB images and depth images from the visual sensor via a convolutional neural network.

[0012] As a preferred embodiment, in step S2, the time-series state sequence also incorporates instruction information for indicating the robot's motion intention, the instruction information including at least the desired forward speed, lateral speed, and rotational speed about the vertical axis.

[0013] As a preferred embodiment, in step S3, the feature embedding is to uniformly map state data with different physical meanings and dimensions to a high-dimensional embedding space through independent linear projection layers; the position encoding adopts fixed position encoding or trainable position encoding based on sine and cosine functions to explicitly represent the absolute or relative time position of each state in the sequence.

[0014] As a preferred embodiment, in step S4, the Transformer reinforcement learning policy model uses a multi-layer Transformer encoder as its core architecture, and its output is connected to an action head network. The action head network is a multi-layer perceptron, which is used to map the context vector output by the encoder into continuous action values ​​in the joint space. The action values ​​represent torque or target angle.

[0015] As a preferred approach, the training process of the Transformer reinforcement learning policy model in step S4 is as follows:

[0016] S4.1 Environment Initialization: Create a parallel simulation environment and initialize the network structure;

[0017] S4.2 Data Collection Loop: Run the policy network simultaneously in multiple simulation environments to collect state-action-reward experience data;

[0018] S4.3 Network Update: When the specified batch size is reached, update the network parameters using the PPO algorithm;

[0019] S4.4 Domain randomization: Randomly changing environmental parameters to enhance the model's generalization ability;

[0020] S4.5 Convergence Judgment: Continue training until the model performance reaches the convergence criterion, then save the final model.

[0021] This solution also relates to a quadruped robot control system based on the Transformer reinforcement learning architecture, comprising: a sensing module for acquiring the quadruped robot's body state information and environmental perception information; a computing processing module electrically connected to the sensing module for performing the following operations: receiving and buffering real-time data from the sensing module; constructing a temporal state sequence; performing feature embedding and position encoding to generate an input token sequence; loading and running a pre-trained Transformer reinforcement learning policy model, performing forward inference, and outputting joint target motion commands; and an execution module connected to the computing processing module for converting the joint target motion commands into motor drive signals to drive each joint of the quadruped robot to perform corresponding movements.

[0022] As a preferred embodiment, in this system, the computing module is integrated into the onboard embedded computer of the quadruped robot for low-latency online real-time inference; the training process of the Transformer reinforcement learning policy model is completed in a high-fidelity simulation environment on an offline server cluster with powerful computing capabilities, and the trained model parameters are then deployed to the onboard computer.

[0023] This solution also relates to a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the quadruped robot control method based on the Transformer reinforcement learning architecture described above.

[0024] This solution also relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the quadruped robot control method based on the Transformer reinforcement learning architecture described above.

[0025] The beneficial effects of this invention are:

[0026] This invention innovatively introduces the Transformer architecture into the underlying motion control of quadruped robots, achieving the following technical effects:

[0027] Firstly, it has excellent long-range dependency modeling capabilities: by utilizing the self-attention mechanism, it directly models the global relationship between any two states in a sequence, overcoming the bottleneck of RNNs and making decision-making more forward-looking.

[0028] Secondly, it has greater environmental adaptability and robustness: based on richer contextual information, it can better cope with unknown terrain, slippage and external disturbances.

[0029] Third, smoother and more energy-efficient motion performance: the learned strategies are more coordinated, reducing jitter and unnecessary energy consumption.

[0030] Fourth, efficient multimodal fusion: Transformer is naturally suitable for fusing heterogeneous information such as proprioception and vision.

[0031] Fifth, high training efficiency: Parallel computing significantly accelerates the training process. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a diagram showing the overall hardware and information flow architecture of the robot control system in this invention;

[0034] Figure 2 This is a detailed structural diagram of the Transformer policy network described in an embodiment of the present invention;

[0035] Figure 3 This is a flowchart illustrating the overall control method described in this invention.

[0036] Figure 4 This is a detailed schematic diagram illustrating the construction and embedding encoding process of the temporal state sequence in an embodiment of the present invention;

[0037] Figure 5 This is a flowchart illustrating the training of a Transformer policy model in a simulation environment, as described in this embodiment of the invention.

[0038] Figure 6 This is a simplified model and joint definition diagram of the quadruped robot used in the embodiments of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail through exemplary embodiments and in conjunction with the accompanying drawings.

[0040] It should be noted that, unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains; the words "a," "an," or "the" and similar terms used in the patent application specification and claims of this invention do not express a quantity limitation, but rather indicate the presence of at least one; the words "comprising" or "including" and similar terms indicate that the elements or objects preceding "comprising" or "including" cover the elements or objects listed following "comprising" or "including" and their equivalents, but do not exclude other elements or objects having the same function;

[0041] A quadruped robot control system based on the Transformer reinforcement learning architecture is disclosed. The control system consists of three core modules: a sensing module, a computing and processing module, and an execution module. The sensing module is used to collect the quadruped robot's body state information and environmental perception information. The computing and processing module receives the real-time data from the sensing module and outputs joint target action commands. The execution module receives the joint target action commands and converts them into motor drive signals to drive the quadruped robot's joints to perform corresponding movements. The entire system forms a complete closed loop from perception to decision-making to execution.

[0042] like Figure 1 As shown, the sensing module consists of an IMU sensor, a joint encoder, and an RGB-D camera, which are responsible for collecting robot body state information and environmental perception information. The IMU sensor provides body posture and motion data, the joint encoder measures the angle and speed of each joint, and the RGB-D camera extracts environmental visual features through a CNN network.

[0043] The computational processing module includes a state sequence builder, a feature embedding layer, a position encoder, and a Transformer policy network. This module receives sensor data, constructs a temporal state sequence, performs feature transformation and position encoding, and finally outputs joint motion commands through inference via the Transformer network.

[0044] The execution module includes a low-level motion controller, motor drivers, and joint motors. It converts the motion commands output by the calculation and processing module into motor drive signals to control the quadruped robot to complete the corresponding actions.

[0045] In this system, the computing module is integrated into the onboard embedded computer of the quadruped robot for low-latency online real-time inference; the training process of the Transformer reinforcement learning policy model is completed in a high-fidelity simulation environment on an offline server cluster with powerful computing capabilities, and the trained model parameters are then deployed to the onboard computer.

[0046] A quadruped robot control method based on the Transformer reinforcement learning architecture includes the following steps:

[0047] S1. Acquire the quadruped robot's body state information and environmental perception information in real time through the sensing module; S2. Construct a temporal state sequence by arranging the acquired body state information and environmental perception information from the current moment and the previous N consecutive historical moments in chronological order; S3. Embed features into each frame of state data in the temporal state sequence obtained in step S2 and introduce position encoding to inject temporal information, forming an input token sequence; S4. Input the input token sequence obtained in step S3 into a pre-trained Transformer reinforcement learning policy model; the Transformer reinforcement learning policy model uses its internal self-attention mechanism as the core to calculate the global dependencies between all elements in the sequence and output the target motion vectors of each joint of the quadruped robot at the current moment; S5. Convert the target motion vectors obtained in step S4 into motor drive signals and control each joint of the quadruped robot to perform corresponding movements through the underlying controller.

[0048] It should be noted that: in step S1, the body state information includes at least the three-dimensional attitude, angular velocity, and linear acceleration of the fuselage from the inertial measurement unit, as well as the joint angles and joint velocities from the joint encoder. The above body state information is collected by the IMU sensor and the joint encoder. The environmental perception information includes compressed visual feature vectors obtained by feature extraction from RGB images and depth images from the visual sensor through a convolutional neural network. The above environmental perception information is collected by the RGB-D camera.

[0049] In this plan, Figure 2 This section details the internal structure of the Transformer policy network, which consists of the following five parts:

[0050] Input layer: Receives a time-series state sequence, containing state data for N consecutive time steps;

[0051] Embedding layer: Maps the state vector to a high-dimensional embedding space through linear projection;

[0052] Location encoding: Adding location information to each time step enables the network to identify positions in the sequence.

[0053] The temporal relationship;

[0054] Transformer encoder stack: consists of multiple encoder layers, each containing self-attention.

[0055] Force mechanisms and feedforward networks are used to capture long-range dependencies in sequences;

[0056] Output Heads: Includes the Actor Head MLP and the Critic Head. The Actor Head MLP generates joint motion vectors, and the Critic Head MLP (used during training) outputs state value estimates.

[0057] In this part, the network receives the temporal state sequence as input, projects the state vector into a high-dimensional space through the embedding layer, and adds it with the position code to form the input token. The token sequence is processed by a multi-layer Transformer encoder. Each encoder layer contains a self-attention mechanism and a feedforward network. The output of the encoder is fed into the Actor head MLP and the Critic head MLP, respectively. This structure can effectively capture global dependencies in long sequences.

[0058] It should be noted that in step S2, the time-series state sequence also incorporates instruction information for indicating the robot's movement intention, which includes at least the desired forward speed, lateral speed, and rotational speed around the vertical axis.

[0059] In this plan, Figure 3 The complete flow of the Transformer-based quadruped robot control method is demonstrated, and the specific flow is as follows:

[0060] Initialization phase: The system starts up and loads the pre-trained model parameters;

[0061] Control loop: Read sensor data in real time, perform data preprocessing and feature extraction;

[0062] Sequence construction: Combining the current state with historical states into a time sequence;

[0063] Network inference: Through feature embedding, position encoding, and Transformer network processing, it outputs action instructions;

[0064] Action execution: Convert the actions output from the network into motor control signals to drive the robot's movement;

[0065] Loop control: Determines whether to continue the control loop or terminate the system operation.

[0066] It should be noted that in step S3, the feature embedding is to uniformly map state data with different physical meanings and dimensions to a high-dimensional embedding space through independent linear projection layers; the position encoding adopts fixed position encoding or trainable position encoding based on sine and cosine functions to explicitly represent the absolute or relative time position of each state in the sequence.

[0067] In this plan, Figure 4The specific process of state sequence construction and embedding encoding is described in detail. In this process, multi-source sensor data (such as IMU data, joint encoder data, visual feature data, and control commands) are first concatenated into a single-frame state vector. The system maintains a fixed-length sequence buffer to store consecutive state frames in chronological order, forming a temporal state sequence. This sequence is projected into a high-dimensional space through a linear embedding layer and added to the position encoding vector to finally generate the input token sequence required by the Transformer network.

[0068] In step S4, the Transformer reinforcement learning policy model uses a multi-layer Transformer encoder as its core architecture, and its output is connected to an action head network. The action head network is a multi-layer perceptron, which is used to map the context vector output by the encoder into continuous action values ​​in the joint space. The action values ​​represent torque or target angle.

[0069] In this plan, Figure 5 The training process of the Transformer policy model in a simulation environment is demonstrated. The training process is as follows:

[0070] S4.1 Environment Initialization: Initialize the simulation environment and build the network structure;

[0071] S4.2 Data Collection Loop: Run the policy network simultaneously in multiple simulation environments to collect state-action-reward experience data;

[0072] S4.3 Network Update: When the specified batch size is reached, update the network parameters using the PPO algorithm;

[0073] S4.4 Domain randomization: Randomly changing environmental parameters to enhance the model's generalization ability;

[0074] S4.5 Convergence Judgment: Continue training until the model performance reaches the convergence criterion, then save the final model.

[0075] In this plan, Figure 6 A simplified model and joint definitions of a quadruped robot are shown. The robot has four legs, each with three active joints: hip abduction / adduction (LF-HAA), hip flexion / extension (LF-HFE), and knee flexion / extension (LF-KFE). These joints together constitute a 12-DOF motion system. The figure also clarifies the order of the 12-dimensional motion vectors, which correspond to the motion commands controlling the 12 joints.

[0076] This solution also relates to a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the quadruped robot control method based on the Transformer reinforcement learning architecture described above.

[0077] This solution also relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the quadruped robot control method based on the Transformer reinforcement learning architecture described above.

[0078] Example 1

[0079] A control system and control method for a quadruped robot based on the Transformer reinforcement learning architecture; detailed explanation with specific implementation process:

[0080] 1. Hardware Configuration

[0081] Robot platform: Based on an open-source 12-DOF unit point quadruped robot, each leg includes three active joints: hip abduction (LF-HAA), hip flexion-extension (LF-HFE), and knee flexion-extension (LF-KFE).

[0082] Computing Unit: NVIDIA Jetson AGX Xavier is used as the onboard main controller, responsible for running ROS nodes and neural network inference.

[0083] sensor:

[0084] IMU: BMI088, provides three-axis acceleration and three-axis angular velocity, and estimates the fuselage attitude through filters.

[0085] Joint encoder: Uses an absolute encoder to directly read the angles of each joint.

[0086] Visual sensor: Intel RealSense D435i, providing RGB and depth images at a resolution of 640x480.

[0087] 2. Software Implementation

[0088] Operating system: Ubuntu 20.04 LTS with ROS Noetic.

[0089] Simulation environment: NVIDIA Isaac Gym. Leveraging its massively parallel simulation capabilities, 4096 robot instances were trained simultaneously.

[0090] Deep learning framework: PyTorch 1.12. Model training was performed on a server equipped with four NVIDIA RTX 4090 GPUs.

[0091] Reinforcement learning algorithm: Proximal Policy Optimization (PPO), using the ray library for distributed training.

[0092] 3. Detailed Implementation of Control Methods

[0093] S1: Data Acquisition and Preprocessing

[0094] Body state: IMU data (500Hz) and joint encoder data (500Hz) are published via ROS topics; the state vector includes: body attitude (quaternion, 4D), angular velocity (3D), linear velocity (3D), gravity vector (3D), 12 joint angles (12D), and 12 joint velocities (12D), for a total of 38 dimensions; the linear velocity is integrated from the IMU acceleration and corrected by the kinematic observer;

[0095] Visual information: RGB and depth images (30Hz) are input into a lightweight ResNet-18 network (pre-trained on ImageNet and fine-tuned for navigation tasks) to extract a 512-dimensional feature vector, which is further compressed to 256 dimensions through a linear layer, and the visual feature frequency is increased to 500Hz through interpolation to align with the ontology state.

[0096] Command: A 3-dimensional vector, representing the desired forward velocity (x), lateral velocity (y), and yaw rate.

[0097] S2: Construction of Temporal State Sequences

[0098] The sequence length N is set to 50, corresponding to 100ms of historical information (control frequency 500Hz).

[0099] The single-frame state s_i is 38 (body) + 256 (visual) + 3 (instruction) = 297 dimensions;

[0100] The temporal state sequence S_t is a 50x297 matrix.

[0101] S3: State Embedding and Position Encoding

[0102] A linear layer is used to project the 297-dimensional state vector into an embedding space of d_model=256 dimensions;

[0103] The sine-cosine position coding proposed by Vaswani et al. is adopted;

[0104] The embedding vector is added to the position vector to form a 50x256-dimensional input token sequence.

[0105] S4: Transformer Policy Network Inference

[0106] Network structure:

[0107] Transformer encoder layer number L=6;

[0108] Self-attention head count h=8;

[0109] The hidden layer dimension of the feedforward network is d_ff=1024;

[0110] Activation function: GELU;

[0111] Dropout rate = 0.1 (during training);

[0112] Inference process: The features of the last token output by the encoder (corresponding to time t) are taken and fed into the Actor head. The Actor head is a two-layer MLP (256->128->12), which outputs the target torque of 12 joints. During the deployment phase, the mean is directly output; during the training phase, samples are taken from a Gaussian distribution centered at the mean with a learnable standard deviation.

[0113] S5: Action Execution

[0114] The output target torque is sent directly to the motor as a command (based on current control mode). Meanwhile, a lightweight PD controller operates at a higher frequency (2kHz) to stabilize the joint position, but its target position is indirectly affected by the torque output by the RL strategy through integration.

[0115] 4. Training Process Details

[0116] Reward functions: A carefully designed multi-objective reward function, including: linear velocity tracking reward, angular velocity tracking reward, orientation reward, survival reward, motion smoothness penalty, joint acceleration penalty, foot slip penalty, collision penalty, etc.

[0117] Domain randomization: The following parameters are randomized for each simulation environment: robot leg mass (±15%), body mass (±10%), ground friction coefficient (0.4~1.2), motor strength (±20%), sensor delay (0~5ms), zero bias and white noise of IMU and joint encoder.

[0118] Training parameters: PPO algorithm parameters: learning rate 3e-4, discount factor γ=0.99, GAE parameter λ=0.95, pruning coefficient ε=0.2, value function coefficient=0.5, entropy coefficient=0.01.

[0119] Training period: A total of approximately 200 million environmental steps were trained (approximately 12 hours).

[0120] 5. Testing and Results: The present invention (Transformer-Based) was compared with [other technologies] on both simulated and physical robots.

[0121] MLP baseline, LSTM baseline controller.

[0122] Passability in complex terrain: On randomly generated staircases, gravel roads, and sloping terrain, the passability of this invention (98%) is significantly higher than that of MLP (85%) and LSTM (90%).

[0123] Disturbance resistance: When a lateral thrust is applied on flat ground, the control system described in this invention takes the shortest time to regain balance and never falls over, while the baseline controller has about a 10% probability of falling over.

[0124] Energy efficiency: When tracking a circular trajectory at the same speed, the total power consumption of the control system described in this invention is reduced by an average of about 15%.

[0125] The parts not described in detail in the above embodiments are existing technologies.

[0126] It should be noted that although the present invention has been described through the above embodiments, the present invention may have many other embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art can obviously make various corresponding changes and modifications to the present invention, but all such changes and modifications should fall within the scope of protection of the appended claims and their equivalents.

Claims

1. A quadruped robot control method based on a Transformer reinforcement learning architecture, characterized in that: The method comprises the following steps: S1, acquiring the body state information and the environment perception information of the quadruped robot in real time through a sensing module; S2, constructing the acquired body state information and environment perception information at the current time and the previous N continuous historical times into a time sequence state sequence in time sequence; the length N of the time sequence state sequence is 50, wherein each frame of state data of the time sequence state sequence comprises a data part composed of the body state information, a data part composed of the environment perception information, and an instruction data part; the time sequence state sequence also fuses instruction information for indicating the motion intention of the robot, and the instruction information at least comprises a desired forward speed, a lateral speed, and a rotation speed around a vertical axis; S3, performing feature embedding on each frame of state data in the time sequence state sequence obtained in step S2, and introducing position encoding to inject time sequence information, to form an input token sequence together; the position encoding adopts fixed position encoding based on a sine-cosine function, and is used to explicitly represent the absolute or relative time position of each state in the sequence; S4, inputting the input token sequence obtained in step S3 into a pre-trained Transformer reinforcement learning strategy model; the Transformer reinforcement learning strategy model takes the self-attention mechanism in it as the core, the number of self-attention mechanism heads is 8, the global dependency relationship between all elements in the sequence is calculated, and a target action vector of each joint of the quadruped robot at the current time is output; in this step, the Transformer reinforcement learning strategy model takes a multi-layer Transformer encoder as the core architecture, the number of layers of the Transformer encoder is 6, an action head network is connected to the output end of the Transformer encoder, the action head network is a multi-layer perceptron, and is used to map the context vector output by the encoder into continuous action values in the joint space, the action values are represented as torques; the training process of the Transformer reinforcement learning strategy model is as follows: S4.1, environment initialization: creating a parallel simulation environment and initializing the network structure; S4.2, data collection cycle: simultaneously running the strategy network in multiple simulation environments, and collecting state-action-reward experience data; S4.3, network updating: when the specified batch size is reached, updating the network parameters using the PPO algorithm; S44, domain randomization: randomly changing the environment parameters to enhance the generalization ability of the model; S4.5, convergence judgment: continuously training until the performance of the model reaches the convergence standard, the training period is a total of 200 million environment steps, and the final model is saved; S5, converting the target action vector obtained in step S4 into a motor driving signal, and controlling each joint of the quadruped robot to perform corresponding motion through a bottom layer controller.

2. The quadruped robot control method based on the Transformer reinforcement learning architecture according to claim 1, characterized in that: In step S1, the body state information at least comprises a three-dimensional attitude, an angular velocity, and a linear acceleration of the fuselage from an inertial measurement unit, and joint angles and joint speeds of each joint from a joint encoder; the environment perception information comprises compressed visual feature vectors obtained after convolutional neural network feature extraction on RGB images and depth images from a visual sensor.

3. The quadruped robot control method based on the Transformer reinforcement learning architecture according to claim 1, characterized in that: In step S3, the feature embedding is unified mapping of state data with different physical meanings and dimensions to a high-dimensional embedding space through an independent linear projection layer; the position encoding adopts fixed position encoding or trainable position encoding based on sine and cosine functions, for explicitly representing the absolute or relative time position of each state in the sequence.

4. A quadruped robot control system based on a Transformer reinforcement learning architecture for implementing the method of any one of claims 1-3. The method comprises the steps of: A sensing module is configured to collect body state information and environment perception information of the quadruped robot. A computing processing module is electrically connected to the sensing module and configured to perform the following operations: receiving and buffering real-time data of the sensing module; constructing a time sequence state sequence; performing feature embedding and position encoding to generate an input token sequence; loading and running a pre-trained Transformer reinforcement learning strategy model to perform forward inference and output joint target action instructions; and a performing module connected to the computing processing module is configured to convert the joint target action instructions into motor driving signals to drive the quadruped robot to perform corresponding movements.

5. The system of claim 4, wherein: The computing processing module is integrated in an on-board embedded computer of the quadruped robot for low-delay online real-time inference; and the training process of the Transformer reinforcement learning strategy model is completed in a high-fidelity simulation environment on an offline server cluster with powerful computing power, and the trained model parameters are deployed to the on-board computer.

6. A computing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor executes the program to implement the steps of the method of any one of claims 1-3.

7. A computer readable storage medium having stored thereon a computer program, characterized in that: The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images