A visual end-to-end quadruped robot control method based on a reinforcement learning method
By employing a visual end-to-end reinforcement learning approach, the encoder is trained using variational autoencoders and proximal policy optimization algorithms. Low-dimensional latent vectors are extracted from noisy images to generate joint motion commands. This solves the problems of traditional methods' dependence on accurate dynamic models and high requirements for perception computing power, thereby improving the autonomous adaptability and control efficiency of quadruped robots in unknown environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU YUNSHENCHU TECH CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional quadruped robot control methods rely on precise dynamic models, which lack generalization ability and robustness, making it difficult to achieve stable and efficient motion control in unknown environments. Deep reinforcement learning methods have high requirements for the accuracy of perceptual information modeling and computing power, and are difficult to adapt to dynamic obstacles.
A vision-based end-to-end reinforcement learning control method is constructed. By training an encoder in a simulation environment to extract low-dimensional latent vectors from noisy images and generate joint motion commands, the reliance on high-precision environment modeling is reduced. An end-to-end control model is generated by joint training using a variational autoencoder and a proximal policy optimization algorithm.
It reduces the requirements for perception computing power, improves the autonomous adaptability and behavioral intelligence of quadruped robots in unknown terrain, and achieves stable and efficient motion control.
Smart Images
Figure CN121492072B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics technology, and more specifically, to a vision-based end-to-end quadruped robot control method based on reinforcement learning. Background Technology
[0002] Currently, quadruped robots occupy an important position in the field of robot control technology, mainly due to their significant advantages over wheeled and tracked platforms. Their biomimetic leg structure provides excellent static and dynamic stability, strong adaptability to complex terrain, and high load potential. These characteristics enable quadruped robots to maintain high mobility in unstructured, rugged, and varied field or industrial environments, thus demonstrating enormous application prospects in a wide range of scenarios such as disaster site search and rescue, geological exploration, facility inspection, warehousing and logistics transportation, and even special operations. They are considered a key development direction for breaking through the environmental limitations of traditional mobile platforms.
[0003] However, achieving high-performance motion control for quadruped robots faces fundamental challenges. Their control systems need to handle high-dimensional, continuous motion spaces and cope with the system's own highly nonlinear dynamic characteristics and uncertainties introduced by external environmental disturbances. Traditional control methods relying on precise, analytical dynamic mathematical models and pre-set trajectories (such as model predictive control or zero-moment point programming) often exhibit limited generalization ability and robustness when faced with these complex factors, making it difficult to achieve stable and efficient motion in unknown and variable environments.
[0004] Currently, deep reinforcement learning methods have become the mainstream research direction in this field. However, when encountering dynamic obstacles or scenarios that cannot be adapted through trial and error (such as crossing deep ditches), deep reinforcement learning controllers still need to receive accurate external perception information as input. Therefore, they rely heavily on the perception algorithm's ability to accurately model the environment, which remains a challenge for quadruped robots with limited computing power that need to adapt to unknown scenarios. Summary of the Invention
[0005] The main purpose of this application is to provide a vision-based end-to-end quadruped robot control method based on reinforcement learning, so as to solve the problem of coordination between perception and motion control algorithms.
[0006] To achieve the above objectives, firstly, this application proposes a vision-based end-to-end quadruped robot control method based on reinforcement learning, comprising:
[0007] In a simulation environment, a model robot is built to obtain robot simulation data;
[0008] By simulating the noise in the robot simulation data, a historical sequence of noise depth data is obtained;
[0009] A visual feature extraction network is used to extract features from the historical sequence of the noise depth data to obtain a low-dimensional visual feature vector.
[0010] The robot simulation data and the low-dimensional visual feature vector are encoded using an encoder, and then decoded using a decoder to reconstruct the noisy data of the target robot. The noisy data of the target robot is then used together with the noiseless data of the target robot to train the visual feature extraction network, encoder, and decoder to obtain the target visual feature extraction network and the target encoder.
[0011] The noisy data is used as the input to the first network, and the noise-free data is used as the input to the second network. The first network and the second network are jointly trained to obtain the target first network.
[0012] The target visual feature extraction network, the target encoder, and the target first network are combined to generate a visual and motion end-to-end control model for joint movements.
[0013] In some feasible methods, the step of constructing a model robot in a simulation environment and obtaining robot simulation data includes:
[0014] In the simulation environment, a model is built to obtain a model robot. The data of the model robot includes a URDF or USD description file, which records the dimensions, mass, moment of inertia and relative position of each component of the robot, as well as the position, attitude and viewing angle information of the depth camera relative to the robot.
[0015] The motion of the model robot is simulated to obtain the robot simulation data, which includes the current state observation value of the model robot, the current depth map observation value, the state observation value sequence and depth map observation value sequence of the past H time moments, the noise-free historical state sequence, the noise-free historical depth map sequence, and the noise-free terrain data.
[0016] In some feasible methods, the step of simulating noise in the robot simulation data to obtain a historical sequence of noise depth data includes:
[0017] In the simulation environment, one or more depth vision sensors are constructed, and the effective sensing distance of the current depth map observation value and the depth map observation value sequence of the past H time moments in the robot simulation data is limited to obtain an initial depth image sequence.
[0018] The initial depth image sequence is blurred using a Gaussian blur model to obtain a smooth depth image sequence;
[0019] Gaussian noise is added to the smoothed depth image sequence using a Gaussian noise model to obtain a Gaussian noise depth image sequence.
[0020] Using the salt-and-pepper noise model, salt-and-pepper noise is applied to the Gaussian noise depth image sequence to simulate holes and outliers in a real sensor, thus obtaining the historical sequence of the noise depth data.
[0021] In some feasible methods, the step of using a visual feature extraction network to extract features from the historical sequence of the noisy depth data to obtain a low-dimensional visual feature vector includes:
[0022] The historical sequence of the noisy depth data is stored as a high-dimensional matrix, and the high-dimensional matrix is processed using the visual feature extraction network M to obtain a depth feature sequence, wherein the depth feature sequence is represented as follows: :
[0023] ;
[0024] ;
[0025] in, This represents the historical sequence of noise depth data at time t, including the past H times. This indicates a reshaping operation, which will... Rearrange them to form a regular four-dimensional tensor. This represents the four-dimensional tensor obtained after reshaping, where 1 indicates the batch size, processing one sample sequence at a time. Indicates the length of the time series, corresponding to the number of depth image frames at a given historical moment. This represents the height of a single frame of depth image. The width of a single frame of depth image is represented by M; M represents the visual feature extraction network, which consists of H neural networks with identical structures but non-shared parameters. Each visual feature extraction network processes one frame of depth image independently. This represents the deep feature sequence after processing by M;
[0026] The deep feature sequence is flattened into a one-dimensional vector, and then passed through a linear output layer of a neural network. Features are extracted to obtain a low-dimensional visual feature vector, wherein the low-dimensional visual feature vector is represented as follows: ,in, The calculation formula is:
[0027] ;
[0028] in, This represents a high-dimensional matrix, where 1 indicates the batch size, processing one sample sequence at a time. The number of channels in the feature map represents the different types of features extracted. This represents the height of a single frame of depth image. Indicates the width of a single frame of depth image; This means flattening multidimensional, spatially structured data into a one-dimensional long vector; This represents the linear output layer of a neural network. This represents a low-dimensional visual feature vector.
[0029] In some feasible methods, the steps of encoding the robot simulation data and the low-dimensional visual feature vector using an encoder, then decoding them using a decoder to reconstruct the noisy data of the target robot, and training the visual feature extraction network, encoder, and decoder together with the noise-free data of the target robot to obtain the target visual feature extraction network and the target encoder include:
[0030] The low-dimensional visual feature vector and the sequence of state observations from the past H time steps are used as input, and compressed using an encoder of a variational autoencoder network to form a latent vector, wherein the latent vector is represented as... The calculation formula is:
[0031] ;
[0032] in, The latent vector representing the depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state; An encoder representing a variational autoencoder. This represents the sequence of state observations over the past H time points. Represents a low-dimensional visual feature vector;
[0033] The latent vectors are decoded using a variational autoencoder network to reconstruct the current depth map data and linear velocity data of the target robot, as well as predict the robot's state at the next moment. The calculation formula is as follows:
[0034] ;
[0035] in, This represents the decoder of the variational autoencoder. This represents the depth image reconstructed by the decoder at the current moment. This represents the linear velocity estimated by the decoder at the current moment. This represents the robot's state predicted by the decoder at the next moment;
[0036] Using the reconstructed current depth map data and linear velocity data, the predicted robot state at the next moment, and noise-free data of the target robot's current depth map data, linear velocity data, and robot state at the next moment obtained from the simulation environment as input, the visual feature extraction network and the variational autoencoder network are trained. The total loss function of the variational autoencoder is calculated, and the formula for calculating the total loss function is as follows:
[0037] ;
[0038] in, Represents the total loss function. The expression represents the relationship between the latent space distribution and the prior distribution. Divergence loss, This represents the mean squared error loss function between the reconstructed depth image and the true depth image. This represents the mean square error loss function between the estimated linear velocity and the true linear velocity. The mean squared error loss function represents the difference between the robot's predicted state and the actual state; MSE represents the mean squared error function.
[0039] ;
[0040] in, This indicates the calculation of the relationship between two probability distributions. divergence, This represents an approximate posterior probability distribution. Represents the prior probability distribution; The latent vector representing the robot's state. This represents the sequence of state observations of the robot over the past H time steps at time t. This represents the sequence of depth map observations of the robot over the past H time points;
[0041] ;
[0042] ;
[0043] ;
[0044] By minimizing the total loss function, the visual feature extraction network, the encoder, and the decoder are jointly trained using a supervised learning method to obtain a trained target visual feature extraction network and a target encoder network. The target visual feature extraction network includes a visual feature extraction network M and a linear output layer of a neural network. .
[0045] In some feasible methods, the step of jointly training the first network and the second network to obtain the target first network by using the noisy data as input to the first network and the noise-free data as input to the second network includes:
[0046] The latent vectors and the current state observations of the target robot obtained from the simulation environment are used as inputs to a first network. The output of this first network yields the action commands that each joint of the target robot should execute. The calculation formula is:
[0047] ;
[0048] in, This represents the joint motion vector generated at time t. Represents the actor network model, This represents the current state observation of the target robot. The latent vector representing the depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state;
[0049] Noise-free historical state sequences, noise-free historical depth map sequences, and noise-free terrain data obtained from the simulation environment are used as common inputs to a second network. This second network processes the data to obtain a state value function. The calculation formula is:
[0050] ;
[0051] in, This represents the state value assessment result at time t. This represents the second network model. Represents a noise-free sequence of historical states. This represents a noise-free historical depth map sequence. Represents noise-free terrain data;
[0052] The PPO reinforcement learning algorithm is optimized using a proximal strategy to maximize the cumulative reward obtained by the robot from the environment. The first network and the second network are jointly trained to optimize the first network and obtain the target first network.
[0053] In some feasible embodiments, the step of combining the target visual feature extraction network, the target encoder, and the target first network to generate a visual and motion end-to-end control model for joint movements includes:
[0054] According to the preset input-output execution relationship, the target visual feature extraction network is connected to the input head of the target encoder to establish a first connection relationship;
[0055] According to the preset input-output execution relationship, the output head of the target encoder is connected to the input head of the target first network to establish a second connection relationship;
[0056] Based on the first connection relationship and the second connection relationship, the target visual feature extraction network, the target encoder network and the target first network are combined to obtain the vision and motion end-to-end control model.
[0057] Secondly, this application provides a vision-based end-to-end quadruped robot control system based on reinforcement learning, applied to the aforementioned vision-based end-to-end quadruped robot control method based on reinforcement learning, including...
[0058] The building block is used to build model robots in a simulation environment and obtain robot simulation data.
[0059] The simulation unit is also used to simulate noise in the robot simulation data to obtain a historical sequence of noise depth data;
[0060] The feature extraction unit is used to extract features from the historical sequence of the noisy depth data using a visual feature extraction network to obtain a low-dimensional visual feature vector.
[0061] The processing unit is used to encode the robot simulation data and the low-dimensional visual feature vector using an encoder, and then decode them using a decoder to reconstruct the noisy data of the target robot. The noisy data of the target robot is then used together with the noiseless data of the target robot to train the visual feature extraction network, the encoder and the decoder to obtain the target visual feature extraction network and the target encoder.
[0062] The training unit is used to take the noisy data as input to the first network and the noise-free data as input to the second network, and jointly train the first network and the second network to obtain the target first network;
[0063] The result unit is used to combine the target visual feature extraction network, the target encoder and the target first network to generate a visual and motion end-to-end control model for joint movements.
[0064] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned visual end-to-end quadruped robot control method based on reinforcement learning.
[0065] Fourthly, this application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned visual end-to-end quadruped robot control method based on reinforcement learning.
[0066] The technical solutions provided by the embodiments of this application may include the following beneficial effects:
[0067] This application presents a vision-based end-to-end quadruped robot control method based on reinforcement learning. By constructing an end-to-end control model from raw visual input to joint movement commands, it effectively addresses the dual challenges of traditional control methods' strong dependence on precise dynamic models and weak generalization ability, as well as the high requirements of existing deep reinforcement learning methods for the modeling accuracy and computational power of external perception algorithms. This application utilizes an encoder trained in simulation to extract low-dimensional latent vectors related to environmental interaction from noisy images, enabling the robot to understand terrain features and generate corresponding actions based on visual information. This eliminates the reliance on independent, high-precision environmental modeling modules, significantly reducing the computational power required for perception and endowing the quadruped robot with stronger autonomous adaptability and behavioral intelligence in unknown, unstructured terrain. Attached Figure Description
[0068] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:
[0069] Figure 1 A flowchart illustrating a vision-based end-to-end quadruped robot control method based on reinforcement learning, provided in this application;
[0070] Figure 2 This application provides an end-to-end model inference flowchart for a vision-based end-to-end quadruped robot control method based on reinforcement learning. Detailed Implementation
[0071] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0073] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.
[0074] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.
[0075] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0076] The following explanations of the terms used in this application are provided to aid in understanding the technical solution presented herein:
[0077] Gaussian blur is an image filtering technique that uses a Gaussian function to smooth images. Its core principle is to use a normally distributed convolution kernel to perform a weighted average of each pixel in the image, thereby smoothing the image and reducing details and noise.
[0078] Gaussian noise is a type of random noise whose statistical properties follow a normal distribution. In an image, it manifests as a Gaussian distribution adding a random value to the brightness value of each pixel, based on the original value.
[0079] Salt and pepper noise, also known as impulse noise or shot noise, is a type of noise that appears as random white and black dots in an image.
[0080] The visual feature extraction network is an encoder module used to process spatiotemporal sequence data. Its core function is to map the high-dimensional, redundant, and noisy raw visual input sequence into a low-dimensional, dense feature representation for use in downstream tasks such as state reconstruction and action generation.
[0081] The linear output layer L of the neural network is a fully connected layer following the visual feature extraction network. Its role is to complete the final transformation and shaping from feature maps to feature vectors. It performs an affine transformation (linear weighted summation plus bias) on the flattened features, projecting them onto a specified low-dimensional space. The linear output layer L of the neural network and the visual feature extraction network are two parts of a coherent model, together forming a complete visual encoder.
[0082] The Actor-Critic framework is a core architecture in reinforcement learning that combines policy learning (Actor) and value function learning (Critic). The Actor network is responsible for executing actions, directly outputting the action the agent (in this case, a quadruped robot) should take based on the current environmental state. The Critic network, instead of directly selecting actions, evaluates the long-term value of the current state (or state-action pair).
[0083] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train Actor-Critic models. It is a policy gradient method that, due to its simplicity, stability, and efficiency, has become a benchmark algorithm in the field of deep reinforcement learning, aiming to find policies that maximize long-term cumulative rewards.
[0084] like Figure 1 and Figure 2 As shown, in a first aspect, this application provides a vision-based end-to-end quadruped robot control method based on reinforcement learning, comprising:
[0085] S100: In the simulation environment, a model robot is built to obtain robot simulation data.
[0086] Specifically, obtaining robot simulation data may include the following steps:
[0087] S101, In the simulation environment, a model is built to obtain a model robot. The data of the model robot includes a description file of URDF or USD. The description file records the dimensions, mass, moment of inertia and relative position of each component of the robot, as well as the position, attitude and viewing angle information of the depth camera relative to the robot.
[0088] Specifically, in the simulation environment, a model robot is built based on physical and configuration parameters. This involves creating and loading a URDF (Uniform Robot Description Format) or USD (Universal Scenario Description) file containing the robot's complete physical properties. This file defines the dimensions, mass, moment of inertia, relative connection positions, and motion constraints of each component of the model robot (such as the torso, legs, and joints). This results in a digital model robot with physical characteristics that can operate in the simulation environment. Furthermore, the physical parameters of the model robot (such as mass and moment of inertia) are set according to the design or approximation principles of a real robot to ensure the matching of simulated dynamics with the real robot.
[0089] S102, Simulate the motion of the model robot to obtain the robot simulation data.
[0090] The robot simulation data includes the current state observations of the robot formed by the model, the current depth map observations, the sequence of state observations and depth map observations over the past H time points, the noise-free historical state sequence, the noise-free historical depth map sequence, and the noise-free terrain data.
[0091] Specifically, the model robot is driven to move within a simulation environment, during which multiple types of data are simultaneously collected from the simulation system interface. These include:
[0092] Read the current state observation value of the model robot. (e.g., joint angles, fuselage attitude) and the sequence of state observations over the past H time points. These data simulate real sensor readings and may include noise introduced by the simulation environment.
[0093] Read the current depth map observations from the simulation environment. and the sequence of depth map observations over the past H time points These data simulate the raw output of a real depth camera. H is a preset positive integer representing the number of historical time steps considered.
[0094] Obtain a noise-free historical state sequence from the simulation environment. Noise-free historical depth map sequence and noiseless terrain data of the robot's location. (Such as precise ground elevation grids). These data are free from any noise model contamination and represent the true state of the environment.
[0095] It should be noted that the current depth map observation value and the depth map observation value sequence are directly simulated in the simulation environment using step S100.
[0096] S200, simulate the noise in the robot simulation data to obtain a historical sequence of noise depth data.
[0097] Specifically, obtaining the historical sequence of noise depth data may include the following steps:
[0098] S201, in the simulation environment, one or more depth vision sensors are constructed, and the effective sensing distance of the current depth map observation value and the depth map observation value sequence of the past H time moments in the robot simulation data is limited to obtain an initial depth image sequence.
[0099] Specifically, after obtaining the current depth map observation value and the depth map observation value sequence from step S100, the current depth map observation value and the depth map observation value sequence are used as raw depth image data. An effective sensing distance is limited for these raw depth image data. The specific principle is that depth cameras (single or multiple depth vision sensors, the specific number can be set as needed, and this application does not limit it) have minimum and maximum effective ranging ranges in the real world; readings outside these ranges are unreliable. Therefore, the depth values of all pixels are limited to a preset effective range, and values outside the range are set to invalid values (such as 0 or the maximum value). For example, this effective distance range is defined as d∈[0.1,3.0] meters, indicating that only environmental data within a range of 0.1 meters to 3.0 meters from the robot is processed. This results in an initial depth image after distance clipping.
[0100] S202, using a Gaussian blur model, the initial depth image sequence is blurred to suppress the jagged effect caused by insufficient simulation resolution, resulting in a smooth depth image sequence.
[0101] Specifically, Gaussian blurring is applied to each frame of the initial depth image. The principle is that the simulated depth image, due to pixel discretization, exhibits a jagged, uneven effect at object edges. By applying a Gaussian blur kernel (a mathematically weighted average filter) from a Gaussian blur model to the initial depth image through convolution, the initial depth image can be smoothed, the jagged effect suppressed, and the blurring effect more closely resembles the natural blurring effect produced by the lens characteristics of a real optical camera.
[0102] Therefore, by using a Gaussian blur model to blur the initial depth image, the jagged effect caused by insufficient simulation resolution is suppressed, thereby obtaining a smooth depth image.
[0103] S203, Gaussian noise is added to the smooth depth image sequence using a Gaussian noise model to obtain a Gaussian noise depth image sequence.
[0104] Specifically, Gaussian noise is added to each frame of the smoothed depth image sequence. The principle is to simulate the random electrical noise of a real depth camera sensor circuit. This random electrical noise statistically follows a Gaussian distribution (i.e., a normal distribution). Therefore, a random perturbation is superimposed on the depth value of each pixel, and the amount of this perturbation is given by the formula:
[0105] ;
[0106] in, This represents the standard normal distribution, also known as the Gaussian distribution. The two numbers (0,1) in parentheses are the parameters of this distribution. This represents the scaling factor, which can be adjusted as needed. The random number generated by the distribution is represented by the perturbation amount, which is expressed by the formula as the depth value of each pixel in the depth image. A random noise perturbation is superimposed on this perturbation value, which is a random number sampled from a normal distribution with a mean of 0 and a standard deviation of 0.06 meters. This results in a depth image with Gaussian noise.
[0107] S204. Using the salt-and-pepper noise model, salt-and-pepper noise is applied to the Gaussian noise depth image sequence to simulate holes and outliers in a real sensor, thereby obtaining the historical sequence of noise depth data.
[0108] Specifically, salt-and-pepper noise is applied to the Gaussian noise depth image sequence. The principle is to simulate data anomalies generated by a real depth camera when encountering light-absorbing surfaces, highly reflective surfaces, or exceeding its measurement range. These anomalies manifest as random bright spots (salt noise, representing invalid large values or infinity) and dark spots (pepper noise, representing invalid small values or holes) in the Gaussian noise depth image. A certain proportion of pixels in the Gaussian noise depth image are randomly selected and their values are set to the maximum (salt noise) or minimum (pepper noise) value. An exemplary formula for the proportion of bright spots (salt noise) and dark spots (pepper noise) to the total number of pixels is as follows:
[0109] ;
[0110] in, This represents the probability that, when salt-and-pepper noise is added, any specific pixel in a Gaussian noise depth image will be randomly selected and set to an extremely bright value (representing an invalid, infinitely distant measurement or a positive noise peak). This value is typically set to the maximum effective value of the Gaussian noise depth image or a very large number. This represents the probability that any particular pixel in the Gaussian noise depth image will be randomly selected and set to an extremely dark value (representing an invalid, near-field measurement failure, hole, or negative noise peak) when salt-and-pepper noise is added. This value is usually set to 0 or a very small number, with 0.01 representing the probability value.
[0111] This means that when adding salt-and-pepper noise, each pixel in the Gaussian noise depth image has a 1% probability of becoming an extremely bright outlier (salt noise) and a 1% probability of becoming an extremely dark outlier (pepper noise). This parameter is key to controlling the sparsity and intensity of salt-and-pepper noise.
[0112] S300: Using a visual feature extraction network, features are extracted from the historical sequence of the noise depth data to obtain a low-dimensional visual feature vector.
[0113] Specifically, obtaining the low-dimensional visual feature vector may include the following steps:
[0114] S301, the historical sequence of the noisy depth data is stored as a high-dimensional matrix, and the high-dimensional matrix is processed using the visual feature extraction network M to obtain a depth feature sequence, wherein the depth feature sequence is represented as... :
[0115] ;
[0116] ;
[0117] in, This represents the historical sequence of noise depth data at time t, including the past H times. This indicates a reshaping operation, which will... Rearrange them to form a regular four-dimensional tensor. This represents the four-dimensional tensor obtained after reshaping, where 1 indicates the batch size, processing one sample sequence at a time. Indicates the length of the time series, corresponding to the number of depth image frames at a given historical moment. This represents the height of a single frame of depth image. The width of a single frame of depth image is represented by M; M represents the visual feature extraction network, which consists of H convolutional neural networks with identical structures but non-shared parameters. Each convolutional neural network independently processes one frame of depth image. This represents the deep feature sequence after processing by M.
[0118] Specifically, the noisy depth data history sequence obtained in step S200 represents the depth image sequence containing the past H time steps at time t. Next, a reshaping operation is performed on the noisy depth data history sequence, rearranging the depth image sequence of the H time steps into a four-dimensional tensor. Furthermore, the principle is to stack H temporally continuous but storage-discrete depth images in the time dimension and add a dimension to represent the batch size, thereby forming a four-dimensional data block. The purpose is to convert it into a tensor format that is processed by the standard deep learning framework, in order to prepare for subsequent batch processing and temporal dimension analysis.
[0119] It should be noted that the reshaped four-dimensional tensor is input into the visual feature extraction network M for processing. This visual feature extraction network M consists of H convolutional neural networks with identical structures but no parameter sharing. The goal is to allow each sub-convolutional neural network to independently process one frame of image, specifically responsible for extracting spatial visual features (such as edges, textures, and shapes) from that frame. Due to the lack of parameter sharing, each convolutional neural network can focus more on learning the unique features of its corresponding frame, thereby better capturing the detailed information of each frame in the sequence, without forcing all frames to pass through the same feature extractor, in order to preserve rich information in the temporal dimension. Next, the visual feature extraction network M compresses the high-resolution pixel data into a low-resolution feature map with a high number of channels (N) that is rich in semantic information through operations such as convolution and pooling. The purpose of this compression is to discard redundant details at the pixel level and extract visual features with strong semantic information (such as edges, textures, and abstract shapes of objects).
[0120] S302, the deep feature sequence is flattened into a one-dimensional vector, and then passed through a linear output layer of a neural network. Features are extracted to obtain a low-dimensional visual feature vector, wherein the low-dimensional visual feature vector is represented as follows: ,in, The calculation formula is:
[0121] ;
[0122] in, This represents a high-dimensional matrix, where 1 indicates the batch size, processing one sample sequence at a time. The number of channels in the feature map represents the different types of features extracted. This represents the height of a single frame of depth image. Indicates the width of a single frame of depth image; This means flattening multidimensional, spatially structured data into a one-dimensional long vector; This represents the linear output layer of a neural network. This represents a low-dimensional visual feature vector.
[0123] Specifically, the feature sequence with spatial structure and channel dimension (N) is flattened into a one-dimensional long vector. This step is a structural transformation, not information compression. Next, the one-dimensional vector is input into the linear output layer L to achieve dimensionality compression, the purpose of which is to project the high-dimensional vector into a low-dimensional vector space of a specified size. The purpose of this compression is to refine and aggregate the feature information to obtain a compact representation of a fixed size that can summarize the core information of the entire sequence for use in subsequent steps.
[0124] It should be noted that the spatially structured sequence is flattened, that is, all the values in the four-dimensional tensor are rearranged sequentially into a one-dimensional continuous vector. The purpose of this operation is to convert the extracted two-dimensional spatial feature map into a one-dimensional vector. The one-dimensional feature vector is then input into a linear output layer L of a neural network. This linear output layer L is a fully connected layer, and its function is to linearly transform the high-dimensional input vector into a lower-dimensional vector space through matrix operations.
[0125] S400: The robot simulation data and the low-dimensional visual feature vector are encoded using an encoder, and then decoded using a decoder to reconstruct the noise data of the target robot. The noise data of the target robot is then used together with the noise-free data of the target robot to train the visual feature extraction network, encoder, and decoder to obtain the target visual feature extraction network and the target encoder.
[0126] The hidden vectors include depth map data, linear velocity data, and robot state.
[0127] Specifically, obtaining the target visual feature extraction network and the target encoder may include the following steps:
[0128] S401, the low-dimensional visual feature vector and the sequence of state observations over the past H time steps are used as input, and compressed using the encoder of a variational autoencoder network to form a latent vector.
[0129] Wherein, the hidden vector is represented as The calculation formula is:
[0130] ;
[0131] in, The latent vector representing the depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state; An encoder representing a variational autoencoder. This represents the sequence of state observations over the past H time points. This represents a low-dimensional visual feature vector.
[0132] Specifically, the low-dimensional visual feature vector obtained from step S300 and the sequence of state observations over the past H time points. The system performs multimodal information fusion and then participates in subsequent training to obtain a trained target visual feature extraction network and target encoder network.
[0133] For example, low-dimensional visual feature vectors and sequences of state observations are concatenated to provide the encoder with comprehensive joint environmental and ontology perception information. The encoder in a variational autoencoder learns and outputs parameters (typically mean and variance) of a probability distribution, assuming the latent vector... It follows a posterior distribution determined by the input data. By sampling from this distribution using reparameterization techniques, specific latent vector values are obtained. The aim is to learn a structured, continuous latent space, ensuring that similar inputs are located close together in the latent space, thus improving the model's generalization and generative capabilities. Next, three independent latent vectors are designed. The principle behind this is that feature decoupling can separate components with different physical meanings (depth vision, linear velocity, and robot state) in the input information and store them in different latent variables, thereby improving the accuracy of reconstruction and prediction and enhancing the interpretability of the model. This results in three sets of independent latent vectors: depth map latent vector, linear velocity latent vector, and robot state latent vector.
[0134] S402, using the decoder of the variational autoencoder network, the latent vector is decoded to obtain the current depth map data and linear velocity data of the target robot, as well as the predicted robot state at the next moment. The calculation formula is as follows:
[0135] ;
[0136] in, This represents the decoder of the variational autoencoder. This represents the depth image reconstructed by the decoder at the current moment. This represents the linear velocity estimated by the decoder at the current moment. This represents the robot's state predicted by the decoder at the next moment.
[0137] Specifically, the decoder utilizes three independent sub-networks (or output heads), each focusing on reconstructing or predicting the robot's state at the next time step from its corresponding latent vectors. For example, the depth map reconstruction branch focuses on how to... The depth image is recovered without being affected by velocity or state information. This leads to the reconstruction of the current depth map data and linear velocity data, as well as the prediction of the robot's state at the next moment.
[0138] S403, using the current depth map data and linear velocity data, the predicted robot state at the next moment, and the noise-free data of the target robot's current depth map data, linear velocity data, and robot state at the next moment obtained from the simulation environment as input, the visual feature extraction network and the variational autoencoder network are trained, and the total loss function of the variational autoencoder is calculated to obtain the total loss function, wherein the formula for calculating the total loss function is:
[0139] ;
[0140] in, Represents the total loss function. The expression represents the relationship between the latent space distribution and the prior distribution. Divergence loss, This represents the mean squared error loss function between the reconstructed depth image and the true depth image. This represents the mean square error loss function between the estimated linear velocity and the true linear velocity. The mean squared error loss function represents the difference between the robot's predicted state and its actual state. MSE stands for mean squared error function.
[0141] ;
[0142] in, This indicates the calculation of the relationship between two probability distributions. divergence, This represents an approximate posterior probability distribution. Represents the prior probability distribution; The latent vector representing the robot's state. This represents the sequence of state observations of the robot over the past H time steps at time t. This represents the sequence of depth map observations of the robot over the past H time points.
[0143] ;
[0144] ;
[0145] .
[0146] Specifically, this is achieved through joint optimization of multi-objective loss functions, simultaneously constraining reconstruction accuracy and latent space structure. Reconstruction Loss , and Using mean square error To minimize the difference between the model output and the noise-free data, and to ensure that the model learns accurate reconstruction and prediction capabilities, noise-free data refers to noise-free historical state sequences, noise-free historical depth map sequences, and noise-free terrain data, etc.
[0147] For example, the total loss function of the variational autoencoder is calculated. This total loss function is a weighted sum of four independent loss terms, specifically including:
[0148] KL divergence loss: used to constrain the distribution of robot states and state observation sequences to be close to a preset prior distribution (such as a standard normal distribution) to ensure that the latent space has good structure and regularity.
[0149] Depth image reconstruction loss: Calculates the mean squared error loss function between the reconstructed depth image and the true depth image.
[0150] Linear velocity estimation loss: Calculate the mean square error loss function between the estimated linear velocity and the true linear velocity.
[0151] State prediction loss: Calculate the mean squared error loss function between the predicted state and the actual state of the target robot.
[0152] S404, by minimizing the total loss function, the visual feature extraction network, the encoder, and the decoder are jointly trained using a supervised learning method to obtain a trained target visual feature extraction network and a target encoder network.
[0153] The target visual feature extraction network includes a visual feature extraction network M and a linear output layer of a neural network. .
[0154] Specifically, using supervised learning and gradient descent algorithms, and aiming to minimize the total loss function, an end-to-end joint training process is performed on the visual feature extraction network, the encoder of the variational autoencoder network, and the decoder of the variational autoencoder network. Training continues until the model's performance on the validation set converges and stabilizes. Upon completion of training, the target visual feature extraction network and the target encoder network are obtained.
[0155] S500, the noisy data is used as the input to the first network, and the noise-free data is used as the input to the second network. The first network and the second network are jointly trained to obtain the target first network.
[0156] Specifically, obtaining the target first network may include the following steps:
[0157] S501, the latent vector and the current state observation value of the target robot at the current moment obtained from the simulation environment are used as the input of the first network, and the first network is used to output to obtain the action instructions that each joint of the target robot should execute.
[0158] The action instructions that each joint should execute are as follows. The calculation formula is:
[0159] ;
[0160] in, This represents the joint motion vector generated at time t. Represents the actor network model, This represents the current state observation of the target robot. The latent vector representing the depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state.
[0161] Specifically, the first network serves as the policy function; for example, the first network is an actor network. The input it receives contains two types of information: one representing environmental perception. and and the current state observation value representing the robot's own state at the current moment. The first network integrates and makes decisions on these two types of information through the learned mapping relationship, thereby generating reasonable actions that match the environment, that is, the action instructions that each joint of the target robot should execute.
[0162] S502, noise-free historical state sequences, noise-free historical depth map sequences, and noise-free terrain data obtained from the simulation environment are used as common inputs to the second network. The second network processes these data to obtain the state value function. The state value function... The calculation formula is:
[0163] ;
[0164] in, This represents the state value assessment result at time t. This represents the second network model. Represents a noise-free sequence of historical states. This represents a noise-free historical depth map sequence. This represents noise-free terrain data.
[0165] Specifically, accurate value guidance is achieved using real-world information. A second network, serving as the value function (e.g., a Critic network), takes noise-free environmental information as input. During training, the Actor network is provided with more accurate and stable learning signals. By using real-world environmental information to evaluate the value of a state, the interference of perceptual noise on value estimation can be avoided, thus more effectively guiding the policy optimization direction of the Actor network. This leads to the state value function, which is used to evaluate the expected long-term cumulative reward under the current state and policy.
[0166] S503, the PPO reinforcement learning algorithm is optimized using a proximal strategy to maximize the cumulative reward obtained by the robot from the environment. The first network and the second network are jointly trained to optimize the first network and obtain the target first network.
[0167] Specifically, the Proximal Policy Optimization (PPO) reinforcement learning algorithm is used to find the policy that maximizes long-term cumulative reward. That is, the PPO reinforcement learning algorithm is used to jointly train the first and second networks. By limiting the maximum magnitude of the difference between the old and new policies, each policy update is ensured to be smooth and gradual, thereby enhancing the stability of the training process and avoiding training crashes. The second network continuously learns to more accurately estimate state values. The second network provides directional guidance for the policy updates of the first network; the first network, in turn, optimizes its policy based on the guidance of the second network to select actions that yield higher expected rewards. This results in a target first network that has completed training and converged. This target first network can generate robust motion control commands that adapt to complex terrain and aim to maximize long-term rewards based on real-time environmental awareness information (latent vectors) and ontology state.
[0168] S600, the target visual feature extraction network, the target encoder and the target first network are combined to generate a visual and motion end-to-end control model for joint movements.
[0169] Wherein, the current network and the next network represent the target visual feature extraction network, the target encoder, and the target first network.
[0170] Specifically, generating a visual and motion end-to-end control model for joint movements may include the following steps:
[0171] S601, according to the preset input-output execution relationship, the target visual feature extraction network is connected to the input head of the target encoder to establish a first connection relationship.
[0172] S602, according to the preset input-output execution relationship, connect the output head of the target encoder to the input head of the target first network to establish a second connection relationship.
[0173] S603, based on the first connection relationship and the second connection relationship, the target visual feature extraction network, the target encoder network and the target first network are combined to obtain the vision and motion end-to-end control model.
[0174] It should be noted that a directional data flow is established from the target visual feature extraction network to feature encoding. The output of the target visual feature extraction network is connected to the input of the target encoder, forming a stable data transmission channel and establishing the first connection relationship to ensure that visual features are accurately transmitted to the encoder. Next, a directional data flow is established from feature encoding to action decision-making. The output of the target encoder is connected to the input of the first network, ensuring the complete transmission of latent vector information and establishing the second connection relationship, forming a data path from the encoder to the actor network.
[0175] Based on two connection relationships, three independently trained network components are integrated into a unified system. This results in a complete vision-motion end-to-end control model with end-to-end processing capabilities from raw visual perception to joint motion control.
[0176] In summary, this application provides a vision-based end-to-end quadruped robot control method based on reinforcement learning, representing a process of directly mapping raw depth camera data input to the joint movements of the model robot. The model robot trained in this way can avoid the modeling accuracy risks that may exist with traditional perception algorithms, enabling the model robot to actively adjust its movement patterns to adapt to complex and rugged terrain, complete tasks such as automatically identifying and crossing deep ditches, and endow the robot with the ability to traverse dangerous terrain.
[0177] Secondly, this application provides a vision-based end-to-end quadruped robot control system based on reinforcement learning, applied to the aforementioned vision-based end-to-end quadruped robot control method based on reinforcement learning, including...
[0178] The building block is used to build model robots in a simulation environment and obtain robot simulation data.
[0179] The simulation unit is also used to simulate noise in the robot simulation data to obtain a historical sequence of noise depth data;
[0180] The feature extraction unit is used to extract features from the historical sequence of the noisy depth data using a visual feature extraction network to obtain a low-dimensional visual feature vector.
[0181] The processing unit is used to encode the robot simulation data and the low-dimensional visual feature vector using an encoder, and then decode them using a decoder to reconstruct the noisy data of the target robot. The noisy data of the target robot is then used together with the noiseless data of the target robot to train the visual feature extraction network, the encoder and the decoder to obtain the target visual feature extraction network and the target encoder.
[0182] The training unit is used to take the noisy data as input to the first network and the noise-free data as input to the second network, and jointly train the first network and the second network to obtain the target first network;
[0183] The result unit is used to combine the target visual feature extraction network, the target encoder and the target first network to generate a visual and motion end-to-end control model for joint movements.
[0184] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned visual end-to-end quadruped robot control method based on reinforcement learning.
[0185] Fourthly, this application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned visual end-to-end quadruped robot control method based on reinforcement learning.
[0186] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0187] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0188] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A vision-based end-to-end quadruped robot control method based on reinforcement learning, characterized in that, include: In a simulation environment, a model robot is built to obtain robot simulation data; By simulating the noise in the robot simulation data, a historical sequence of noise depth data is obtained; A visual feature extraction network is used to extract features from the historical sequence of the noise depth data to obtain a low-dimensional visual feature vector. The robot simulation data and the low-dimensional visual feature vector are encoded using an encoder, and then decoded using a decoder to reconstruct the noisy data of the target robot. The noisy data of the target robot is then used together with the noiseless data of the target robot to train the visual feature extraction network, encoder, and decoder to obtain the target visual feature extraction network and the target encoder. The noisy data is used as the input to the first network, and the noise-free data is used as the input to the second network. The first network and the second network are jointly trained to obtain the target first network. The target visual feature extraction network, the target encoder, and the target first network are combined to generate a visual and motion end-to-end control model for joint movements.
2. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 1, characterized in that, The steps of constructing a model robot and obtaining robot simulation data in a simulation environment include: In the simulation environment, a model is built to obtain a model robot. The data of the model robot includes a URDF or USD description file, which records the dimensions, mass, moment of inertia and relative position of each component of the robot, as well as the position, attitude and viewing angle information of the depth camera relative to the robot. The motion of the model robot is simulated to obtain the robot simulation data, which includes the current state observation value of the model robot, the current depth map observation value, the state observation value sequence and depth map observation value sequence of the past H time moments, the noise-free historical state sequence, the noise-free historical depth map sequence, and the noise-free terrain data.
3. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 1, characterized in that, The step of simulating the noise in the robot simulation data to obtain a historical sequence of noise depth data includes: In the simulation environment, one or more depth vision sensors are constructed, and the effective sensing distance of the current depth map observation value and the depth map observation value sequence of the past H time moments in the robot simulation data is limited to obtain an initial depth image sequence. The initial depth image sequence is blurred using a Gaussian blur model to obtain a smooth depth image sequence; Gaussian noise is added to the smoothed depth image sequence using a Gaussian noise model to obtain a Gaussian noise depth image sequence. Using the salt-and-pepper noise model, salt-and-pepper noise is applied to the Gaussian noise depth image sequence to simulate holes and outliers in a real sensor, thus obtaining the historical sequence of the noise depth data.
4. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 1, characterized in that, The step of using a visual feature extraction network to extract features from the historical sequence of noisy depth data to obtain a low-dimensional visual feature vector includes: The historical sequence of the noisy depth data is stored as a high-dimensional matrix, and the high-dimensional matrix is processed using the visual feature extraction network M to obtain a depth feature sequence, wherein the depth feature sequence is represented as follows: : ； ； in, This represents the historical sequence of noise depth data at time t, including the past H times. This indicates a reshaping operation, which will... Rearrange them to form a regular four-dimensional tensor. This represents the four-dimensional tensor obtained after reshaping, where 1 indicates the batch size, processing one sample sequence at a time. Indicates the length of the time series, corresponding to the number of depth image frames at a given historical moment. This represents the height of a single frame of depth image. The width of a single frame of depth image is represented by M; M represents the visual feature extraction network, which consists of H neural networks with identical structures but non-shared parameters. Each visual feature extraction network processes one frame of depth image independently. This represents the deep feature sequence after processing by M; The deep feature sequence is flattened into a one-dimensional vector, and then passed through a linear output layer of a neural network. Features are extracted to obtain a low-dimensional visual feature vector, wherein the low-dimensional visual feature vector is represented as follows: ,in, The calculation formula is: ； in, This represents a high-dimensional matrix, where 1 indicates the batch size, processing one sample sequence at a time. The number of channels in the feature map represents the different types of features extracted. This represents the height of a single frame of depth image. Indicates the width of a single frame of depth image; This means flattening multidimensional, spatially structured data into a one-dimensional long vector; This represents the linear output layer of a neural network. This represents a low-dimensional visual feature vector.
5. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 4, characterized in that, The steps of encoding the robot simulation data and the low-dimensional visual feature vector using an encoder, decoding them using a decoder to reconstruct the noisy data of the target robot, and training the visual feature extraction network, encoder, and decoder together with the noise-free data of the target robot to obtain the target visual feature extraction network and the target encoder include: The low-dimensional visual feature vector and the sequence of state observations from the past H time steps are used as input, and compressed using an encoder of a variational autoencoder network to form a latent vector, wherein the latent vector is represented as... The calculation formula is: ； in, The latent vector representing depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state; An encoder representing a variational autoencoder. This represents the sequence of state observations over the past H time points. Represents a low-dimensional visual feature vector; The latent vectors are decoded using a variational autoencoder network to reconstruct the current depth map data and linear velocity data of the target robot, as well as predict the robot's state at the next moment. The calculation formula is as follows: ； in, This represents the decoder of the variational autoencoder. This represents the depth image reconstructed by the decoder at the current moment. This represents the linear velocity estimated by the decoder at the current moment. This represents the robot's state predicted by the decoder at the next moment; Using the current depth map data and linear velocity data, the predicted robot state at the next moment, and noise-free data of the target robot's current depth map data, linear velocity data, and robot state at the next moment obtained from the simulation environment as input, the visual feature extraction network and the variational autoencoder network are trained. The total loss function of the variational autoencoder is calculated, and the formula for calculating the total loss function is as follows: ； in, Represents the total loss function. The latent space distribution represents the relationship between the prior distribution and the latent space distribution. Divergence loss, This represents the mean squared error loss function between the reconstructed depth image and the true depth image. This represents the mean square error loss function between the estimated linear velocity and the true linear velocity. The mean squared error loss function represents the difference between the robot's predicted state and the actual state; MSE represents the mean squared error function. ； in, This indicates the calculation of the relationship between two probability distributions. divergence, This represents an approximate posterior probability distribution. Represents the prior probability distribution; The latent vector representing the robot's state. This represents the sequence of state observations of the robot over the past H time steps at time t. This represents the sequence of depth map observations of the robot over the past H time points; ； ； ； By minimizing the total loss function, the visual feature extraction network, the encoder, and the decoder are jointly trained using a supervised learning method to obtain a trained target visual feature extraction network and a target encoder network. The target visual feature extraction network includes a visual feature extraction network M and a linear output layer of a neural network. .
6. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 1, characterized in that, The step of using the noisy data as input to the first network and the noise-free data as input to the second network, and jointly training the first network and the second network to obtain the target first network, includes: The latent vectors and the current state observations of the target robot obtained from the simulation environment are used as inputs to a first network. The output of this first network yields the action commands that each joint of the target robot should execute. The calculation formula is: ； in, This represents the joint motion vector generated at time t. Represents the actor network model, This represents the current state observation of the target robot. The latent vector representing depth map data. The implicit vector representing linear velocity data. The implicit vector representing the robot's state; Noise-free historical state sequences, noise-free historical depth map sequences, and noise-free terrain data obtained from the simulation environment are used as common inputs to a second network. This second network processes the data to obtain a state value function. The calculation formula is: ； in, This represents the state value assessment result at time t. This represents the second network model. Represents a noise-free historical state sequence. This represents a noise-free historical depth map sequence. Represents noise-free terrain data; The PPO reinforcement learning algorithm is optimized using a proximal strategy to maximize the cumulative reward obtained by the robot from the environment. The first network and the second network are jointly trained to optimize the first network and obtain the target first network.
7. The vision-based end-to-end quadruped robot control method based on reinforcement learning as described in claim 1, characterized in that, The step of combining the target visual feature extraction network, the target encoder, and the target first network to generate a visual and motion end-to-end control model for joint movements includes: According to the preset input-output execution relationship, the target visual feature extraction network is connected to the input head of the target encoder to establish a first connection relationship; According to the preset input-output execution relationship, the output head of the target encoder is connected to the input head of the target first network to establish a second connection relationship; Based on the first connection relationship and the second connection relationship, the target visual feature extraction network, the target encoder network and the target first network are combined to obtain the vision and motion end-to-end control model.
8. A vision-based end-to-end quadruped robot control system based on reinforcement learning, characterized in that, The vision-based end-to-end quadruped robot control method based on reinforcement learning, as described in any one of claims 1-7, includes: The building block is used to build model robots in a simulation environment and obtain robot simulation data. The simulation unit is also used to simulate noise in the robot simulation data to obtain a historical sequence of noise depth data; The feature extraction unit is used to extract features from the historical sequence of the noisy depth data using a visual feature extraction network to obtain a low-dimensional visual feature vector. The processing unit is used to encode the robot simulation data and the low-dimensional visual feature vector using an encoder, and then decode them using a decoder to reconstruct the noisy data of the target robot. The noisy data of the target robot is then used together with the noiseless data of the target robot to train the visual feature extraction network, the encoder and the decoder to obtain the target visual feature extraction network and the target encoder. The training unit is used to take the noisy data as input to the first network and the noise-free data as input to the second network, and jointly train the first network and the second network to obtain the target first network; The result unit is used to combine the target visual feature extraction network, the target encoder and the target first network to generate a visual and motion end-to-end control model for joint movements.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the visual end-to-end quadruped robot control method based on reinforcement learning method as described in any one of claims 1 to 7.
10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the visual end-to-end quadruped robot control method based on reinforcement learning method as described in any one of claims 1 to 7.