Reinforced learning agent training method applied to quadruped robot

Abstract

The invention relates to the technical field of reinforcement learning, and discloses a reinforcement learning agent training method applied to a quadruped robot, and the method comprises the steps: enabling a reinforcement learning agent to start the training of this round, and obtaining the network parameters of an adversarial network structure of the last round; based on the adversarial network structure, obtaining environment interaction data obtained by interaction of the quadruped robot and various environment conditions in the current round of training; determining a cumulative reward based on the environmental interaction data; updating the network parameters of the adversarial network structure of the last round based on the accumulated rewards, obtaining the network parameters of the adversarial network structure of the training round, and completing the training round; the training process is repeated for multiple rounds of training, the target reinforcement learning agent is obtained, and the quadruped robot works by applying the target reinforcement learning agent. According to the method, the target reinforcement learning agent is trained by acquiring the environmental interaction data under various environmental conditions, so that the quadruped robot can normally work in a complex environment by applying the target reinforcement learning agent, and the environmental adaptability and robustness are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a reinforcement learning intelligent agent training method applied to a quadruped robot. Background Art

[0002] A quadruped robot is a bionic robot with four mechanical legs, each of which consists of multiple joints and can achieve complex gaits and movements. The torso of a quadruped robot is equipped with multiple sensors, such as IMU (Inertial Measurement Unit), LiDAR, and cameras, which are used to sense the environment and maintain balance. Although quadruped robots have a relatively stable structure and a variety of sensors, they still face many challenges when working in complex environments.

[0003] The ground conditions in complex environments vary greatly, such as uneven ground, soft sand, slippery surfaces, etc. This requires the quadruped robot to be able to adapt to different terrain features and adjust the foot joint torque. At the same time, dynamic and static obstacles in the environment may hinder the forward route of the quadruped robot, requiring the quadruped robot to have good obstacle avoidance and timely response capabilities. In addition, the sensors equipped with the quadruped robot may be interfered by noise, resulting in distorted or inaccurate data. Therefore, how to enable the quadruped robot to work normally in complex environments is a problem that needs to be solved. Summary of the invention

[0004] In view of this, the present invention provides a reinforcement learning agent training method applied to a quadruped robot to solve the problem that the quadruped robot can work normally in a complex environment.

[0005] In a first aspect, the present invention provides a reinforcement learning agent training method applied to a quadruped robot, the method comprising:

[0006] When the reinforcement learning agent of the quadruped robot starts this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are obtained. The adversarial network structure includes an adversarial network, a policy network, and an evaluation network.

[0007] Based on the adversarial network structure, multiple sets of environmental interaction data are obtained from the interaction between the quadruped robot and various environmental conditions in this round of training. Any set of environmental interaction data includes environmental state, action and reward.

[0008] Determine the cumulative reward for this round of training based on multiple sets of environmental interaction data;

[0009] Based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, thus completing this round of training;

[0010] The above training process is repeated for multiple rounds of training to obtain the target reinforcement learning agent of the quadruped robot, so that the quadruped robot can work using the target reinforcement learning agent.

[0011] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention integrates the structures of an adversarial network, a policy network, and an evaluation network, obtains environmental interaction data under various environmental conditions, and iteratively updates the network parameters of multiple networks based on the accumulated rewards. Multiple rounds of training are performed to obtain a target reinforcement learning agent, thereby improving the robustness of the target reinforcement learning agent. When the quadruped robot uses the target reinforcement learning agent to work, it can still work normally in a variety of complex environments, thereby enhancing its adaptability and robustness to complex environments.

[0012] In an optional implementation, based on the adversarial network structure, multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training are obtained, including:

[0013] When acquiring the first set of environmental interaction data, the initial environmental state is acquired based on multiple sensors of the quadruped robot;

[0014] Determine the adversarial noise corresponding to the initial environment state based on the adversarial network;

[0015] Based on the policy network, the corresponding action of the quadruped robot in the initial environment state is determined according to the initial environment state and the adversarial noise corresponding to the initial environment state;

[0016] Based on the evaluation network, when the quadruped robot takes an action in the initial environment state, it is rewarded;

[0017] The environmental state obtained by the interaction of the action with any environmental condition is used as the initial environmental state for the next set of environmental interaction data. The above process of obtaining the initial environmental state, action, and reward is repeated to obtain multiple sets of environmental interaction data for this round of training.

[0018] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention utilizes the sensors of the quadruped robot to capture environmental states under different environmental conditions, determines adversarial noise under the environmental states through an adversarial network, and determines the action to be taken by the quadruped robot based on the current environmental state and the adversarial noise through a policy network, determines a reward based on the result after the quadruped robot executes the action through an evaluation network, and repeats the above process to obtain multiple sets of environmental interaction data, thereby providing rich training samples for the training of reinforcement learning agents.

[0019] In an optional implementation, based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, including:

[0020] Use the adversarial network as the adversarial agent and the policy network as the ontological agent to construct a zero-sum game problem;

[0021] With the goal of solving the Nash equilibrium point of the zero-sum game problem, based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training.

[0022] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention constructs the adversarial network and the policy network into a zero-sum game problem, solves the Nash equilibrium point to update the network parameters, finds the optimal solution in the dynamic balance between the two agents, and improves the training efficiency of the reinforcement learning agent.

[0023] In an optional implementation, with the goal of solving the Nash equilibrium point of the zero-sum game problem, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, including:

[0024] The network parameters of the previous round of training of the policy network and the evaluation network are fixed. Based on the accumulated rewards of this round of training, the network parameters of the previous round of training of the adversarial network are gradient increased to obtain the network parameters of this round of training of the adversarial network.

[0025] The network parameters of the previous round of training of the adversarial network and the evaluation network are fixed. Based on the accumulated rewards of this round of training, the network parameters of the previous round of training of the policy network are gradient descended to obtain the network parameters of this round of training of the policy network.

[0026] The network parameters of the previous round of training of the adversarial network and the policy network are fixed, and the network parameters of the previous round of training of the evaluation network are updated to obtain the network parameters of the current round of training of the evaluation network.

[0027] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention updates the parameters of the adversarial network in a direction that can maximize the cost of the policy network through the gradient ascent of the adversarial network, and finds the optimal strategy in the presence of adversarial noise through the gradient descent of the policy network, that is, minimizing the long-term cost and maximizing the cumulative reward. By iterating the above process, the policy network gradually learns to make the best decision in an environment containing adversarial noise, and at the same time the adversarial network reaches its optimal interference strategy. The two tend to stabilize and form a Nash equilibrium point, thereby obtaining a reinforcement learning agent with good generalization ability and robustness.

[0028] In an optional implementation, the above training process is repeated for multiple rounds of training to obtain a target reinforcement learning agent of a quadruped robot, including:

[0029] Repeat the above training process. When the training rounds reach the preset rounds, the policy network in the adversarial network structure is used as the target reinforcement learning agent.

[0030] or,

[0031] Repeat the above training process, and when the network parameters of the adversarial network structure reach convergence, use the policy network in the adversarial network structure as the target reinforcement learning agent.

[0032] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention continuously iterates and optimizes the strategy network until preset conditions are met, so that the target reinforcement learning agent of the quadruped robot has high stability and good generalization ability, ensuring that the quadruped robot using the target reinforcement learning agent can work normally in a variety of complex environments.

[0033] In an optional implementation, the adversarial network includes a first input layer, a first hidden layer, and a first output layer, and determining the adversarial noise corresponding to the initial environment state based on the adversarial network includes:

[0034] Based on the first input layer of the adversarial network, an initial environment state is received, a vector dimension of the initial environment state is adjusted, and a data feature representation of the initial environment state is obtained;

[0035] Based on the first hidden layer of the adversarial network, a complex representation is constructed on the data feature representation of the initial environment state and nonlinearity is introduced;

[0036] Based on the first output layer of the adversarial network, the vector dimension is adjusted on the basis of the first hidden layer to output adversarial noise.

[0037] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention enables subsequent layers to more effectively understand and analyze environmental characteristics by adjusting the vector dimension of the first input layer of the adversarial network, laying the foundation for generating high-quality adversarial noise. The first hidden layer introduces nonlinearity to help capture nonlinear relationships in environmental states, so that the adversarial noise contains deeper environmental knowledge. The first output layer is responsible for adjusting the vector dimension and then outputting the final adversarial noise, providing support for the training of the reinforcement learning agent.

[0038] In an optional implementation, the strategy network includes a second input layer, a second hidden layer, and a second output layer. Based on the strategy network, according to the initial environment state and the adversarial noise corresponding to the initial environment state, the corresponding action of the quadruped robot in the initial environment state is determined, including:

[0039] Based on the second input layer of the policy network, the initial environment state and the adversarial noise are received, the vector dimensions of the initial environment state and the adversarial noise are adjusted, and the data feature representation of the initial environment state and the data feature representation of the adversarial noise are obtained;

[0040] Based on the second hidden layer of the policy network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the anti-noise and nonlinearity is introduced;

[0041] Based on the second output layer of the policy network, the vector dimension is adjusted based on the second hidden layer and the action is output.

[0042] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention receives and integrates the initial environment state and adversarial noise through the second input layer of the policy network, thereby increasing the complexity of the decision-making of the policy network. The second hidden layer constructs a complex representation of the information of the second input layer and introduces nonlinearity, thereby guiding the quadruped robot to perform the most appropriate actions in a given environment, and providing support for the training of the reinforcement learning agent.

[0043] In an optional embodiment, the evaluation network includes a third input layer, a third hidden layer and a third output layer. Based on the evaluation network, when the quadruped robot takes an action in the initial environment state, it is rewarded, including:

[0044] Based on the third input layer of the evaluation network, the initial environment state and the action are received, the vector dimensions of the initial environment state and the action are adjusted, and the data feature representation of the initial environment state and the data feature representation of the action are obtained;

[0045] Based on the third hidden layer of the evaluation network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the action and nonlinearity is introduced;

[0046] Based on the third output layer of the evaluation network, the vector dimension is adjusted based on the third hidden layer and the reward is output.

[0047] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention can evaluate the actions taken by the quadruped robot in a specific environmental state and determine the corresponding reward feedback by combining the environmental state and the execution action into an evaluation network, thereby providing support for the training of the reinforcement learning agent.

[0048] In a second aspect, the present invention provides a reinforcement learning agent training device applied to a quadruped robot, the device comprising:

[0049] The first acquisition module is used to obtain the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training when the reinforcement learning agent of the quadruped robot starts the current round of training. The adversarial network structure includes an adversarial network, a policy network and an evaluation network.

[0050] The second acquisition module is used to obtain multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training based on the adversarial network structure. Any set of environmental interaction data includes environmental state, action and reward;

[0051] A determination module is used to determine the cumulative reward of this round of training based on multiple sets of environment interaction data;

[0052] An updating module is used to update the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training based on the accumulated rewards of this round of training, obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, and complete this round of training;

[0053] The training module is used to repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and enable the quadruped robot to work using the target reinforcement learning agent.

[0054] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the reinforcement learning intelligent agent training method applied to a quadruped robot according to the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the reinforcement learning agent training method for a quadruped robot according to the first aspect or any corresponding embodiment thereof.

[0056] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the reinforcement learning agent training method for a quadruped robot according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 is a flow chart of a reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention;

[0059] Figure 2 is a flow chart of another reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of obtaining environmental interaction data according to an embodiment of the present invention;

[0061] Figure 4 is a flow chart of another reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention;

[0062] Figure 5 is a structural block diagram of a reinforcement learning agent training device applied to a quadruped robot according to an embodiment of the present invention;

[0063] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0065] Since quadruped robots face many challenges when working in complex environments, such as changing ground conditions, dynamic and static obstacles in the environment, and sensors that may be interfered by noise, resulting in distorted or inaccurate data. Therefore, how to enable quadruped robots to work normally in complex environments is a problem that needs to be solved. The reinforcement learning agent training method applied to quadruped robots provided in an embodiment of the present invention integrates the structures of adversarial networks, policy networks, and evaluation networks, obtains environmental interaction data under various environmental conditions, and conducts multiple rounds of training to obtain a target reinforcement learning agent, so that when the quadruped robot uses the target reinforcement learning agent to work, it can still work normally in a variety of complex environments, thereby enhancing its adaptability and robustness to complex environments.

[0066] According to an embodiment of the present invention, an embodiment of a reinforcement learning agent training method applied to a quadruped robot is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0067] In this embodiment, a reinforcement learning agent training method for a quadruped robot is provided, which can be used for a quadruped robot. Figure 1 is a flow chart of a reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0068] Step S101, when the reinforcement learning agent of the quadruped robot starts this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are obtained, and the adversarial network structure includes an adversarial network, a policy network, and an evaluation network. Specifically, since not all quadruped robots have built-in reinforcement learning agents, and even if the quadruped robot has a built-in reinforcement learning agent, it cannot meet the requirements of the normal operation of the quadruped robot in a complex environment, it is necessary to iteratively train the reinforcement learning agent of the quadruped robot, by obtaining the network parameters of the adversarial network structure of the reinforcement learning agent of the previous round of training, and continuously optimizing the network parameters of the adversarial network structure through training, thereby improving the robustness of the reinforcement learning agent in the face of complex environments.

[0069] Step S102, based on the adversarial network structure, obtain multiple groups of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training, and any group of environmental interaction data includes environmental state, action and reward. Specifically, when training the reinforcement learning agent of the quadruped robot, by setting multiple environmental conditions, different environments in actual work are simulated, such as rugged terrain, narrow space, unstable ground, etc., and the quadruped robot interacts with the set multiple environmental conditions, and subsequent reinforcement learning agent training is performed based on the environmental interaction data obtained by the interaction, so that the trained reinforcement learning agent improves its adaptability and generalization ability to complex environments. Optionally, the frequency of obtaining environmental interaction data can be to obtain a group for each time step, and there is no limit on the specific time length corresponding to a time step.

[0070] Step S103, based on multiple sets of environmental interaction data, determine the cumulative reward of this round of training. Specifically, the reward in any set of environmental interaction data includes indicators such as tracking instructions, body balance, energy consumption, and foot contact, which are used to reflect the quality of the actions in the environmental interaction data. Optionally, the cumulative reward of this round of training can be determined by the following formula (1). By accumulating the rewards of each set of environmental interaction data, the reinforcement learning agent learns the behavioral strategy of maximizing the cumulative reward in the long-term sequence, which prompts it to gradually master efficient actions under specific environmental conditions.

[0071]

[0072] Where π1 represents the adversarial network; π2 represents the policy network; i is 1 or 2. When the value is 1, π1 represents the adversarial network, and when the value is 2, π2 represents the policy network; f(π1,π2) represents the cumulative reward; E() represents the expectation function; t represents the tth group of environmental interaction data; T represents the number of environmental interaction data; γ represents the discount rate; R() represents the reward function used to calculate the reward; s t represents the initial environment state in the tth group of environment interaction data; a t Represents the action in the tth group of environmental interaction data.

[0073] Step S104, based on the cumulative rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, and this round of training is completed. Specifically, based on the cumulative rewards obtained by the reinforcement learning agent of the quadruped robot during this round of training, the overall performance of the reinforcement learning agent's behavior is quantified, and the network parameters are updated according to the cumulative rewards, so that the generalization ability and performance of the updated adversarial network structure are improved.

[0074] Step S105, repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and enable the quadruped robot to work using the target reinforcement learning agent. Specifically, multiple rounds of training of the reinforcement learning agent are performed according to the above steps S101 to S104 to obtain the target reinforcement learning agent, so that the quadruped robot can still work normally in a variety of complex environments using the target reinforcement learning agent, which is of great significance for the smooth performance of complex tasks such as search and rescue and cargo transportation by the quadruped robot.

[0075] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention integrates the structures of an adversarial network, a policy network, and an evaluation network, obtains environmental interaction data under various environmental conditions, and iteratively updates the network parameters of multiple networks based on the accumulated rewards. Multiple rounds of training are performed to obtain a target reinforcement learning agent, thereby improving the robustness of the target reinforcement learning agent. When the quadruped robot uses the target reinforcement learning agent to work, it can still work normally in a variety of complex environments, thereby enhancing its adaptability and robustness to complex environments.

[0076] In this embodiment, a reinforcement learning agent training method for a quadruped robot is provided, which can be used for the above-mentioned quadruped robot. Figure 2 is a flowchart of another reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0077] Step S201, when the reinforcement learning agent of the quadruped robot starts this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are obtained. The adversarial network structure includes an adversarial network, a policy network and an evaluation network. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0078] Step S202, based on the adversarial network structure, obtain multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training, where any set of environmental interaction data includes environmental state, action and reward.

[0079] Specifically, the above step S202 includes:

[0080] Step S2021, when acquiring the first set of environmental interaction data, based on the multiple sensors of the quadruped robot, the initial environmental state is acquired. Specifically, the quadruped robot collects information through multiple sensors, and forms the initial environmental state after processing. The sensor observation space size of the quadruped robot is 34, of which 1-4 bits are foot contact sensor vectors, which detect whether the feet of the quadruped robot are in contact with the ground or other surfaces, and provide the magnitude of the contact force; 5-10 bits are inertial measurement unit sensor vectors, which provide the linear acceleration and angular velocity of the quadruped robot when moving; 11-34 bits are motor angle acceleration vectors, which represent the second-order derivative of the motor rotation angle of the quadruped robot, that is, angular acceleration, which describes the speed of the motor angular velocity change. On the basis of 34 bits, 12 bits of action and 3 bits of speed instructions are added to form a 49-bit state space for a single time step. In order to capture the continuity of dynamic behavior and the context of environmental interaction, the current environment state is combined with the historical observations of the previous 11 time steps. The total observation space size is obtained by multiplying 49 bits by 11 and adding the 49 bits of information of the current time step, forming a total of 588 bits of observation space, which is used as the initial environment state.

[0081] Step S2022, determining the adversarial noise corresponding to the initial environmental state based on the adversarial network, where the adversarial network includes a first input layer, a first hidden layer, and a first output layer.

[0082] In some optional implementations, the above step S2022 includes:

[0083] Step a1, based on the first input layer of the adversarial network, receiving the initial environment state, adjusting the vector dimension of the initial environment state, and obtaining the data feature representation of the initial environment state. Specifically, in the first input layer, the vector dimension can be adjusted by adding a dimensionality reduction layer or using a fully connected layer with fewer output nodes, so as to extract the key feature representation of the initial environment state.

[0084] Step a2, based on the first hidden layer of the adversarial network, a complex representation is constructed on the data feature representation of the initial environment state and nonlinearity is introduced. Specifically, the first hidden layer includes multiple fully connected layers, and the multiple fully connected layers perform complex nonlinear transformations through a series of weight matrix and bias term transformations, as well as the application of activation functions, to capture and express high-level features and potential relationships in the data feature representation of the initial environment state.

[0085] Step a3, based on the first output layer of the adversarial network, the vector dimension is adjusted on the basis of the first hidden layer, and adversarial noise is output. Specifically, based on the complex representation of the first hidden layer, the first output layer finally generates adversarial noise by adjusting the vector dimension. The adversarial noise represents the interference that has the greatest impact on the policy network, so that the policy network finds the optimal action of the quadruped robot under the maximum interference, so that the obtained action has generalization ability in other non-maximum interference situations.

[0086] Step S2023, based on the strategy network, according to the initial environment state and the adversarial noise corresponding to the initial environment state, determine the corresponding action of the quadruped robot in the initial environment state, the strategy network includes a second input layer, a second hidden layer and a second output layer.

[0087] In some optional implementations, the above step S2023 includes:

[0088] Step b1, based on the second input layer of the policy network, receiving the initial environment state and the adversarial noise, adjusting the vector dimensions of the initial environment state and the adversarial noise, and obtaining the data feature representation of the initial environment state and the data feature representation of the adversarial noise. Specifically, the policy network receives the initial environment state and the adversarial noise that interferes with the initial environment state the most, and sends them to the second input layer of the policy network, that is, the dimension of the second input layer is the sum of the dimension of the initial environment state and the dimension of the adversarial noise. The vector dimensions of these two parts of the input are adjusted by applying the fully connected layer or the embedding layer of the second input layer, and converted into a form suitable for subsequent network processing.

[0089] Step b2, based on the second hidden layer of the policy network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the anti-noise and nonlinearity is introduced. Specifically, the second hidden layer includes multiple fully connected layers, each of which is followed by a nonlinear activation function to construct a complex and abstract feature representation of the initial environment state and the anti-noise, which helps the policy network understand which state features are most critical under the interference of the anti-noise, so as to obtain the most appropriate action that the quadruped robot should take when facing the anti-noise in the initial environment state.

[0090] Step b3, based on the second output layer of the policy network, the vector dimension is adjusted based on the second hidden layer to output the action. Specifically, in the second output layer, based on the complex representation generated by the second hidden layer, the vector dimension is adjusted through the fully connected layer of the second output layer to output the final action. The action is output through the second output layer of the policy network to guide the quadruped robot to make the most appropriate action in a given environment, providing support for the training of reinforcement learning agents.

[0091] Step S2024, based on the evaluation network, when the quadruped robot takes action in the initial environment state, it is rewarded, and the evaluation network includes a third input layer, a third hidden layer and a third output layer.

[0092] In some optional implementations, the above step S2024 includes:

[0093] Step c1, based on the third input layer of the evaluation network, receiving the initial environment state and action, adjusting the vector dimensions of the initial environment state and action, and obtaining the data feature representation of the initial environment state and the data feature representation of the action. Specifically, the evaluation network receives the initial environment state and the action taken by the quadruped robot, and uses the fully connected layer of the third input layer to extract features and adjust the dimensions of the vectors of the initial environment state and action, respectively, to form their respective feature representations, so that the evaluation network can comprehensively consider the joint impact of the state and action on the subsequent results.

[0094] Step c2, based on the third hidden layer of the evaluation network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the action and nonlinearity is introduced. Specifically, the third hidden layer includes multiple fully connected layers, and nonlinear activation functions are used between each layer to construct a representation that can capture the complex interaction between the two, and extract a high-level representation reflecting the potential reward value from the state-action pair, so that the evaluation network can learn and quantify the degree of goodness of a certain action taken by the quadruped robot in a specific state, that is, the reward.

[0095] Step c3, based on the third output layer of the evaluation network, adjust the vector dimension on the basis of the third hidden layer and output the reward. Specifically, based on the complex representation of the third hidden layer, the vector dimension is adjusted through the third output layer, and the expected reward value that can be obtained by taking a specific action under a given initial environmental state is finally predicted or estimated. Outputting rewards through the third output layer of the evaluation network directly reflects the expected benefits of the quadruped robot taking a specific action under a given environmental state. It is the core feedback mechanism for guiding strategy updates and optimizing cumulative rewards in reinforcement learning, and provides support for the training of reinforcement learning agents.

[0096] It should be noted that the above steps a1 to a3, steps b1 to b3 and steps c1 to c3 are all prior art, and the above process is only an example.

[0097] Step S2025, the environmental state obtained by the interaction of the action with any environmental condition is used as the initial environmental state of the next set of environmental interaction data, and the above process of obtaining the initial environmental state, action and reward is repeated to obtain multiple sets of environmental interaction data for this round of training. Specifically, a set of environmental interaction data is obtained through the above steps S2021 to S2024, and the actions in the set of environmental interaction data are made to interact with the environmental conditions corresponding to any complex environment to obtain a new environmental state represented by a 588-bit vector as the initial environmental state of the next set of environmental interaction data. Repeat the above steps of obtaining environmental interaction data to obtain multiple sets of environmental interaction data in complex environments, so that the reinforcement learning agent trained based on multiple sets of environmental interaction data has good generalization ability and robustness.

[0098] In some optional embodiments, Figure 3 is a schematic diagram of obtaining environmental interaction data according to an embodiment of the present invention, such as Figure 3 As shown in the figure, when obtaining any set of environmental interaction data, the initial environmental state is input into the adversarial network, and the adversarial network determines the adversarial noise based on the initial environmental state. Then, the initial environmental state and the adversarial noise are input into the policy network together to obtain the action that the quadruped robot should take when the adversarial noise exists in the initial environmental state. Then, the evaluation network evaluates the action taken by the quadruped robot and obtains a reward. Finally, the action is interacted with any environmental condition to obtain the initial environmental state when obtaining the next set of environmental interaction data, and the next set of environmental interaction data is obtained.

[0099] Step S203, based on multiple sets of environment interaction data, determine the cumulative reward for this round of training. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0100] Step S204: Based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, thus completing this round of training. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0101] Step S205, repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and make the quadruped robot work using the target reinforcement learning agent. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0102] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention integrates the structures of an adversarial network, a policy network, and an evaluation network, obtains environmental interaction data under various environmental conditions, and iteratively updates the network parameters of multiple networks based on the accumulated rewards. Multiple rounds of training are performed to obtain a target reinforcement learning agent, thereby improving the robustness of the target reinforcement learning agent. When the quadruped robot uses the target reinforcement learning agent to work, it can still work normally in a variety of complex environments, thereby enhancing its adaptability and robustness to complex environments.

[0103] In this embodiment, a reinforcement learning agent training method for a quadruped robot is provided, which can be used for the above-mentioned quadruped robot. Figure 4 is a flowchart of another reinforcement learning agent training method applied to a quadruped robot according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0104] Step S401, when the reinforcement learning agent of the quadruped robot starts this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are obtained. The adversarial network structure includes an adversarial network, a policy network and an evaluation network. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0105] Step S402: Based on the adversarial network structure, multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training are obtained. Each set of environmental interaction data includes environmental state, action and reward. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0106] Step S403: Determine the cumulative reward for this round of training based on multiple sets of environment interaction data. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0107] Step S404, based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, and this round of training is completed.

[0108] Specifically, the above step S404 updates the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training based on the accumulated rewards of this round of training, and obtains the network parameters of the adversarial network structure of the quadruped robot in this round of training, including:

[0109] Step S4041, taking the adversarial network as the adversarial agent and the policy network as the ontology agent, constructing a zero-sum game problem. Specifically, a zero-sum game refers to the parties involved in the game. Under strict competition, the gain of one party must mean the loss of the other party, and the sum of the gains and losses of the parties to the game is always "zero". The goal of the ontology agent in reinforcement learning is to learn the optimal actions to perform in a given environment to maximize the cumulative reward. The purpose of the adversarial agent is to challenge the ontology agent and enhance the robustness of the ontology agent by imposing interference, providing opponent behavior or other forms of confrontation. In an embodiment of the present invention, the adversarial network is used as the adversarial agent, and the policy network is used as the ontology agent. The sum of the gain of the adversarial network and the loss of the policy network is zero, and the two constitute a zero-sum game problem.

[0110] Step S4042, with the goal of solving the Nash equilibrium point of the zero-sum game problem, based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training.

[0111] In some optional implementations, the above step S4042 includes:

[0112] Step d1, fix the network parameters of the previous round of training of the policy network and the evaluation network, and based on the cumulative rewards of this round of training, perform gradient ascent on the network parameters of the previous round of training of the adversarial network to obtain the network parameters of the current round of training of the adversarial network. Specifically, the Nash equilibrium point for solving the zero-sum game problem can be converted into a saddle point for solving the function, so as to update the network parameters of the adversarial network and the policy network. When updating the network parameters of the adversarial network, fix the network parameters of the other two networks obtained in the previous round of training, and perform gradient ascent on the network parameters of the adversarial network obtained in the previous round of training through the following formula (2), increase the adversarial effect generated by the adversarial network in the game with the policy network, so that its parameters are updated in the direction that can maximize the cost of the policy network, indirectly promoting the maximization of the cumulative rewards, thereby obtaining the network parameters of the adversarial network obtained in this round of training. Optionally, gradient ascent is a prior art and will not be repeated here.

[0113]

[0114] Among them, k represents the kth round of training; k-1 represents the previous round of training before the kth round of training; α represents the learning rate; represents the network parameters of the adversarial network in the kth round of training; Represents the network parameters of the policy network in the previous round of training of the kth round of training.

[0115] Step d2, fix the network parameters of the previous round of training of the adversarial network and the evaluation network, and based on the cumulative reward of this round of training, perform gradient descent on the network parameters of the previous round of training of the policy network to obtain the network parameters of this round of training of the policy network. Specifically, when updating the network parameters of the policy network, fix the network parameters of the other two networks obtained in the previous round of training, and perform gradient descent on the network parameters of the policy network obtained in the previous round of training through the following formula (3), and find the optimal strategy under the condition of adversarial noise to maximize the cumulative reward, thereby obtaining the network parameters of the policy network obtained in this round of training.

[0116] Optionally, gradient ascent is a prior art and will not be described in detail here.

[0117]

[0118] Among them, k represents the kth round of training; k-1 represents the previous round of training before the kth round of training; α represents the learning rate; represents the network parameters of the policy network in the kth round of training; Represents the network parameters of the evaluation network in the previous round of training before the kth round of training.

[0119] Step d3, fix the network parameters of the previous round of training of the adversarial network and the policy network, update the network parameters of the previous round of training of the evaluation network, and obtain the network parameters of the current round of training of the evaluation network. Specifically, by fixing the network parameters of the adversarial network and the policy network in the previous round of training, TD-error (Temporal Differenceerror) can be used to update the network parameters of the evaluation network in the previous round of training. Optionally, other methods can also be used to update the network parameters of the evaluation network, which is not limited in the embodiment of the present invention.

[0120] Step S405, repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and enable the quadruped robot to work using the target reinforcement learning agent.

[0121] Specifically, the above step S405 repeats the above training process, performs multiple rounds of training, and obtains the target reinforcement learning agent of the quadruped robot, including:

[0122] Step S4051, repeat the above training process, and when the training rounds reach the preset rounds, use the policy network in the adversarial network structure as the target reinforcement learning agent. Specifically, the preset rounds can be 10 times, 20 times or 30 times, etc., and the embodiment of the present invention does not limit this. In each round of training, the network parameters of the adversarial network and the policy network are updated by the above formulas (2) and (3). When the training rounds reach the preset rounds, the policy network is used as the target reinforcement learning agent, so that the quadruped robot can use the target reinforcement learning agent in a complex environment and take the most suitable action for the current environment.

[0123] Alternatively, step S4052, repeat the above training process, and when the network parameters of the adversarial network structure reach convergence, the policy network in the adversarial network structure is used as the target reinforcement learning agent. Specifically, when the network parameters of the adversarial network structure reach convergence, the network parameters of the reinforcement learning agent are close to the optimal solution, tend to be stable, and the performance hardly changes. Optionally, the network parameters of the policy network in the adversarial network structure are generally focused on, and after each training round, the model parameters obtained from the previous round of training are compared. If the difference is less than a preset threshold, it is considered that convergence has been achieved, and the policy network is used as the target reinforcement learning agent. By continuously iterating and optimizing the policy network until the preset conditions are met, the target reinforcement learning agent of the quadruped robot has a high degree of stability and good generalization ability, ensuring that the quadruped robot can work normally in a variety of complex environments using the target reinforcement learning agent.

[0124] The reinforcement learning agent training method for a quadruped robot provided in an embodiment of the present invention integrates the structures of an adversarial network, a policy network, and an evaluation network, obtains environmental interaction data under various environmental conditions, and iteratively updates the network parameters of multiple networks based on the accumulated rewards. Multiple rounds of training are performed to obtain a target reinforcement learning agent, thereby improving the robustness of the target reinforcement learning agent. When the quadruped robot uses the target reinforcement learning agent to work, it can still work normally in a variety of complex environments, thereby enhancing its adaptability and robustness to complex environments.

[0125] In this embodiment, a reinforcement learning agent training device for a quadruped robot is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0126] This embodiment provides a reinforcement learning agent training device for a quadruped robot. Figure 5 As shown, including:

[0127] The first acquisition module 501 is used to obtain the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training when the reinforcement learning agent of the quadruped robot starts this round of training. The adversarial network structure includes an adversarial network, a policy network and an evaluation network.

[0128] The second acquisition module 502 is used to obtain multiple groups of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training based on the adversarial network structure, and any group of environmental interaction data includes environmental state, action and reward.

[0129] The determination module 503 is used to determine the cumulative reward of this round of training based on multiple sets of environment interaction data.

[0130] The updating module 504 is used to update the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training based on the accumulated rewards of this round of training, obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training, and complete this round of training.

[0131] The training module 505 is used to repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and enable the quadruped robot to work using the target reinforcement learning agent.

[0132] In some optional implementations, the second acquisition module 502 includes:

[0133] The first acquisition unit is used to acquire an initial environment state based on multiple sensors of the quadruped robot when acquiring the first set of environment interaction data.

[0134] The first determining unit is used to determine the adversarial noise corresponding to the initial environment state based on the adversarial network.

[0135] The second determination unit is used to determine the corresponding action of the quadruped robot under the initial environment state based on the strategy network and according to the initial environment state and the countermeasure noise corresponding to the initial environment state.

[0136] The third determination unit is used to obtain a reward based on the evaluation network when the quadruped robot takes an action in the initial environment state.

[0137] The second acquisition unit is used to use the environmental state obtained by the interaction of the action with any environmental condition as the initial environmental state of the next set of environmental interaction data, repeat the above process of obtaining the initial environmental state, action and reward, and obtain multiple sets of environmental interaction data for this round of training.

[0138] In some optional implementations, the updating module 504 includes:

[0139] A construction unit is used to construct a zero-sum game problem using the adversarial network as the adversarial agent and the policy network as the ontological agent.

[0140] The solving unit is used to solve the Nash equilibrium point of the zero-sum game problem, and based on the accumulated rewards of this round of training, update the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training.

[0141] In some optional implementations, the solving unit includes:

[0142] The first updating subunit is used to fix the network parameters of the previous round of training of the policy network and the evaluation network, and based on the accumulated rewards of this round of training, perform gradient ascent on the network parameters of the previous round of training of the adversarial network to obtain the network parameters of this round of training of the adversarial network.

[0143] The second updating subunit is used to fix the network parameters of the previous round of training of the adversarial network and the evaluation network, and based on the accumulated rewards of this round of training, perform gradient descent on the network parameters of the previous round of training of the policy network to obtain the network parameters of this round of training of the policy network.

[0144] The third updating subunit is used to fix the network parameters of the previous round of training of the adversarial network and the policy network, update the network parameters of the previous round of training of the evaluation network, and obtain the network parameters of the current round of training of the evaluation network.

[0145] In some optional implementations, the training module 505 includes:

[0146] The third determination unit is used to repeat the above training process, and when the training rounds reach a preset round, the policy network in the adversarial network structure is used as the target reinforcement learning agent.

[0147] Alternatively, the fourth determination unit is used to repeat the above training process, and when the network parameters of the adversarial network structure reach convergence, use the policy network in the adversarial network structure as the target reinforcement learning agent.

[0148] In some optional implementations, the adversarial network includes a first input layer, a first hidden layer, and a first output layer, and the first determining unit includes:

[0149] The first adjustment subunit is used to receive the initial environment state based on the first input layer of the adversarial network, adjust the vector dimension of the initial environment state, and obtain the data feature representation of the initial environment state.

[0150] The first construction subunit is used to construct a complex representation and introduce nonlinearity on the data feature representation of the initial environment state based on the first hidden layer of the adversarial network.

[0151] The first output subunit is used to adjust the vector dimension based on the first hidden layer based on the first output layer of the adversarial network and output adversarial noise.

[0152] In some optional implementations, the policy network includes a second input layer, a second hidden layer, and a second output layer, and the second determination unit includes:

[0153] The second adjustment subunit is used to receive the initial environment state and the adversarial noise based on the second input layer of the strategy network, adjust the vector dimensions of the initial environment state and the adversarial noise, and obtain the data feature representation of the initial environment state and the data feature representation of the adversarial noise.

[0154] The second construction subunit is used to construct a complex representation and introduce nonlinearity on the data feature representation of the initial environment state and the data feature representation of the anti-noise based on the second hidden layer of the policy network.

[0155] The second output subunit is used to adjust the vector dimension based on the second hidden layer based on the second output layer of the policy network and output the action.

[0156] In some optional implementations, the evaluation network includes a third input layer, a third hidden layer, and a third output layer, and the second determination unit includes:

[0157] The third adjustment subunit is used to receive the initial environment state and action based on the third input layer of the evaluation network, adjust the vector dimensions of the initial environment state and the action, and obtain the data feature representation of the initial environment state and the data feature representation of the action.

[0158] The third construction subunit is used to construct a complex representation and introduce nonlinearity on the data feature representation of the initial environment state and the data feature representation of the action based on the third hidden layer of the evaluation network.

[0159] The third output subunit is used to adjust the vector dimension based on the third hidden layer based on the third output layer of the evaluation network and output the reward.

[0160] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0161] The reinforcement learning agent training device applied to a quadruped robot in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0162] The embodiment of the present invention also provides a computer device having the above Figure 5 The reinforcement learning agent training device applied to a quadruped robot is shown.

[0163] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0164] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0165] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0166] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0168] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0169] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0170] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0171] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A reinforcement learning agent training method for a quadruped robot, characterized in that: The method comprises: When the reinforcement learning agent of the quadruped robot starts this round of training, network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are obtained, wherein the adversarial network structure includes an adversarial network, a policy network, and an evaluation network; Based on the adversarial network structure, multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training are obtained, wherein any set of environmental interaction data includes environmental state, action and reward; Determine the cumulative reward for this round of training based on the multiple sets of environment interaction data; Based on the accumulated rewards of the current round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in the current round of training, thereby completing the current round of training; The above training process is repeated for multiple rounds of training to obtain the target reinforcement learning agent of the quadruped robot, so that the quadruped robot uses the target reinforcement learning agent to work.

2. The method according to claim 1, characterized in that Based on the adversarial network structure, multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training are obtained, including: When acquiring the first set of environmental interaction data, acquiring an initial environmental state based on a plurality of sensors of the quadruped robot; Determine the adversarial noise corresponding to the initial environmental state based on the adversarial network; Based on the strategy network, determining the action of the quadruped robot corresponding to the initial environmental state according to the initial environmental state and the countermeasure noise corresponding to the initial environmental state; Based on the evaluation network, when the quadruped robot takes the action in the initial environment state, a reward is obtained; The environmental state obtained by the interaction of the action with any environmental condition is used as the initial environmental state of the next set of environmental interaction data, and the above process of obtaining the initial environmental state, action and reward is repeated to obtain multiple sets of environmental interaction data for this round of training.

3. The method according to claim 1, characterized in that The method of updating the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training based on the accumulated rewards of the current round of training to obtain the network parameters of the adversarial network structure of the quadruped robot in the current round of training includes: The adversarial network is used as an adversarial agent, and the policy network is used as an ontology agent to construct a zero-sum game problem; With the goal of solving the Nash equilibrium point of the zero-sum game problem, based on the accumulated rewards of this round of training, the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training are updated to obtain the network parameters of the adversarial network structure of the quadruped robot in this round of training.

4. The method according to claim 3, characterized in that The method aims to solve the Nash equilibrium point of the zero-sum game problem, updates the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training, and obtains the network parameters of the adversarial network structure of the quadruped robot in this round of training, including: The network parameters of the previous round of training of the policy network and the evaluation network are fixed, and based on the accumulated rewards of the current round of training, the network parameters of the previous round of training of the adversarial network are gradient increased to obtain the network parameters of the current round of training of the adversarial network; The network parameters of the previous round of training of the adversarial network and the evaluation network are fixed, and based on the accumulated rewards of the current round of training, the network parameters of the previous round of training of the policy network are gradient descended to obtain the network parameters of the current round of training of the policy network; The network parameters of the previous round of training of the adversarial network and the policy network are fixed, and the network parameters of the previous round of training of the evaluation network are updated to obtain the network parameters of the current round of training of the evaluation network.

5. The method according to claim 1, characterized in that The above training process is repeated for multiple rounds of training to obtain the target reinforcement learning agent of the quadruped robot, including: Repeat the above training process, and when the number of training rounds reaches a preset number, use the policy network in the adversarial network structure as the target reinforcement learning agent; or, The above training process is repeated, and when the network parameters of the adversarial network structure reach convergence, the policy network in the adversarial network structure is used as the target reinforcement learning agent.

6. The method according to claim 2, characterized in that The adversarial network includes a first input layer, a first hidden layer, and a first output layer, and determining the adversarial noise corresponding to the initial environmental state based on the adversarial network includes: Based on the first input layer of the adversarial network, receiving the initial environment state, adjusting the vector dimension of the initial environment state, and obtaining a data feature representation of the initial environment state; Based on the first hidden layer of the adversarial network, a complex representation is constructed on the data feature representation of the initial environment state and nonlinearity is introduced; Based on the first output layer of the adversarial network, the vector dimension is adjusted on the basis of the first hidden layer to output the adversarial noise.

7. The method according to claim 2, characterized in that The strategy network includes a second input layer, a second hidden layer, and a second output layer. The determining, based on the strategy network and according to the initial environment state and the adversarial noise corresponding to the initial environment state, the corresponding action of the quadruped robot under the initial environment state includes: Based on the second input layer of the strategy network, receiving the initial environment state and the adversarial noise, adjusting the vector dimensions of the initial environment state and the adversarial noise, and obtaining a data feature representation of the initial environment state and a data feature representation of the adversarial noise; Based on the second hidden layer of the policy network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the anti-noise and nonlinearity is introduced; Based on the second output layer of the policy network, the vector dimension is adjusted on the basis of the second hidden layer, and the action is output.

8. The method according to claim 2, characterized in that: The evaluation network includes a third input layer, a third hidden layer and a third output layer. The method of obtaining a reward based on the evaluation network when the quadruped robot takes the action in the initial environment state includes: Based on the third input layer of the evaluation network, the initial environment state and the action are received, and the vector dimensions of the initial environment state and the action are adjusted to obtain a data feature representation of the initial environment state and a data feature representation of the action; Based on the third hidden layer of the evaluation network, a complex representation is constructed on the data feature representation of the initial environment state and the data feature representation of the action and nonlinearity is introduced; Based on the third output layer of the evaluation network, the vector dimension is adjusted on the basis of the third hidden layer, and the reward is output.

9. A reinforcement learning agent training device for a quadruped robot, characterized in that: The device comprises: A first acquisition module is used to acquire network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training when the reinforcement learning agent of the quadruped robot starts the current round of training, wherein the adversarial network structure includes an adversarial network, a policy network and an evaluation network; A second acquisition module is used to acquire multiple sets of environmental interaction data obtained by the quadruped robot interacting with multiple environmental conditions in this round of training based on the adversarial network structure, where any set of environmental interaction data includes environmental state, action and reward; A determination module, configured to determine a cumulative reward for this round of training based on the multiple sets of environment interaction data; An updating module, configured to update the network parameters of the adversarial network structure of the quadruped robot obtained in the previous round of training based on the accumulated rewards of the current round of training, obtain the network parameters of the adversarial network structure of the quadruped robot in the current round of training, and complete the current round of training; The training module is used to repeat the above training process, perform multiple rounds of training, obtain the target reinforcement learning agent of the quadruped robot, and enable the quadruped robot to work using the target reinforcement learning agent.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the training method of the reinforcement learning agent according to any one of claims 1 to 8 by executing the computer instructions.

Images