Robot control method, device and equipment, storage medium and computer program product
Through the confrontational continuous dynamic domain alignment reinforcement learning framework and the teacher-student reinforcement learning framework, the continuous change law of dynamic parameters is extracted, and the problem of continuous changes in dynamic parameters in the existing technology is solved, and the robot is improved in the real environment. Control performance.
Patent Information
- Application Number
- CN202410951742.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-16
AI Technical Summary
Existing research defines the target environment only as a specific environment for kinetic parameters, but does not consider that kinetic parameters are continuous values, and the laws in continuous changes in kinetic parameters are not extracted, resulting in the performance of robot control decisions in the real world.
Adopting the adversarial-based continuous dynamic domain alignment reinforcement learning framework, control the robot through a pre-constructed environmental dynamic adaptation model, and extracting domain invariant features using the adversarial dynamic domain alignment model and the teacher-student reinforcement learning framework to reduce continuous dynamics The impact of change.
Effectively reduce the impact of factors such as camera rotation, mass, robot connecting rod and surface friction on robot performance, and improve control performance in real environments.
Smart Images

Figure CN119057770A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a robot control method, device, equipment, storage medium and computer program product. Background Art
[0002] When performing deep reinforcement learning on robots, robot control tasks are usually trained in physical simulators rather than on real hardware, due to the higher efficiency and better safety of simulator training. However, it is difficult for physical simulators to simulate real physical environments and complex dynamic changes, and the resulting domain differences will have a negative impact on the performance of robot control decisions in the real world.
[0003] In this regard, existing studies only define the target environment as an environment with specific kinetic parameters, but do not take into account that the kinetic parameters are continuous values, and do not attempt to extract the laws of continuous changes in kinetic parameters.
[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present application is to provide a robot control method, device, equipment, storage medium and computer program product, aiming to solve the technical problem that existing research only defines the target environment as an environment with specific dynamic parameters, but does not take into account that the dynamic parameters are continuous values and does not attempt to extract the laws of continuous changes in the dynamic parameters.
[0006] To achieve the above objectives, the present application proposes a robot control method, the method comprising:
[0007] In response to control instructions, the robot is controlled by a pre-constructed environment dynamics adaptation model, which is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
[0008] In one embodiment, the step of controlling the robot by a pre-built environmental dynamics adaptation model in response to the control instruction comprises:
[0009] Collect the trajectories of the robot controlled by the pre-built teacher reinforcement learning model to perform tasks in the source domain and the target domain respectively, which are recorded as the source domain trajectory dataset and the target domain trajectory dataset;
[0010] Selecting trajectory data from the source domain trajectory dataset and the target domain trajectory dataset as support sets respectively;
[0011] Selecting trajectory data from the source domain trajectory dataset as a query set;
[0012] Performing domain-invariant feature extraction on the support set using a pre-built adversarial dynamics domain alignment model to obtain a discriminator loss;
[0013] Instantiating the encoder in the teacher reinforcement learning model as meta-initial parameters;
[0014] Performing gradient descent on the discriminator loss by a meta-optimization method to update the meta-initial parameters to first task-specific parameters;
[0015] Updating a pre-built student reinforcement learning model based on the first task-specific parameters;
[0016] Calculating the query set through the updated student reinforcement learning model to obtain a similarity loss;
[0017] The encoder, the policy network and the value network in the student reinforcement learning model are back-propagated and updated again based on the similarity loss and the discriminator loss, and the updated student reinforcement learning model is recorded as the environment dynamics adaptation model.
[0018] In one embodiment, the step of respectively collecting trajectories of controlling the robot to perform tasks in the source domain and the target domain using the pre-built teacher reinforcement learning model includes:
[0019] The teacher reinforcement learning model is constructed based on the proximal strategy optimization algorithm;
[0020] The student reinforcement learning model is constructed based on the teacher reinforcement learning model, and the strategy network and value network of the student reinforcement learning model are aligned with the strategy network and value network of the teacher reinforcement learning model.
[0021] In one embodiment, the adversarial dynamics domain alignment model uses a policy network and a value network in a reinforcement learning algorithm as task regressors, and the adversarial dynamics domain alignment model further includes an encoder corresponding to the policy network, an encoder corresponding to the value network, and a continuous domain discriminator.
[0022] The step of extracting domain-invariant features from the support set using the pre-built adversarial dynamics domain alignment model to obtain a discriminator loss comprises:
[0023] Extracting the support set through an encoder corresponding to the policy network and an encoder corresponding to the value network in the adversarial dynamics domain alignment model to obtain a first intermediate feature and a second intermediate feature;
[0024] splicing the first intermediate feature and the second intermediate feature and inputting the concatenated features into a continuous domain discriminator of the adversarial dynamic domain alignment model;
[0025] By performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network and the continuous domain discriminator in the adversarial dynamics domain alignment model, the distribution difference between the source domain trajectory dataset and the target domain trajectory dataset in the support set is minimized to obtain the discriminator loss.
[0026] In one embodiment, the step of performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network, and the continuous domain discriminator includes:
[0027] The loss of the task regressor is minimized through the encoder corresponding to the policy network and the encoder corresponding to the value network, the loss of the continuous domain discriminator is maximized, and the loss of the continuous domain discriminator is minimized through the continuous domain discriminator, thereby extracting domain invariant features.
[0028] In one embodiment, the step of back-propagating based on the similarity loss and the discriminator loss and updating the encoder, policy network and value network in the student reinforcement learning model again comprises:
[0029] updating the meta-initial parameters to second task-specific parameters based on the similarity loss and the discriminator loss;
[0030] The encoder, policy network, and value network in the student reinforcement learning model are updated based on the second task-specific parameters.
[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a robot control device, the robot control device comprising:
[0032] The robot control module is used to control the robot in response to the control instruction through a pre-built environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on the adversarial continuous dynamics domain alignment reinforcement learning framework.
[0033] In addition, to achieve the above-mentioned objectives, the present application also proposes a robot control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the robot control method described above.
[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the robot control method described above are implemented.
[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the robot control method described above are implemented.
[0036] One or more technical solutions proposed in this application have at least the following technical effects:
[0037] The present application controls the robot in response to control instructions through a pre-built environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework. The adversarial continuous dynamics domain alignment reinforcement learning framework is a model-independent reinforcement learning framework used for dynamics adaptive robot control, which can reduce the impact of various continuous dynamic changes (such as camera rotation, mass, robot links and surface friction) on robot performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0040] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the robot control method of the present application;
[0041] Figure 2 A schematic diagram of the process of obtaining an environmental dynamics adaptation model by training the robot control method of this application based on the adversarial continuous dynamics domain alignment reinforcement learning framework;
[0042] Figure 3 A comparison chart of the Chinese and English names of the algorithms used in the experimental verification of this application;
[0043] Figure 4 is the average total reward value of each algorithm under the Pendulum task;
[0044] Figure 5 is the average total reward value of each algorithm under the half-cheetah task;
[0045] Figure 6 is the average total reward value of each algorithm in the lame half cheetah task;
[0046] Figure 7is the average total reward value of each algorithm under the Ant task;
[0047] Figure 8 The experimental data diagram of the robot control method of this application in the real environment;
[0048] Fig. 9 This is a schematic diagram of the module structure of the robot control device according to an embodiment of the present application;
[0049] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the robot control method in the embodiment of the present application.
[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0053] The main solution of the embodiment of the present application is: in response to control instructions, the robot is controlled by a pre-constructed environment dynamics adaptation model, and the environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
[0054] Technical terms involved in the embodiments of the present invention:
[0055] Pendulum task: Inverted Pendulum task. In this task, the goal of the agent is to control a pendulum so that it remains balanced in a vertical position. The agent needs to achieve this goal by applying force or torque to keep the pendulum upright for as long as possible.
[0056] Half-cheetah task: A common motion control task that usually involves a half-cheetah-shaped robot in a simulated environment. In this task, the agent's goal is to make the robot move at the highest speed by controlling the joint angles or forces in order to move as far as possible in a given time.
[0057] Lame Half Cheetah Task: This is a variation of the Half Cheetah Task, where the robot is designed to have some kind of impairment or limitation in its movement capabilities. This impairment can be limited movement of certain joints or an irregular gait, which increases the complexity and challenge of the task.
[0058] Ant Task: This is another classic reinforcement learning control task. In this task, the agent needs to control a multi-jointed simulated ant robot to move and navigate smoothly in a simulated environment.
[0059] In this embodiment, for ease of description, the following description is made with the robot control device as the execution subject.
[0060] Since existing studies only define the target environment as an environment with specific kinetic parameters, but do not take into account that kinetic parameters are continuous values, no attempt is made to extract the laws of continuous changes in kinetic parameters.
[0061] The present application provides a solution. The present application controls the robot in response to control instructions through a pre-built environment dynamics adaptation model. The environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework. The adversarial continuous dynamics domain alignment reinforcement learning framework is a model-independent reinforcement learning framework used for dynamics adaptive robot control, which can reduce the impact of various continuous dynamic changes (such as camera rotation, mass, robot links and surface friction) on robot performance.
[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a robot control device, etc. The following takes a robot control device as an example to illustrate this embodiment and the following embodiments.
[0063] Based on this, the present application embodiment provides a robot control method, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the robot control method of the present application.
[0064] In this embodiment, the robot control method includes step S100:
[0065] Step S100, in response to a control instruction, the robot is controlled by a pre-constructed environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
[0066] It should be noted that this application proposes a novel adversarial-based continuous dynamics domain alignment reinforcement learning (CDA-RL) framework, which combines adversarial domain adaptation with reinforcement learning to eliminate the impact of dynamic changes.
[0067] Furthermore, the training of the environment dynamics adaptation model based on the adversarial continuous dynamics domain alignment reinforcement learning framework may include steps S10 to S90:
[0068] Step S10, respectively collecting trajectories of the robot controlled by the pre-built teacher reinforcement learning model to perform tasks in the source domain and the target domain, recorded as the source domain trajectory dataset and the target domain trajectory dataset;
[0069] It should be noted that the teacher reinforcement learning model is constructed based on the proximal policy optimization algorithm (PPO algorithm).
[0070] The virtual environment used to train the robot is considered as the source domain, and the real environment used for testing is considered as the target domain. For each environment, there is a set of dynamic parameter values (dynamic parameter values are a set of dynamic parameters such as mass, friction, etc., and U is used to represent the set of different values of these dynamic parameters). This value is denoted as u s ,u t ∈U and use it as the domain label of the environment.
[0071] Each kinetic parameter has its own range of values and is continuous. Assume that the kinetic parameter of the source domain is restricted to a specific range of values, denoted as u s , while the dynamic parameter ut of the target domain has another range of values, which is similar to u s Not inclusive of each other, that is
[0072] The robot is controlled by the teacher reinforcement learning model to train in the source domain and the target domain respectively to obtain different trajectories.
[0073] In this method, the teacher reinforcement learning model is responsible for collecting multiple trajectories τ containing states s, actions a, and rewards r. The trajectory datasets sampled in the source domain and the target domain are denoted as the source domain trajectory dataset τ s and the target domain trajectory dataset τ t .
[0074] In addition, the source domain trajectory dataset and the target domain trajectory dataset are stored in a buffer.
[0075] Step S20, selecting trajectory data from the source domain trajectory dataset and the target domain trajectory dataset as support sets respectively;
[0076] Step S30, selecting trajectory data from the source domain trajectory data set as a query set;
[0077] Among them, this application adopts an unsupervised domain adaptation framework and uses the source domain trajectory dataset τ s The reinforcement learning model is trained on a small target domain trajectory dataset τt We fine-tune a reinforcement learning model by using only domain labels.
[0078] Step S40, performing domain invariant feature extraction on the support set by using a pre-built adversarial dynamics domain alignment model to obtain a discriminator loss;
[0079] It should be noted that the adversarial dynamics domain alignment model proposed in the present invention adopts an adversarial domain adaptive algorithm to realize cross-domain reinforcement learning.
[0080] Adversarial domain adaptation is a classic domain transfer algorithm that minimizes the distribution difference between the source domain and the target domain through the confrontation between the encoder and the discriminator to obtain domain-invariant features. It is essentially a dual-objective model consisting of an encoder, a discriminator, and a task regressor.
[0081] The encoder aims to extract discriminative features and minimize the classifier loss. Then, the discriminator aims to minimize the domain classifier loss. The optimization process of the two models is similar to a minimum-maximum game between two players, that is, the encoder constantly tries to extract features that can maximize the discriminative effect of confusing the discriminator, and the discriminator constantly tries to improve its domain discrimination ability for the encoder's extracted features, and finally the encoder extracts domain-invariant features.
[0082] In addition, it should be noted that the support set is composed of the training environment trajectory data and the test environment trajectory data. The features of the training environment trajectory data and the test environment trajectory data are extracted through the adversarial dynamics domain alignment model, and then the adversarial dynamics domain alignment model is used to make the features of the training environment trajectory data and the features of the test environment trajectory data look the same (that is, the discriminator cannot distinguish whether these features are from the training environment or the test environment. At this time, it can be considered that these features have eliminated the dynamics effect), which is the domain invariant feature.
[0083] It should be noted that the adversarial dynamics domain alignment model proposed in the present invention adopts the strategy network π in the reinforcement learning algorithm. θ and value network V π As task regressors, two encoders E are designed for the policy network and the value network. a 、E c To perform domain-invariant feature extraction, a continuous domain discriminator D is designed. The optimization objectives are summarized as follows:
[0084]
[0085] Among them, L a and L c Represents the policy network loss and value network loss in reinforcement learning, L disc represents the discriminator loss, λd Controls the weight of the discriminator loss in the total loss, so that the model strikes a balance between the two tasks that need to be completed.
[0086] In scenarios using real robots, the dynamics naturally vary in a continuous space. However, most existing works assume that the dynamic parameters are discrete within a limited range.
[0087] To address this problem, the adversarial dynamics alignment model adopts CIDA and designs a continuous domain discriminator. This method treats the domain classification task as a regression problem rather than a discrete problem, thereby discovering the relationship between the domain index and the data distribution. To achieve this goal, the discriminator calculates the mean D of the embeddings respectively. μ and variance D σ2 , and align both to the same embedding. The discriminator loss is shown in formula (3-2):
[0088]
[0089] Here, s is the state, u is the domain label (dynamic parameters of the environment), and E(s,u) is the feature extracted after feeding these two parameters into the encoder.
[0090] In general, the above-mentioned min-max optimization of the adversarial dynamics alignment model mentioned in (3-1) may lead to fluctuating behavior, which may hinder the convergence of reinforcement learning. Therefore, the method proposed in this paper divides the adversarial dynamics alignment model into two parts and proposes teacher-student reinforcement learning and meta-optimization to ensure a stable training process, as described below.
[0091] During training, reinforcement learning shows high sensitivity to the simulated and real environments. The actor and critic encoders are modified in adversarial learning, and the resulting instability may cause the reinforcement learning model to accumulate inaccurate behaviors. To address this challenge, CDA-RL proposes a teacher-student reinforcement learning framework to enable the adversarial domain adaptation model to interact stably with the simulator. As shown in (3-1), a teacher reinforcement learning model interacts with the environment, while a student reinforcement learning model cooperates with the adversarial domain adaptation model to achieve domain adaptation.
[0092] Specifically, the teacher reinforcement learning in teacher-student reinforcement learning consists of the strategy network π, the value network V π and its encoder, which are used to interact with the simulated environment and generate the best action for each observation.
[0093] For robot control, the PPO method is selected as the reinforcement learning algorithm. The actor and critic are optimized using policy gradient and generalized advantage estimation (GAE). The optimization objectives of the actor and critic in teacher reinforcement learning are summarized as follows:
[0094]
[0095] Among them, π θ is the policy network, V π is the value network, s is the state, a is the action, u s is the dynamical parameter of the source domain, E a (s,u s ) is to input these two parameters into the encoder E a The extracted features, E c (s,u s ) is to input these two parameters into the encoder E c The extracted features, A π Denotes the advantage function. Different from the original formula, compared with the standard PPO, the performer and critic use the encoder E in the calculation a 、E c The extracted features Eactor and Ecritic are used instead of the state s.
[0096] The teacher reinforcement learning samples from the training and test environments and generates trajectories τ s , τ t , including state s, action a and reward r, are stored in a buffer to train the student reinforcement learning.
[0097] Student reinforcement learning has a similar structure and is designed to approximate teacher-student reinforcement learning through similarity loss. Given an observation s, the policy network π of the teacher reinforcement learning θ Output Gaussian distribution N(μ θ ,σ θ )=π θ (E a (s,u)) as an action, the value network V πθ Output reward r θ =V πθ (E c (s,u)) as the estimated return.
[0098] Policy network π for student reinforcement learning φ and value network V πφ is designed to be aligned with the policy network and value network of the teacher-student reinforcement learning so that the similarity of actions and estimated rewards is close, that is, when given the same s and u, N(μ t ,σ t )=N(μθ ,σ θ ) and r t =r θ Where N is a normal distribution, μ and σ are the mean and variance, r is the output of the value network, the subscript t indicates that this parameter is the output of the teacher reinforcement learning network, and the subscript θ indicates that this parameter is the output of the student reinforcement learning network.
[0099] KL divergence and L1 loss are used to measure the distance between actions and estimated rewards respectively. The similarity loss function is summarized as follows:
[0100]
[0101] Where μ and σ are the mean and variance, r is the output of the value network, the subscript t indicates that this parameter is the output of the teacher reinforcement learning network, and the subscript θ indicates that this parameter is the output of the student reinforcement learning network. E′ represents the executor and critic encoder E′a, E′c of the student reinforcement learning. This teacher-student reinforcement learning design avoids the interaction between teacher-student reinforcement learning and the adversarial domain adaptation model, thereby preventing drastic changes in the predictor loss during the minimum-maximum adversarial learning process. It ensures that the teacher-student reinforcement learning finds the optimal action and enables the student reinforcement learning to learn the domain-independent executor and critic encoder for collaborative learning.
[0102] Therefore, the action and reward evaluation losses of the adversarial domain adaptation model are replaced by the similarity loss L sim Instead, the optimization objective (3-1) of the adversarial domain adaptation model evolves as follows:
[0103]
[0104] Where E′ represents the executor and critic encoder E′a, E′c of the student reinforcement learning, D is the continuous domain discriminator, and λ d Controls the weight of the discriminator loss in the total loss, so that the model can achieve a balance between the two tasks that need to be completed. θ is the strategy network of teacher reinforcement learning, π φ It is the strategy network for students’ reinforcement learning.
[0105] Specifically, step S40, extracting domain-invariant features from the support set by using the pre-built adversarial dynamics domain alignment model to obtain the discriminator loss includes steps S41 to S43:
[0106] Step S41, extracting the support set through an encoder corresponding to the policy network and an encoder corresponding to the value network in the adversarial dynamics domain alignment model to obtain a first intermediate feature and a second intermediate feature;
[0107] Different from the standard adversarial domain adaptation framework, this paper designs two encoders E a 、E c To extract features from a state or observation s, where E a Responsible for extracting intermediate features Zactor for the policy network π, while E c Responsible for the value network V π Extract the intermediate feature Zcritic.
[0108] The reason for using two encoders instead of one is that during the gradient back-propagation process, using the same encoder to extract the intermediate features will result in shared parameters between the policy network and the value network, which will have a negative impact on the performance of reinforcement learning.
[0109] Step S42, inputting the first intermediate feature and the second intermediate feature into the policy network and the value network of the adversarial dynamics domain alignment model respectively, or concatenating the first intermediate feature and the second intermediate feature and then inputting them into the continuous domain discriminator of the adversarial dynamics domain alignment model;
[0110] Among them, the extracted intermediate features Zactor and Zcritic will be spliced into one intermediate feature and input into the continuous domain discriminator.
[0111] Step S43, by performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network and the continuous domain discriminator in the adversarial dynamics domain alignment model, the distribution difference between the source domain trajectory dataset and the target domain trajectory dataset in the support set is minimized to obtain the discriminator loss.
[0112] Further, step S43 includes step S431: minimizing the loss of the task regressor through the encoder corresponding to the policy network and the encoder corresponding to the value network, maximizing the loss of the continuous domain discriminator, and minimizing the loss of the continuous domain discriminator through the continuous domain discriminator, thereby extracting domain invariant features.
[0113] It should be noted that steps S50 to S90 are optimization means used by this application to solve the problem of too little target domain data in virtual-real migration. The optimization means are explained below:
[0114] 1) By building a training framework similar to the MAML model-independent meta-learning method, we can find the initial parameters that can adapt to various tasks most quickly. Even when there is not much data in the target domain, we can achieve the maximum performance improvement through a single gradient update.
[0115] 2) Based on this training framework, the loss function used for the inner loop update (i.e., the discriminator loss in the adversarial dynamics domain alignment model) and the loss function used for the outer loop update (i.e., the similarity loss in the student reinforcement learning model) are modified to ensure that the gradient directions generated by the outer loop update for the two tasks of up-domain invariant feature extraction and improving the total reward value are similar, so as to improve the performance of both at the same time, rather than generating two mutually exclusive gradients that cause performance fluctuations.
[0116] Specifically, the goal of the meta-optimization method draws on the idea of the model-independent meta-learning method by finding a set of initial parameters φ e (denoted as the initialization parameter), used for encoder E a 、E c , and the parameters φ of the strategy network and value network of the student reinforcement learning φ (also denoted as meta-initialization parameter), which enables the CDA-RL model to achieve optimal performance with extremely small data size and a small number of gradient update iterations.
[0117] Step S50, instantiating the encoder in the teacher reinforcement learning model as a meta-initial parameter;
[0118] It should be noted that this application divides the data set into support set R sup and the query set R que The size of the support set is significantly smaller than the query set. The meta-optimization method performs the initial gradient updates for the inner loop on the support set and the outer loop (fine-tuning) on the query set.
[0119] In the inner loop, the meta-optimization method transforms the encoder E in the teacher reinforcement learning model a 、E c Instantiated with meta-initialization parameter φ e .
[0120] Step S60, performing gradient descent on the discriminator loss by a meta-optimization method, and updating the meta-initial parameters to first task-specific parameters;
[0121] Among them, a batch of tasks are sampled from the support set, and the discriminator loss L is used disc Perform gradient descent and set the initial parameter φ e Updated to first mission specific parameters The details are as follows:
[0122]
[0123] Where D is the continuous domain discriminator and α is the learning rate of the inner loop. The inner loop is used to learn a domain-independent encoder without involving the policy network and the value network. In the outer loop, the optimization objectives 3-6 are used as the loss function of the outer loop as follows:
[0124]
[0125] Among them, π θ is the strategy network of teacher reinforcement learning, π φ is the strategy network of student reinforcement learning, V πφ is the value network of student reinforcement learning, φ e is the initial parameter, is the first task-specific parameter, λ d Controls the weight of the discriminator loss in the total loss, so that the model strikes a balance between the two tasks that need to be completed.
[0126] Step S70, updating the pre-built student reinforcement learning model based on the first task specific parameters;
[0127] The pre-built student reinforcement learning model is updated based on the first task-specific parameters, that is, the encoder in the student reinforcement learning model is updated to the first task-specific parameters. The corresponding encoder, then, through the updated encoder and policy network π in the student reinforcement learning model φ and value network V πφ To calculate the similarity loss L sim This helps to make L sim and L disc To verify this, we can substitute (3-7) into (3-8) and perform a first-order Taylor expansion on the first term of (3-8), and obtain the following result:
[0128]
[0129] Among them, α is the learning rate, π θ is the strategy network of teacher reinforcement learning, π φ is the strategy network for student reinforcement learning. e is the initial parameter, V πφ is the value network of student reinforcement learning, and D is the continuous domain discriminator.
[0130] The second term contains L disc and L sim This means that when the encoder and the actor-critic reinforcement learning model generate gradients in opposite directions during back-propagation, the second term of (3-9) will be negative, resulting in an increase in the overall loss.
[0131] Step S80, calculating the query set by using the updated student reinforcement learning model to obtain a similarity loss;
[0132] Step S90, back-propagating and re-updating the encoder, policy network and value network in the student reinforcement learning model based on the similarity loss and the discriminator loss, and recording the re-updated student reinforcement learning model as the environmental dynamics adaptation model.
[0133] Among them, the similarity loss and the discriminator loss are back-propagated and the parameters φ in the student reinforcement learning model are updated again e and φ φ .
[0134] Further, step S90 includes steps S91-S92:
[0135] Step S91, updating the meta-initial parameters to second task-specific parameters based on the similarity loss and the discriminator loss;
[0136] Among them, through formula (3-8), based on the similarity loss L sim and the discriminator loss L disc Calculate the total loss L total , then based on the total loss L total The initial parameter φ e and φ φ Updated to the second task-specific parameter φ' e and φ' φ . The specific formula is as shown in formula (3-10):
[0137]
[0138] Among them, φ e and φ φ is the initial parameter, φ' e and φ' φ is the second task-specific parameter and α is the learning rate.
[0139] Step S92, updating the encoder in the student reinforcement learning model based on the second task-specific parameters.
[0140] Among them, the second task-specific parameter φ' e and φ' φ Updates to the encoder in the student reinforcement learning model.
[0141] The updated student reinforcement learning model is then applied to a robotic control scenario to mitigate the impact of various continuous dynamic changes (such as camera rotation, mass, robot links, and surface friction) on the robot performance.
[0142] In another feasible implementation, step S10, before respectively collecting the trajectories of controlling the robot to perform tasks in the source domain and the target domain using the pre-built teacher reinforcement learning model, may include steps A11 to A12:
[0143] Step A11, constructing the teacher reinforcement learning model based on the proximal strategy optimization algorithm;
[0144] Step A12, constructing the student reinforcement learning model based on the teacher reinforcement learning model, wherein the strategy network and value network of the student reinforcement learning model are aligned with the strategy network and value network of the teacher reinforcement learning model.
[0145] In summary, the present invention constructs an environmental dynamics adaptation method based on adversarial domain adaptation and meta-reinforcement learning. This method draws on the idea of extracting domain-invariant features in traditional adversarial domain adaptation methods and improves on their classic network structure.
[0146] First, to prevent gradient conflicts between the value and policy networks, separate encoders are constructed for the adversarial domain adaptation module and the meta-reinforcement learning module.
[0147] Subsequently, in order to eliminate the impact of continuous changes in dynamic parameters on robot control, this paper uses an adversarial domain adaptation method under continuous indexing, so that the encoder of the adversarial domain adaptation module can capture the laws under continuous changes in dynamic parameters.
[0148] In addition, in order to better extract domain-invariant features, the meta-reinforcement learning module is divided into two parts: teacher and student. The student reinforcement learning model is responsible for extracting domain-invariant features and imitating the actions of the teacher reinforcement learning model, which makes feature extraction smoother.
[0149] Finally, in order to address the problem of too little data in the target domain, this paper uses a meta-optimization method to improve the virtual-to-real migration effect under few samples and optimizes the gradient direction to speed up training.
[0150] Through these four optimization points, the virtual-real migration problem in the case of continuous dynamics mismatch is solved.
[0151] The present embodiment provides a robot control method. The present application controls the robot in response to control instructions through a pre-constructed environment dynamics adaptation model. The environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework. The adversarial continuous dynamics domain alignment reinforcement learning framework is a model-independent reinforcement learning framework used for dynamics adaptive robot control, which can reduce the impact of various continuous dynamic changes (such as camera rotation, mass, robot links and surface friction) on robot performance.
[0152] For example, in order to help understand the implementation process of the robot control method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 2 , Figure 2 A flow chart of the robot control method of this application for training an environment dynamics adaptation model based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
[0153] like Figure 2 As shown, the present invention proposes an adversarial continuous dynamics domain alignment reinforcement learning framework, which combines adversarial domain adaptation with reinforcement learning to eliminate the impact of dynamics changes.
[0154] To achieve this goal, the framework has two key components: adversarial dynamics domain alignment model and teacher-student reinforcement learning. The upper left corner is the classic adversarial domain adaptation model and reinforcement learning model.
[0155] In addition, the framework also includes a meta-optimization part.
[0156] Among them, the adversarial domain adaptation model combines the policy network and value network encoders and the discriminator to extract domain-invariant features.
[0157] An adversarial dynamics domain alignment model is proposed based on the adversarial domain adaptation model.
[0158] The teacher-student reinforcement learning interacts with the environment and stores the trajectories in a buffer.
[0159] Student reinforcement learning utilizes a dynamics-invariant encoder to approximate teacher-student reinforcement learning and attempts to generate optimal actions.
[0160] Meta-optimization aligns the optimization losses of previous methods and enables dynamics adaptation with little information.
[0161] Finally, the policy network and value network of the student reinforcement learning and their encoder are deployed in a real environment.
[0162] The following are the specific steps for training the environment dynamics adaptation model based on the adversarial continuous dynamics domain alignment reinforcement learning framework:
[0163] S300, a teacher reinforcement learning model is constructed based on the proximal strategy optimization algorithm, and a student reinforcement learning model is constructed based on the teacher reinforcement learning model.
[0164] S301, controlling the robot to perform tasks in the training environment and the test environment respectively through the teacher reinforcement learning model, thereby obtaining the training environment trajectory data and the test environment trajectory data.
[0165] S302, storing the training environment trajectory data and the test environment trajectory data in a buffer for subsequent use.
[0166] S303, taking out part of the training environment trajectory data and part of the test environment trajectory data from the buffer as a support set, and extracting domain-invariant features from the support set through an adversarial dynamics domain alignment model proposed based on the adversarial domain adaptation model. The adversarial dynamics domain alignment model uses the policy network and the value network in the reinforcement learning algorithm as task regressors, and the adversarial dynamics domain alignment model also includes an encoder corresponding to the policy network, an encoder corresponding to the value network, and a continuous domain discriminator.
[0167] Specifically, the support set is extracted by an encoder corresponding to the policy network and an encoder corresponding to the value network in the adversarial dynamics domain alignment model to obtain a first intermediate feature and a second intermediate feature;
[0168] splicing the first intermediate feature and the second intermediate feature and inputting the concatenated features into a continuous domain discriminator of the adversarial dynamic domain alignment model;
[0169] By performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network and the continuous domain discriminator in the adversarial dynamics domain alignment model, the distribution difference between the source domain trajectory dataset and the target domain trajectory dataset in the support set is minimized to obtain the discriminator loss.
[0170] Step S304: Encoder E in the teacher reinforcement learning model a 、E c Instantiated as the meta-initial parameter φ e and φ φ , the discriminator loss is gradient descented through a meta-optimization method to update the meta-initial parameters to the first task-specific parameters.
[0171] Step S305, updating the encoder in the pre-constructed student reinforcement learning model based on the first task specific parameters.
[0172] Step S306, the query set is calculated using the updated student reinforcement learning model to obtain the similarity loss L sim .
[0173] Step S307, back-propagating and re-updating the encoder, policy network and value network in the student reinforcement learning model based on the similarity loss and the discriminator loss, and recording the re-updated student reinforcement learning model as the environmental dynamics adaptation model.
[0174] The present invention has been experimentally verified in a virtual environment. The experiment proves that the performance of the environmental dynamics adaptation model is better than the existing advanced algorithms. The Chinese and English names of the existing advanced algorithms are shown in the figure below. Figure 3 shown.
[0175] Reference Figure 4 , Figure 4 is the average total reward value of each algorithm under the Pendulum task.
[0176] Among them, according to Figure 4 It can be seen from the numerical values in that the performance of the environmental dynamics adaptation model proposed in the present invention is better than other algorithms in both the common target domain and the extreme target domain.
[0177] Reference Figure 5 , Figure 5 is the average total reward value of each algorithm under the half-cheetah task.
[0178] Among them, according to Figure 5 It can be seen from the numerical values in that the performance of the environmental dynamics adaptation model proposed in the present invention is better than other algorithms in the source domain, the common target domain and the extreme target domain.
[0179] Reference Figure 6 , Figure 6 is the average total reward value of each algorithm in the lame half-cheetah task.
[0180] Among them, according to Figure 6 It can be seen from the numerical values in that the performance of the environmental dynamics adaptation model proposed in the present invention is better than other algorithms in both the common target domain and the extreme target domain.
[0181] Reference Figure 7 , Figure 7 is the average total reward value of each algorithm under the Ant task.
[0182] Among them, according to Figure 7 It can be seen from the numerical values in that the performance of the environmental dynamics adaptation model proposed in the present invention is better than other algorithms in the source domain, the common target domain and the extreme target domain.
[0183] At the same time, the environmental dynamics adaptation model proposed in the present invention is also tested in a real environment. In the real environment, the position of the camera is different from that in the virtual environment to verify the adaptability of the environmental dynamics adaptation model when the settings of the virtual environment and the real environment are inconsistent. The results are as follows: Figure 8 As shown, under different camera offset angles, the environmental dynamics adaptation model proposed in the present invention uses fewer average steps to complete the task and has a higher overall success rate.
[0184] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot control method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0185] This application also provides a robot control device, please refer to Fig. 9 , the robot control device comprises:
[0186] The robot control module 10 is used to control the robot in response to the control instruction through a pre-constructed environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
[0187] The robot control device provided by the present application adopts the robot control method in the above embodiment, which can solve the technical problem that the existing research does not consider that the dynamic parameters are continuous values and does not try to extract the rules of the continuous changes of the dynamic parameters. Compared with the prior art, the beneficial effects of the robot control device provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, and the other technical features in the robot control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0188] The present application provides a robot control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robot control method in the above-mentioned embodiment one.
[0189] Reference below Fig.10 , which shows a schematic diagram of the structure of a robot control device suitable for implementing the embodiment of the present application. The robot control device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The robot control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0190] like Fig.10As shown, the robot control device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the robot control device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the robot control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a robot control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0191] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0192] The robot control device provided by the present application adopts the robot control method in the above embodiment, which can solve the technical problem that the existing research does not consider that the dynamic parameters are continuous values and does not try to extract the rules of the continuous changes of the dynamic parameters. Compared with the prior art, the beneficial effects of the robot control device provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, and the other technical features in the robot control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0193] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0194] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0195] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the robot control method in the above-mentioned embodiment.
[0196] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0197] The computer-readable storage medium may be included in the robot control device; or may exist independently without being assembled into the robot control device.
[0198] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the robot control device, the robot control device: responds to the control instruction and controls the robot through a pre-built environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on the adversarial continuous dynamic domain alignment reinforcement learning framework.
[0199] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0200] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0201] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0202] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned robot control method, and can solve the technical problem that the existing research does not consider that the dynamic parameters are continuous values and does not attempt to extract the rules of the continuous changes in the dynamic parameters. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the robot control method provided by the above-mentioned embodiment, and will not be repeated here.
[0203] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the robot control method as described above are implemented.
[0204] The computer program product provided by the present application can solve the technical problem that the existing research does not consider that the dynamic parameters are continuous values and does not attempt to extract the rules of the continuous changes of the dynamic parameters. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, which will not be repeated here.
[0205] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A robot control method, characterized in that: The method comprises: In response to control instructions, the robot is controlled by a pre-constructed environment dynamics adaptation model, which is trained based on an adversarial continuous dynamics domain alignment reinforcement learning framework.
2. The method according to claim 1, characterized in that The step of controlling the robot by a pre-built environmental dynamics adaptation model in response to the control instruction includes: Collect the trajectories of the robot using the pre-built teacher reinforcement learning model to perform tasks in the source domain and the target domain, respectively, and record them as the source domain trajectory dataset and the target domain trajectory dataset; Selecting trajectory data from the source domain trajectory dataset and the target domain trajectory dataset as support sets respectively; Selecting trajectory data from the source domain trajectory dataset as a query set; Performing domain-invariant feature extraction on the support set using a pre-built adversarial dynamics domain alignment model to obtain a discriminator loss; Instantiating the encoder in the teacher reinforcement learning model as meta-initial parameters; Performing gradient descent on the discriminator loss by a meta-optimization method to update the meta-initial parameters to first task-specific parameters; Updating a pre-built student reinforcement learning model based on the first task-specific parameters; Calculating the query set through the updated student reinforcement learning model to obtain a similarity loss; The encoder, the policy network and the value network in the student reinforcement learning model are back-propagated and updated again based on the similarity loss and the discriminator loss, and the updated student reinforcement learning model is recorded as the environment dynamics adaptation model.
3. The method according to claim 2, characterized in that The step of respectively collecting trajectories of controlling the robot to perform tasks in the source domain and the target domain using the pre-built teacher reinforcement learning model includes: The teacher reinforcement learning model is constructed based on the proximal strategy optimization algorithm; The student reinforcement learning model is constructed based on the teacher reinforcement learning model, and the strategy network and value network of the student reinforcement learning model are aligned with the strategy network and value network of the teacher reinforcement learning model.
4. The method according to claim 2, characterized in that The adversarial dynamics domain alignment model uses the policy network and the value network in the reinforcement learning algorithm as task regressors, and the adversarial dynamics domain alignment model also includes an encoder corresponding to the policy network, an encoder corresponding to the value network, and a continuous domain discriminator. The step of extracting domain-invariant features from the support set using the pre-built adversarial dynamics domain alignment model to obtain a discriminator loss comprises: Extracting the support set through an encoder corresponding to the policy network and an encoder corresponding to the value network in the adversarial dynamics domain alignment model to obtain a first intermediate feature and a second intermediate feature; Input the first intermediate feature and the second intermediate feature into the policy network and the value network of the adversarial dynamics domain alignment model respectively, or, concatenate the first intermediate feature and the second intermediate feature and then input them into the continuous domain discriminator of the adversarial dynamics domain alignment model; By performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network and the continuous domain discriminator in the adversarial dynamics domain alignment model, the distribution difference between the source domain trajectory dataset and the target domain trajectory dataset in the support set is minimized to obtain the discriminator loss.
5. The method according to claim 4, characterized in that The step of performing adversarial training between the encoder corresponding to the policy network, the encoder corresponding to the value network, and the continuous domain discriminator comprises: The loss of the task regressor is minimized through the encoder corresponding to the policy network and the encoder corresponding to the value network, the loss of the continuous domain discriminator is maximized, and the loss of the continuous domain discriminator is minimized through the continuous domain discriminator, thereby extracting domain invariant features.
6. The method according to claim 2, characterized in that The step of back-propagating based on the similarity loss and the discriminator loss and updating the encoder, the policy network and the value network in the student reinforcement learning model again comprises: updating the meta-initial parameters to second task-specific parameters based on the similarity loss and the discriminator loss; An encoder in the student reinforcement learning model is updated based on the second task-specific parameters.
7. A robot control device, characterized in that: The device comprises: The robot control module is used to control the robot in response to the control instruction through a pre-built environment dynamics adaptation model, wherein the environment dynamics adaptation model is trained based on the adversarial continuous dynamics domain alignment reinforcement learning framework.
8. A robot control device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the robot control method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Teacher consensus aggregation learning method based on random response differential privacy technology
CN112885468A
Reinforcement learning robot control method based on consistency constraint modeling and system thereof
CN113485107A
Cross-scene adaptive pedestrian trajectory prediction method and device
CN117011939A
Small sample robust imitation learning training method oriented to high-dimensional disturbance environment, electronic equipment and storage medium
CN117193008A
Long-term service elevator guide rail fault diagnosis method based on transfer learning and data driving
CN117312962A
Cited By
Robot hybrid intelligent self-adaptive control method and system adopting reinforcement learning
CN120715906A
