Offline reinforcement learning training method, action prediction method, device and medium

Abstract

The present application discloses an offline reinforcement learning training method, an action prediction method, a device and a medium, wherein the method comprises: obtaining a first static data set of a robot controlled by a model to be trained; the first static data set comprises a plurality of historical running trajectories of the robot; the network of the model to be trained comprises a value function network, a diffusion model network and an inverse dynamics network; training the value function network based on the first static data set; generating performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network; training the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label. The method and device provided by the present application improve the learning accuracy of the diffusion model, and improve the training efficiency and prediction accuracy of the model to be trained.

Description

Technical Field

[0001] The present application relates to the technical field of deep reinforcement learning, and more specifically, to an offline reinforcement learning training method, an action prediction method, a device and a medium. Background Art

[0002] Offline reinforcement learning can obtain effective action prediction strategies from pre-collected large-scale datasets. Unlike online reinforcement learning, which requires continuous exploration in a real-time environment, offline reinforcement learning uses existing data and avoids the costs associated with interacting with the real world under an exploration strategy. Therefore, offline reinforcement learning is often used to pre-train robot control strategies, autonomous driving and other fields, but the performance of models trained by related offline reinforcement learning methods is limited by the quality of pre-collected datasets, and it is difficult to simulate diverse behaviors in complex datasets.

[0003] As an emerging deep generative model, the diffusion model has shown strong expressive power in the field of computer vision. This ability has also been borrowed from offline reinforcement learning to characterize policy distribution. The diffusion model is used to model the behavior strategy, target strategy, trajectory distribution, or Markov decision process tuple distribution in the data set, and can simulate diverse behaviors in complex data sets. However, the diffusion model has difficulty distinguishing data of different qualities in the data set, resulting in low prediction accuracy.

[0004] Therefore, how to introduce the diffusion model into the model to be trained to train the model and improve the prediction accuracy of the model to be trained has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The present application provides an offline reinforcement learning training method, an action prediction method, a device and a medium to solve the technical problem in the prior art of how to introduce a diffusion model into a model to be trained to train the model to be trained and improve the prediction accuracy of the model to be trained.

[0006] In a first aspect, the present application provides an offline reinforcement learning training method, comprising:

[0007] Acquire a first static data set of a robot controlled by a to-be-trained model; the first static data set includes a plurality of historical running trajectories of the robot; the network of the to-be-trained model includes a value function network, a diffusion model network and an inverse dynamics network;

[0008] Training the value function network based on the first static data set;

[0009] Generate performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network;

[0010] The diffusion model network and the inverse dynamics network are trained based on each trajectory segment and each performance label.

[0011] In some embodiments, the historical running trajectory includes the state and action of the robot at each historical moment, and the environmental reward of the robot interacting with the environment; the trajectory segment is obtained based on the following steps:

[0012] Determine the value of each state based on the trained value function network;

[0013] preprocessing the first static data set based on the Gaussian distribution of the value;

[0014] Extracting a first historical running trajectory from the preprocessed first static data set, wherein the first historical running trajectory is any unextracted historical running trajectory;

[0015] Any state in the first historical running trajectory, the state after any state within a preset time interval, the action and the environmental reward are spliced ​​to obtain the trajectory segment.

[0016] In some embodiments, the generating of the performance labels corresponding to the respective trajectory segments of the historical running trajectory based on the trained value function network includes:

[0017] Use the trained value function network as the first value function network;

[0018] Inputting the terminal state of any trajectory segment into the first value function network to obtain the value corresponding to the terminal state output by the first value function network;

[0019] A performance label of any trajectory segment is determined based on the environmental rewards corresponding to each state in any trajectory segment and the value corresponding to the terminal state.

[0020] In some embodiments, the training of the value function network based on the first static data set includes:

[0021] Collecting the state and environment reward of the robot from the first static data set;

[0022] Based on the collected state and environmental rewards, the loss function of the value function network is updated, and the network parameters of the value function network are updated;

[0023] When the training result meets the first preset termination condition, the training is terminated to obtain the trained value function network;

[0024] Among them, the first preset termination condition includes being greater than or equal to a preset number of training times, or the loss function of the value function network converges; the loss function of the value function network is constructed based on the state in the first static data set, the environmental reward and the hyperparameters of the value function network.

[0025] In some embodiments, after obtaining the trajectory segment, the method further includes:

[0026] generating a second static data set based on the trajectory segments and the respective performance labels;

[0027] The step of training the diffusion model network based on each trajectory segment and each performance label includes:

[0028] Randomly collecting each trajectory segment and each performance label corresponding to each trajectory segment from the second static data set;

[0029] Extracting state trajectory segments from the trajectory segments, and updating the loss function of the diffusion model network and the loss function of the inverse dynamics network based on the state trajectory segments and the performance labels, and updating the network parameters of the diffusion model network and the network parameters of the inverse dynamics network;

[0030] When the training result meets the second preset termination condition, the training is terminated;

[0031] Among them, the second preset termination condition includes being greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the diffusion model network is constructed based on the trajectory segment, the performance label corresponding to the trajectory segment and the hyperparameters of the diffusion model network; the loss function of the inverse dynamics network is constructed based on the state, action in the trajectory segment and the hyperparameters of the inverse dynamics network.

[0032] In some embodiments, extracting the state trajectory segment from the trajectory segment includes:

[0033] Extract the states in each trajectory segment sequentially based on the time order;

[0034] The states in the extracted trajectory segments are sequentially spliced ​​to obtain the state trajectory segment.

[0035] In a second aspect, the present application provides an action prediction method, comprising:

[0036] Inputting the state of the robot in the current prediction environment into the value function network of the action prediction model to obtain the value corresponding to the state output by the value function network;

[0037] Generate a first state trajectory segment based on the Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise;

[0038] Replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment;

[0039] Inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments;

[0040] The trajectory feasibility of each trajectory segment in the state trajectory segment set is combined into a trajectory feasibility set;

[0041] Based on the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set, the state is input into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network;

[0042] Wherein, the action prediction model is trained based on the above-mentioned offline reinforcement learning training method.

[0043] In a third aspect, the present application provides an offline reinforcement learning training device, comprising:

[0044] An acquisition module, used for acquiring a first static data set of a robot controlled by a to-be-trained model; the first static data set includes a plurality of historical running trajectories of the robot; the network of the to-be-trained model includes a value function network, a diffusion model network and an inverse dynamics network;

[0045] A first training module, used for training the value function network based on the first static data set;

[0046] A label module, used for generating a performance label corresponding to each trajectory segment of the historical running trajectory based on the trained value function network;

[0047] The second training module is used to train the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label.

[0048] Fourthly, the present application provides an action prediction device, comprising:

[0049] A value module, used to input the state of the robot in the current prediction environment into the value function network of the action prediction model, and obtain the value corresponding to the state output by the value function network;

[0050] A generating module, configured to generate a first state trajectory segment based on the Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise;

[0051] A replacement module, used for replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment;

[0052] A collection module, used for inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments;

[0053] A forming module, used for forming a trajectory feasibility set by forming the trajectory feasibility of each trajectory segment in the state trajectory segment set;

[0054] An action module is used to input the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network;

[0055] Wherein, the action prediction model is trained based on the above-mentioned offline reinforcement learning training method.

[0056] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when executed by a processor.

[0057] The offline reinforcement learning training method, action prediction method, device and medium provided in the present application train the value function network through the first static data set, and can evaluate the quality of historical operation trajectories through the value function, and guide the diffusion model to identify the quality of the trajectories; by introducing the diffusion model network into the network of the model to be trained, complex trajectories can be modeled; by training the diffusion model network and the inverse dynamics network through trajectory fragments and performance labels, the diffusion model can be assisted in distinguishing the quality of trajectories, thereby improving the learning accuracy of the diffusion model, and improving the training efficiency and prediction accuracy of the model to be trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 One of the flow charts of the offline reinforcement learning training method provided in the embodiment of the present application;

[0061] Figure 2 The second flowchart of the offline reinforcement learning training method provided in the embodiment of the present application;

[0062] Figure 3 One of the flow charts of the action prediction method provided in the embodiment of the present application;

[0063] Figure 4 The second flowchart of the action prediction method provided in the embodiment of the present application;

[0064] Figure 5 A schematic diagram of the structure of an offline reinforcement learning training device provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of the action prediction device provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0068] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or equipment.

[0069] The offline reinforcement learning training method provided in the embodiment of the present application is applicable to a terminal, which may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to servers, smart phones, tablet computers, laptop computers, and desktop computers.

[0070] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0071] Figure 1 One of the flow charts of the offline reinforcement learning training method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes step 110, step 120, step 130 and step 140. The steps of the method flow are only a possible implementation of the present application.

[0072] Step 110, obtaining a first static data set of the robot controlled by the model to be trained; the first static data set includes multiple historical operation trajectories of the robot; the network of the model to be trained includes a value function network, a diffusion model network and an inverse dynamics network.

[0073] Specifically, the execution subject of the offline reinforcement learning training method provided in the embodiment of the present application is an offline reinforcement learning training device, which can be a hardware device independently set in the terminal, or a software program running in the terminal. For example, when the terminal is a desktop computer, the offline reinforcement learning training device can be embodied as an application such as training software in the desktop computer.

[0074] The model to be trained is a model using an offline reinforcement learning system, and the model to be trained can be used to predict the actions of the robot. The first static data set is a set of historical running trajectories of the robot controlled by the model to be trained.

[0075] The historical running trajectory can include the robot's state and action at each historical moment, as well as the environmental rewards of the robot's interaction with the environment.

[0076] The network of the model to be trained includes a value function network, a diffusion model network, and an inverse dynamics network. The value function network is used to train the strategy of the diffusion model. The diffusion model network is the strategy network represented by the diffusion model. The inverse dynamics network is used to analyze the state of the diffusion model output and convert the state into an action that the robot can perform.

[0077] Step 120: Train the value function network based on the first static data set.

[0078] Specifically, the value function can evaluate the value of the trajectory, and the value estimated by the value function can be put into the diffusion model to guide the diffusion model to identify the quality of the trajectory, thereby learning the correct running trajectory and improving the training efficiency and accuracy of the model to be trained.

[0079] The Adam optimizer can be used to update the parameters of the value function network according to the state of the robot and the environmental reward in the first static data set, thereby obtaining a value function network with excellent performance.

[0080] Step 130: Generate performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network.

[0081] Specifically, the trajectory of the robot in the first static data set is divided into trajectory segments of fixed length, and the performance labels corresponding to each trajectory segment of the historical running trajectory are generated according to the trained value function network.

[0082] Performance labels are identifiers that can identify the quality of a trajectory.

[0083] Step 140: training the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label.

[0084] Specifically, the values ​​of all robot states can be generated based on the trained value function network, and the value distribution of all states can be counted. The values ​​can be fitted using Gaussian distribution, and the mean and variance of the Gaussian distribution can be recorded. The accuracy of the actions and states generated by the diffusion model can be determined by Gaussian distribution.

[0085] Diffusion models can recognize the multimodality of behavior, that is, multiple different actions can be generated in the same state, and model them.

[0086] The diffusion model network has a strong generation capability, can effectively characterize multi-peak Gaussian distribution, and has a good fitting ability for the complex trajectory space in the data set. Using the diffusion model network to model the robot's trajectory fragments during the training process of the model to be trained can effectively improve the accuracy of future trajectory prediction.

[0087] The Adam optimizer can be used to iteratively update the parameters of the diffusion model network and the inverse dynamics network.

[0088] The offline reinforcement learning training method provided in the embodiment of the present application trains the value function network through the first static data set, and can evaluate the quality of historical operation trajectories through the value function, and guide the diffusion model to identify the quality of the trajectories; by introducing the diffusion model network into the network of the model to be trained, complex trajectories can be modeled; by training the diffusion model network and the inverse dynamics network through trajectory fragments and performance labels, the diffusion model can be assisted in distinguishing the quality of trajectories, thereby improving the learning accuracy of the diffusion model, and improving the training efficiency and prediction accuracy of the model to be trained.

[0089] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0090] In some embodiments, the trajectory segment is obtained based on the following steps:

[0091] Determine the value of each state based on the trained value function network;

[0092] preprocessing the first static data set based on a Gaussian distribution of values;

[0093] Extracting a first historical running trajectory from the preprocessed first static data set, where the first historical running trajectory is any unextracted historical running trajectory;

[0094] Any state in the first historical running trajectory, the state after any state within a preset time interval, the action and the environmental reward are spliced ​​to obtain a trajectory segment.

[0095] After obtaining the trajectory fragment, it also includes:

[0096] A second static dataset is generated based on the trajectory segments and the respective performance labels.

[0097] Specifically, a trajectory of a robot that has not been extracted is extracted from the first static data set, that is, the first historical running trajectory is extracted. For each state of the robot in the trajectory, the state of the robot at a fixed length of time, the action of the robot at a fixed length of time, and the environmental reward at a fixed length of time are spliced ​​to obtain a trajectory segment. This step can be iteratively performed until each trajectory in the first static data set is divided into trajectory segments. The length of the data after splicing within the preset time interval is fixed, so the length of each trajectory segment is the same.

[0098] The terminal state in the trajectory segment is input into the value function network to calculate the value of the terminal state of the trajectory, and the performance label of the trajectory segment is calculated according to the reward corresponding to the state in the trajectory segment and the value of the terminal state. The trajectory segment and the performance label are added to the second static data set.

[0099] The offline reinforcement learning training method provided in the embodiment of the present application improves the training efficiency of the model to be trained by dividing the trajectory into trajectory segments of fixed length.

[0100] In some embodiments, generating performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network includes:

[0101] Use the trained value function network as the first value function network;

[0102] Input the terminal state of any trajectory segment into the first value function network to obtain the value corresponding to the terminal state output by the first value function network;

[0103] The performance label of any trajectory segment is determined based on the environmental rewards corresponding to each state in any trajectory segment and the value corresponding to the terminal state.

[0104] Specifically, reasonable training conditions are the key to generating expected data using a diffusion model. Therefore, the embodiment of the present application models the state trajectory fragments through a diffusion model, and therefore it is necessary to find suitable training conditions to guide the reverse diffusion process of the diffusion model.

[0105] Since the goal of offline reinforcement learning is to maximize the expected reward, the return It is generally used as a condition for trajectory modeling to mark the quality of the trajectory. Where n is the initial time, t is the time, r is the environmental reward, N is the trajectory length, τ is the number of trajectory segments, and γ is the discount factor.

[0106] Since the model does not cover the complete trajectory but only a trajectory fragment, it is not a reasonable choice to use the reward value as the performance label to evaluate the trajectory fragment. First, when the same trajectory fragment appears in multiple trajectories, the results of its reward may be different in different trajectories, resulting in relatively large variance, making it difficult to learn the guided relationship. Second, the evaluation of the current trajectory fragment may be affected by the subsequent trajectory behavior. Due to the uncertainty of the behavioral strategy when collecting the dataset, the strategy may perform well in the early stage, but be affected by the randomness of the strategy in the later stage, resulting in poor trajectory quality. In this case, using the reward R(τ) will affect the evaluation of the excellent trajectory fragments in the early stage. Similarly, the trajectory fragments with poor quality may also be overestimated. Therefore, accurate evaluation of trajectory fragments becomes the key to sequence modeling methods and the prerequisite for achieving excellent sub-trajectory fragment splicing.

[0107] To solve the above problems, this application adopts a more accurate trajectory segment evaluation method based on n-step returns to calculate the performance label G(τ) of the state trajectory segment.

[0108] Input the terminal state of any trajectory segment into the first value function network to obtain the value corresponding to the terminal state; determine the performance label of any trajectory segment based on the environmental rewards corresponding to each state in any trajectory segment and the value corresponding to the terminal state. The performance label of the trajectory segment can be obtained by adding the environmental rewards corresponding to all states in the trajectory segment and the value of the terminal state. The performance label calculation formula is as follows:

[0109]

[0110] Among them, h is the length of the sliding window, V is the state value, and s t and r t+1 are the robot's state and environment reward at time t, s t+1 is the state of the robot at time t+1.

[0111] By introducing the state value V(s) in G(τ) n+h ) to reduce the high variance of different subsequent trajectory segments on the same trajectory segment. At the same time, this value reflects the evaluation performance of the current policy in that state, thereby solving the problem of overestimation or underestimation of trajectory segments caused by behaviors in subsequent trajectory segments. In addition, the n-step return retains the reward for each step in the trajectory segment, allowing for a more accurate evaluation of the current trajectory segment.

[0112] The offline reinforcement learning training method provided in the embodiment of the present application can obtain accurate performance labels corresponding to each trajectory segment, thereby improving the learning accuracy of the diffusion model and improving the prediction accuracy of the model to be trained. The state trajectory is modeled using the diffusion model, and the modeling uses the reward corresponding to the state in the trajectory segment and the value of the terminal state as conditions to characterize the performance of the generated trajectory segment, thereby avoiding underestimation of the better trajectory segments in the static data set during the modeling process, and promoting the model to make better use of the good segments in the suboptimal trajectory in the data set.

[0113] This application aims to complete the splicing of suboptimal trajectories in the data set and the stable control task of the robot under the condition of offline training with pre-collected data sets. In order to solve the problem of splicing better sub-segments in the suboptimal trajectories in the data set, this application combines the rewards and values ​​corresponding to the states in the trajectory to jointly measure the performance of the trajectory segments. In order to solve the problem that the inherent instability of the diffusion model leads to a certain randomness in the generated trajectory, which affects the stable control of the robot.

[0114] The value function network based on the expected regression function as the loss function can be used to estimate the value of the robot's state, that is, implicit value function learning. This loss function can avoid using states that do not exist in the data set to estimate the optimal value function of the state during the process of learning the value function, thereby alleviating the problem of overestimation of the value of the state caused by offline reinforcement learning in the process of learning the optimal value function.

[0115] In some embodiments, step 120 includes:

[0116] Collecting the robot's state and environment reward from the first static dataset;

[0117] Update the loss function of the value function network based on the collected state and environmental rewards, and update the network parameters of the value function network;

[0118] When the training result meets the first preset termination condition, the training is terminated to obtain the trained value function network;

[0119] Among them, the first preset termination condition includes greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the value function network is constructed based on the state, environmental rewards and hyperparameters of the value function network in the first static data set.

[0120] Specifically, the robot's state and environment reward are randomly sampled from the first static data set, and the state and environment reward are used to update the loss function of the value function network, thereby updating the parameters of the value function network.

[0121] The optimization of parameters can use the RMSProp optimizer. For each weight parameter in the value function network, the update rule formula is as follows:

[0122]

[0123] Among them, φ is the weight parameter of the value function network, is φ at time t t The gradient of φ is the learning rate of the optimizer, its value can be set to 0.0001, and i is the time number.

[0124] The training process of the embodiment of the present application can be iteratively executed at least once until a preset number of training times is reached or the loss function of the value function network converges, that is, the training is terminated when the training result meets the first preset termination condition to obtain a trained value function network.

[0125] Loss function of the value function network for:

[0126]

[0127] Among them, E represents the The expectation under the condition that a is the action of the robot, That is, the expected regression function, u=(r t+1 +γV φ′ (s t+1 )-V φ (s t ))、 is the indicator function, when u(r t+1 +γV φ′ (s t+1 )-V φ (s t )<0, it takes 1, otherwise it takes 0, the hyperparameter λ takes 0.7, V φ (s t ) is to convert s t Input to the value function network to calculate the output, V φ′ (s t+1 ) is to convert s t The output calculated by inputting into the target value function network is updated by soft update, that is, each time the value function network V is updated φ After the weight parameter φ, let the target value network V φ′ The weight parameter φ′ is:

[0128] φ′ t+1 =(1-κ)φ′ t +κφ t+1 ;

[0129] where κ is set to 0.005, φ′ t+1 The target value network V φ′ The weight parameter at time t+1, φt t The target value network V φ′ The weight parameter at time t, φ′ t The target value network V φ′ The weight parameter at time t.

[0130] The offline reinforcement learning training method provided in the embodiment of the present application improves the training efficiency of the value function network by iteratively updating the loss function of the value function network.

[0131] In some embodiments, step 140 includes:

[0132] Randomly collect each trajectory segment and each performance label corresponding to each trajectory segment from the second static data set;

[0133] Extracting state trajectory segments from the trajectory segments, and updating the loss function of the diffusion model network and the loss function of the inverse dynamics network based on the state trajectory segments and each performance label, and updating the network parameters of the diffusion model network and the network parameters of the inverse dynamics network;

[0134] When the training result meets the second preset termination condition, the training is terminated;

[0135] Among them, the second preset termination condition includes being greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the diffusion model network is constructed based on the trajectory segments, the performance labels corresponding to the trajectory segments, and the hyperparameters of the diffusion model network; the loss function of the inverse dynamics network is constructed based on the states and actions in the trajectory segments and the hyperparameters of the inverse dynamics network.

[0136] Extract the state trajectory fragment from the trajectory fragment, including:

[0137] Extract the states in each trajectory segment sequentially based on the time order;

[0138] The states in each extracted trajectory segment are spliced ​​in sequence to obtain a state trajectory segment.

[0139] Specifically, the robot's trajectory segments and the performance labels corresponding to the trajectory segments are randomly sampled from the second static data set. The state trajectory segments are extracted from the trajectory segments, and the loss function of the diffusion model network is updated according to the state trajectory segments and the corresponding performance labels, and the loss function of the inverse dynamics network is updated according to the trajectory segments, and the parameters of the diffusion model network and the inverse dynamics network are updated.

[0140] The diffusion model is a powerful generative model that generates data through a continuous denoising process. The model includes two processes: forward denoising and reverse denoising. In the forward denoising process, by continuously injecting noise into the original data, the original data can be made close to Gaussian noise. This process can be modeled as a Markov chain with a length of K, represented by x 0:K Each step is determined by the conditional probability p(x k |x k-1 ) definition, where k∈[1,K],x 0 ~p data (x) is from the dataset data, α 0:k is a hyperparameter that controls the strength of the variance, represents a Gaussian distribution with mean μ and variance Σ, and I represents a diagonal matrix with the same dimension as x. The denoising process is parameterized by a Markov chain, and the formula is as follows:

[0141]

[0142] Among them, q represents the denoising process distribution, θ is the parameter of the diffusion model, and the entire denoising process starts from random noise, that is, By iteratively sampling from the Gaussian distribution, the noise in the data is eliminated until the original data is reconstructed. In the denoising process, neural networks are generally not used directly to estimate μ θ (), but for the noise term ∈ θ At each step in the reverse diffusion process, the mean of the Gaussian distribution can be obtained by the following formula:

[0143]

[0144] in, The iterative denoising process can be achieved by optimizing the following loss function To noise∈ θ Make an estimate:

[0145]

[0146] In order to control the quality of the generated trajectory, a diffusion model network without classifier guidance can be used to model the conditional sequence of the trajectory. The output of the value function is y, and different state trajectories s(τ) = {s t ,s t+1 ,...,s t+h-1}, that is, the training objective formula of the model is as follows:

[0147]

[0148] Therefore, we first need to learn a conditional noise network ∈ θ (s k (τ),G(τ),k), where in the conditional noise network ∈ θ In the training process, s0(τ) is the state trajectory segment corresponding to the trajectory segment randomly sampled from the second data set. In the iterative denoising process, the reconstructed state trajectory can be generated by adding the disturbed Gaussian noise. The formula is as follows:

[0149]

[0150] Where w is the guidance strength, the time step is k~{1,...,K}, G(τ) is the performance label corresponding to the trajectory segment τ, Indicates that the condition is empty. The perturbed noise term is called the conditional noise term reconstructed according to the noise term after the introduction of the unclassified guide. Therefore, the mean of the Gaussian distribution in the iterative denoising process of the diffusion model can be rewritten as follows:

[0151]

[0152] It refers to the new mean of the Gaussian distribution in the iterative denoising process obtained by reconstructing the noise term after the introduction of the unclassified guide. Through iterative denoising, the diffusion model network will generate the expected state trajectory in the action prediction process.

[0153] Since the action is a high-frequency signal, the effect of directly using the diffusion model to generate the corresponding action sequence while generating the state trajectory is not ideal. Therefore, an inverse dynamics network is used to generate the robot's action. By inputting the current state s in the state trajectory into the network t and the state s at the next moment t+1 , then the state s can be generated t The corresponding action a t .

[0154] Loss function of the diffusion model network and the loss function of the inverse dynamics network The formulas are as follows:

[0155]

[0156]

[0157] Among them, f Φ (s t ,s t+1 ) means that the state s t and status t+1The output action, ∈, obtained by inputting into the inverse dynamics network is the noise sampled from the Gaussian distribution, s k (τ) is the state trajectory segment obtained by adding noise ∈ to the state trajectory segment in the trajectory segment τ, is the predicted noise of the kth step output by the conditional noise network in the diffusion model network, Bern(p) is the Bernoulli distribution with parameter p, and β is the result of sampling from the distribution.

[0158] The embodiment of the present application updates the parameters of the diffusion model network and the inverse dynamics network based on the Adam optimizer.

[0159] For each weight parameter in the diffusion model network and the inverse dynamics network, the update rule is:

[0160]

[0161] Among them, (θ t ,Φ t ) represents the parameters in the diffusion model network and the inverse dynamics network, Indicates that at time t (θ t ,Φ t ); η is the learning rate of the Adam optimizer, and its value can be set to 0.0002.

[0162] The above training process is iterated at least once until a preset number of training times is reached or the loss functions of the diffusion model network and the inverse dynamics network converge, that is, the training is terminated when the training result meets the second preset termination condition.

[0163] The offline reinforcement learning training method provided in the embodiment of the present application updates the network parameters of the diffusion model network and the network parameters of the inverse dynamics network through the trajectory segments in the second static data set, the performance labels corresponding to the trajectory segments, and the state trajectory segments, thereby improving the training efficiency of the model to be trained.

[0164] Figure 2 The second flowchart of the offline reinforcement learning training method provided in the embodiment of the present application is as follows: Figure 2 As shown, the method includes:

[0165] Step 210, defining the state s of the robot in the offline reinforcement learning system of the model to be trained t With action a t , where t is the current time.

[0166] This step is the initialization step of the environment. Take the robot operating environment as the robot arm grasping environment as an example. The robot's state in the environment s tThe information included includes: the coordinates of the grasped target in the environment, the coordinates of the target placement position, the position and posture of the robotic arm and the end effector, and the opening and closing status of the end effector.

[0167] The operating environment obtains the next state of the robot according to the action selected by the offline reinforcement learning system, and feeds back environmental rewards to the offline reinforcement learning system.

[0168] The value function network corresponding to the offline reinforcement learning system is used to calculate the state value of the robot. The diffusion model network corresponding to the offline reinforcement learning system predicts the state trajectory of the robot at a future moment, and the state trajectory of the robot can be generated by inputting the trajectory fragment and the value of the robot state.

[0169] The inverse dynamics network corresponding to the offline reinforcement learning system is used to calculate the robot's actions by inputting the state s corresponding to the current time t in the predicted state trajectory segment t And the predicted state s at the next moment t+1 Appropriate actions can be output for the robot.

[0170] In this embodiment, the robot's action space includes two action subspaces, namely, the movement action a of the end of the robot arm and i,1 The action of whether the end effector of the robot arm is open or closed i,2 .

[0171] Step 220, initialize the weight parameter φ of the value function network, the weight parameter θ of the diffusion model network, the weight parameter Φ of the inverse dynamics network, and the weight parameter φ′ of the target value network, which is used to assist the training of the value function network.

[0172] This step is the initialization step of the algorithm.

[0173] Step 230, randomly extract the state and action (s t ,a t ,r t+1 ,s t+1 ).

[0174] Step 240, calculate the loss function according to the state and environment reward, train the value function network according to the loss function and update its parameters.

[0175] Step 250, determine whether the training termination condition is met, that is, whether the training result meets the first preset termination condition. If so, execute step 260, otherwise repeat steps 230-250.

[0176] Step 260: Count all adjacent states s of the robot in the first static data set t ,st+1 The value difference V(s t+1 )-V(s t ), use Gaussian distribution to fit it, and record the mean μ of the Gaussian distribution d and variance σ d The first static dataset is preprocessed, and the complete trajectory of the robot interacting with the environment is divided into trajectory segments of fixed length h to generate the second static dataset, and the value function and the reward value in the trajectory segment are used to calculate the corresponding performance label for each trajectory segment.

[0177] In this embodiment, for a complete trajectory {s0, a0, r0, s1, a1, r1, ..., s N ,a N ,r N}, use a sliding window of length h to divide the complete trajectory into the following trajectory segments: {s0, a0, r0, …, s h ,a h ,r h}, {s1,a1,r1,…,s h+1 ,a h+1 ,r h+1}, …, {s N-h ,a N-h ,r N-h ,…,s N ,a N ,r N},{s N-h+1 ,a N-h+1 ,r N-h+1 ,…,s N ,a N ,r N The above operation is performed on each trajectory in the first static data, and the corresponding performance label G(τ) is calculated for each trajectory segment using the following formula to obtain a second static data set including all trajectory segments and the performance labels corresponding to all trajectory segments.

[0178] Step 270 , randomly extracting trajectory segments of the robot and performance labels corresponding to the trajectory segments in the second static data set.

[0179] Step 280 , respectively calculating the loss functions of the diffusion model network and the inverse dynamics network according to the trajectory segments and the performance labels, and training the diffusion model network and the inverse dynamics network according to the loss functions and updating their parameters.

[0180] Step 290, determine whether the training termination condition is met, that is, whether the training result meets the second preset termination condition. If so, execute step 2110, otherwise repeat steps 270-290.

[0181] Step 2110, the training is finished, and the parameters of the value function network, diffusion model network and inverse dynamics network updated for the last time are obtained as the training results.

[0182] In this embodiment, the preset number of training times in the first preset termination condition is the number of times the value network parameters are updated, and the preset number of training times may be 100,000 times. The preset number of training times in the second preset termination condition is the number of times the network parameters of the diffusion model network and the inverse dynamics network are updated, and the preset number of training times may be 500,000 times, that is, the training stops after the two networks are updated 500,000 times respectively. After the training is completed, the parameters of the value function network, the diffusion model network, and the inverse dynamics network are retained to obtain a model to be trained that can perform action prediction.

[0183] The present application defines the state and action space of a robot in an offline reinforcement learning system based on the robot's operating environment; initializes the weight parameters of each neural network; uses an implicit value function learning method to learn the robot's state value function, thereby alleviating the problem of overestimating the value of the state in the process of learning the optimal value function in offline reinforcement learning; samples data from the first static data, trains the value function network and updates its parameters; repeats the above until a preset number of training times is reached; divides the complete trajectory of the robot in the first static data set into trajectory segments of fixed length, generates a second static data set, and annotates the performance of the trajectory segments therein; in the annotation process, a more accurate trajectory segment evaluation method based on n-step returns is used to calculate the performance label of the state trajectory segment, so that the algorithm can more accurately model the performance of the trajectory segment, thereby improving the algorithm's ability to splice suboptimal trajectories; uses a diffusion model to conditionally model the trajectory segments in the second data, thereby effectively characterizing the complex trajectory space; samples data from the second static data, trains the diffusion model network and the inverse dynamics network and updates their parameters; repeats the above until a preset number of training times is reached. This application provides a more accurate estimate of the performance of trajectory segments in the dataset, enhances the stitching problem of suboptimal trajectories in the offline dataset, and enables the model to better model the trajectory space.

[0184] Figure 3 One of the flow charts of the action prediction method provided in the embodiment of the present application is as follows: Figure 3 As shown, the method includes steps 310 to 360. The steps of the method flow are only a possible implementation of the present application.

[0185] Step 310: Input the state of the robot in the current prediction environment into the value function network of the action prediction model to obtain the value corresponding to the state output by the value function network;

[0186] Step 320: Generate a first state trajectory segment based on the Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise;

[0187] Step 330: Replace the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment;

[0188] Step 340: input the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments;

[0189] Step 350, combining the trajectory feasibility of each trajectory segment in the state trajectory segment set into a trajectory feasibility set;

[0190] Step 360: input the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network;

[0191] Among them, the action prediction model is trained based on the above-mentioned offline reinforcement learning training method.

[0192] Specifically, the action prediction model is a model used to predict the action of the robot.

[0193] Get the state of the robot in the current environment. Input the state of the robot into the value function network to calculate the value of the robot's state. Generate a fixed-length first state trajectory segment composed of Gaussian noise.

[0194] The robot's historical state is used to replace the corresponding state in the initial state trajectory segment to obtain a second state trajectory segment. The value of the second state trajectory segment and the robot's state is input into the diffusion model network to calculate multiple different third state trajectory segments, and the multiple different third state trajectory segments are combined into a predicted state trajectory segment set.

[0195] Extract a third state trajectory segment whose trajectory feasibility has not been calculated in the state trajectory segment set, calculate the state feasibility and state value of each state for each state in the third state trajectory segment, and calculate the trajectory feasibility of the third state trajectory segment according to the state feasibility and state value of each state. Iterate this step until the trajectory feasibility of each trajectory segment in the state trajectory segment set is obtained, and form the trajectory feasibility of each trajectory segment in the state trajectory segment set into a trajectory feasibility set.

[0196] The third state trajectory segment corresponding to the feasibility maximum value in the trajectory feasibility set is obtained as the fourth state trajectory segment. The state in the fourth state trajectory is obtained, and the state is input into the inverse dynamics network to calculate the action of the robot.

[0197] Each state and the next state of each state in the fourth state trajectory can be input into the value function network respectively to calculate the value of each state and the value of the next state. The state feasibility of each state is calculated according to the value of each state, the value of the next state and the mean and variance of the Gaussian distribution.

[0198] Figure 4 The second flow chart of the action prediction method provided in the embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0199] Step 410: Get the robot's state s at the current time t t .

[0200] Step 420: Set the robot's state s t Input into the value function network to calculate the state s t The value of V(s t ).

[0201] Step 430: From Gaussian distribution Sampling is performed to generate a state trajectory segment of length h composed of Gaussian noise

[0202] Step 440: Replace the trajectory segment s with the robot's historical state K (τ) to obtain the corresponding state in the second state trajectory segment. Specifically, the length of the historical state to be replaced each time is m=C, where the historical state sequence is {s t-m+1 ,s t-m+2 ,…,s t}, get the state trajectory fragment If the observed historical state sequence {s0,s1,…,s t} length is less than C, then let m = t, and get the state trajectory segment

[0203] In this embodiment, a historical state sequence {s t-m+1 ,s t-m+2 ,…,s t} to record the robot's historical information, each time the latest observation s t Insert into this sequence.

[0204] Step 450: convert the state trajectory segment s K(τ) Copy b copies: {s K,1 (τ),s K,2 (τ),…,s K,b (τ)}, and then the b state trajectories and the robot's state value V(s t ) is input into the diffusion model network to calculate b different state trajectory segments {s 0,1 (τ),s 0,2 (τ),…,s 0,b (τ)}, these state trajectory segments are combined into a set of predicted state trajectory segments.

[0205] In this embodiment, for each state trajectory s K,i (τ) performs the denoising process to obtain the state trajectory s 0,i (τ), where i ~ [1, b], the initial value of k is K. Through the noise network, using the condition V (s t ) Calculate the disturbed Gaussian noise. In the action prediction process, since the subsequent state and reward cannot be obtained, the n-step return used in the training phase cannot be used as a condition to guide the generation of the diffusion process. Therefore, the value of the current state is used to approximate the n-step return to guide the generation of the state trajectory in the diffusion process. Use Gaussian distribution Sampling to obtain the state trajectory s k-1,i (τ), where the mean of the Gaussian distribution is calculated as follows:

[0206]

[0207] Replace the trajectory segment s with the robot's historical state k,i (τ) corresponds to the state in the table. Specifically, the length of the historical state to be replaced each time is m=C, where the historical state sequence is {s t-m+1 ,s t-m+2 ,…,s t}, get the state trajectory fragment If the observed historical state sequence {s0,s1,…,s t} length is less than C, then let m = t, and get the state trajectory segment

[0208] Let k = k-1, repeat the above steps until k = 0, and get the predicted state trajectory segment s 0,i (τ).

[0209] Step 460: extract a state trajectory segment s from the state trajectory segment set whose trajectory feasibility has not been calculated. 0,i (τ).

[0210] Step 470: For each state in the state trajectory segment, calculate the state feasibility and state value of the state, and calculate the trajectory feasibility of the third state trajectory segment based on the state feasibility and state value of each state.

[0211] In this example, the premise of generating a reasonable strategy is to ensure that the generated trajectory is feasible, that is, the transfer between two adjacent states in the trajectory can be achieved. Therefore, this application defines the state transition feasibility ω to evaluate the feasibility of transferring from the current state to the next state in the generated trajectory. Specifically, by statistically analyzing the difference distribution between the values ​​of all adjacent states, it can be found that the difference between adjacent states obeys a Gaussian distribution. This phenomenon is intuitive. In a normal trajectory, the value of a state should not change drastically after a transfer. When a drastic change occurs, it means that the strategy is facing failure. Based on this phenomenon, this application uses the following formula to define the state transition feasibility ω:

[0212]

[0213] Where x = V(s t+1 )-V(s t ). d and σ d The mean and variance of the distribution of the value differences of adjacent states in the second static data set statistically analyzed during the training phase. Function ρ(x) represents the Gaussian distribution The density function of .

[0214] It should be noted that the transition probability P(s) of adjacent states is not used in this example. t+1 |s t ) as the feasibility of state transition. This is because the state transition probability reflects the distribution of state transition probabilities under the behavior strategy, rather than the true state transition probability distribution. This distribution is limited by the performance of the behavior strategy, thus suppressing good behaviors that contradict the behavior strategy.

[0215] In this example, in order to select the best possible subsequent trajectory, the feasibility-weighted state value is used to evaluate the trajectory feasibility of the generated trajectory segments. The formula is as follows:

[0216]

[0217] where ξ i Indicates state s i The probability of appearing at time i is:

[0218]

[0219] Feasibility Factor Used to evaluate the trade-off between feasibility and value of a state. When the state-feasibility weight is 0, when When , the weight of the state value is approximately equal to zero. In this example The value of is set to 0.1.

[0220] Step 480 , determine whether the training termination condition is met, that is, the trajectory feasibility of all state trajectory segments has been obtained. If so, execute step 490 , otherwise repeat steps 460 - 470 .

[0221] Step 490: Obtain the state trajectory segment s corresponding to the maximum value of trajectory feasibility 0,p (τ), where

[0222] Step 4110: Prediction is completed and state trajectory segment s is obtained 0,p The state s in (τ) t and Input these two states into the inverse dynamics network to obtain action a t , and let the robot output the action.

[0223] The offline reinforcement learning method provided in this application is used to train the action prediction model to control the robot as case one, and other offline reinforcement learning algorithms based on the diffusion model are used to train the action prediction model to control the robot as case two and case three. The control results are as follows: in case one, the robot can successfully pick up and place the cube in the target area. In case two, although the robot picks up the target object, it does not place it in the target position. In case three, the robot fails to successfully pick up the target object.

[0224] Therefore, the offline reinforcement learning method provided in this embodiment of the present application can effectively utilize the training data set for learning, while avoiding unreasonable trajectories, reducing the impact of the inherent randomness in the diffusion model on the control stability of the robot, and improving the control performance of the robot.

[0225] The action prediction method provided in the present application selects multiple state trajectories generated by the diffusion model network through state feasibility and state value, and avoids unreasonable trajectories while selecting trajectories with higher value, thereby reducing the impact of the randomness inherent in the diffusion model on the control stability of the robot and improving the control performance of the robot.

[0226] The offline reinforcement learning training device provided in an embodiment of the present application is described below. The offline reinforcement learning training device described below and the offline reinforcement learning training method described above can be referenced to each other.

[0227] Figure 5 A schematic diagram of the structure of an offline reinforcement learning training device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the apparatus includes an acquisition module 510 , a first training module 520 , a label module 530 and a second training module 540 .

[0228] An acquisition module is used to acquire a first static data set of a robot controlled by a to-be-trained model; the first static data set includes a plurality of historical running trajectories of the robot; the network of the to-be-trained model includes a value function network, a diffusion model network and an inverse dynamics network;

[0229] A first training module, used for training a value function network based on a first static data set;

[0230] A label module is used to generate performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network;

[0231] The second training module is used to train the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label.

[0232] Specifically, according to an embodiment of the present application, any multiple modules among the acquisition module, the first training module, the label module and the second training module can be combined into one module for implementation, or any one of the modules can be split into multiple modules.

[0233] Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.

[0234] According to an embodiment of the present application, at least one of the acquisition module, the first training module, the label module, and the second training module can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them.

[0235] Alternatively, at least one of the acquisition module, the first training module, the label module and the second training module may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0236] The offline reinforcement learning training device provided in the embodiment of the present application trains the value function network through the first static data set, and can evaluate the quality of historical operation trajectories through the value function, and guide the diffusion model to identify the quality of the trajectories; by introducing the diffusion model network into the network of the model to be trained, complex trajectories can be modeled; by training the diffusion model network and the inverse dynamics network through trajectory fragments and performance labels, the diffusion model can be assisted in distinguishing the quality of trajectories, thereby improving the learning accuracy of the diffusion model, and improving the training efficiency and prediction accuracy of the model to be trained.

[0237] In some embodiments, the historical running trajectory includes the robot's state and action at each historical moment, as well as the environmental reward of the robot's interaction with the environment; the trajectory segment is obtained based on the following steps:

[0238] Determine the value of each state based on the trained value function network;

[0239] preprocessing the first static data set based on a Gaussian distribution of values;

[0240] Extracting a first historical running trajectory from the preprocessed first static data set, where the first historical running trajectory is any unextracted historical running trajectory;

[0241] Any state in the first historical running trajectory, the state after any state within a preset time interval, the action and the environmental reward are spliced ​​to obtain a trajectory segment.

[0242] In some embodiments, the tag module is specifically used to:

[0243] Use the trained value function network as the first value function network;

[0244] Input the terminal state of any trajectory segment into the first value function network to obtain the value corresponding to the terminal state output by the first value function network;

[0245] The performance label of any trajectory segment is determined based on the environmental rewards corresponding to each state in any trajectory segment and the value corresponding to the terminal state.

[0246] In some embodiments, the first training module is specifically used to:

[0247] Collecting the robot's state and environment reward from the first static dataset;

[0248] Update the loss function of the value function network based on the collected state and environmental rewards, and update the network parameters of the value function network;

[0249] When the training result meets the first preset termination condition, the training is terminated to obtain the trained value function network;

[0250] Among them, the first preset termination condition includes greater than or equal to the preset number of training times, or the loss function of the value function network converges; the loss function of the value function network is constructed based on the state, environmental rewards and hyperparameters of the value function network in the first static data set.

[0251] In some embodiments, after obtaining the trajectory segment, the method further includes: generating a second static data set based on the trajectory segment and each performance label. The second training module is specifically used for:

[0252] Randomly collect each trajectory segment and each performance label corresponding to each trajectory segment from the second static data set;

[0253] Extracting state trajectory segments from the trajectory segments, and updating the loss function of the diffusion model network and the loss function of the inverse dynamics network based on the state trajectory segments and each performance label, and updating the network parameters of the diffusion model network and the network parameters of the inverse dynamics network;

[0254] When the training result meets the second preset termination condition, the training is terminated;

[0255] Among them, the second preset termination condition includes being greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the diffusion model network is constructed based on the trajectory segments, the performance labels corresponding to the trajectory segments, and the hyperparameters of the diffusion model network; the loss function of the inverse dynamics network is constructed based on the states and actions in the trajectory segments and the hyperparameters of the inverse dynamics network.

[0256] In some embodiments, extracting a state trajectory segment from a trajectory segment includes:

[0257] Extract the states in each trajectory segment sequentially based on the time order;

[0258] The states in each extracted trajectory segment are spliced ​​in sequence to obtain a state trajectory segment.

[0259] It should be noted here that the offline reinforcement learning training device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned offline reinforcement learning training method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0260] The offline reinforcement learning training device provided in an embodiment of the present application is described below. The offline reinforcement learning training device described below and the offline reinforcement learning training method described above can be referenced to each other.

[0261] Figure 6A schematic diagram of the structure of the action prediction device provided in the embodiment of the present application is shown in FIG. Figure 6 As shown, the apparatus includes a value module 610 , a generation module 620 , a replacement module 630 , a collection module 640 , a construction module 650 and an action module 660 .

[0262] The value module is used to input the state of the robot in the current prediction environment into the value function network of the action prediction model to obtain the value corresponding to the state output by the value function network;

[0263] A generating module, used for generating a first state trajectory segment based on a Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise;

[0264] A replacement module, used for replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment;

[0265] A collection module, used for inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments;

[0266] A building module, used for forming a trajectory feasibility set by combining the trajectory feasibility of each trajectory segment in the state trajectory segment set;

[0267] An action module is used to input the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network;

[0268] Among them, the action prediction model is trained based on the above-mentioned offline reinforcement learning training method.

[0269] Specifically, according to an embodiment of the present application, any multiple modules among the value module, generation module, replacement module, collection module, assembly module and action module can be combined into one module for implementation, or any one of the modules can be split into multiple modules.

[0270] Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.

[0271] According to an embodiment of the present application, at least one of the value module, generation module, replacement module, collection module, assembly module and action module can be at least partially implemented as a hardware circuit, such as an FPGA, PLA, system on chip, system on substrate, system on package, ASIC, or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them.

[0272] Alternatively, at least one of the value module, the generation module, the replacement module, the collection module, the assembly module and the action module may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0273] It should be noted here that the action prediction device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned action prediction method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0274] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730 and a communication bus (Communications Bus) 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic command in the memory 730 to execute the above-mentioned offline reinforcement learning training method and action prediction method, and the offline reinforcement learning training method includes:

[0275] Acquire a first static data set of a robot controlled by a to-be-trained model; the first static data set includes multiple historical running trajectories of the robot; the network of the to-be-trained model includes a value function network, a diffusion model network, and an inverse dynamics network;

[0276] Training the value function network based on the first static data set;

[0277] Generate performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network;

[0278] The diffusion model network and the inverse dynamics network are trained based on each trajectory segment and each performance label.

[0279] Motion prediction methods include:

[0280] Input the state of the robot in the current prediction environment into the value function network of the action prediction model to obtain the value corresponding to the state output by the value function network;

[0281] Generate a first state trajectory segment based on a Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise;

[0282] Replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment;

[0283] Inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments;

[0284] The trajectory feasibility of each trajectory segment in the state trajectory segment set is combined into a trajectory feasibility set;

[0285] The state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set is input into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network;

[0286] Among them, the action prediction model is trained based on the above-mentioned offline reinforcement learning training method.

[0287] In addition, the logic commands in the above-mentioned memory can be implemented in the form of software function modules and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0288] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0289] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0290] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0291] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0292] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0294] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An offline reinforcement learning training method, characterized in that: include: Acquire a first static data set of a robot controlled by a motion prediction model to be trained; the first static data set includes multiple historical running trajectories of the robot; the network of the motion prediction model to be trained includes a value function network, a diffusion model network and an inverse dynamics network; the value function network is used to train the strategy of the diffusion model; the diffusion model network is a strategy network represented by the diffusion model; the inverse dynamics network is used to parse the state output by the diffusion model and convert the state into an action that the robot can perform; Training the value function network based on the first static data set; Generate performance labels corresponding to each trajectory segment of the historical running trajectory based on the trained value function network; Training the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label; The historical running trajectory includes the state and action of the robot at each historical moment, and the environmental reward of the robot interacting with the environment; the trajectory segment is obtained based on the following steps: Determine the value of each state based on the trained value function network; preprocessing the first static data set based on the Gaussian distribution of the value; Extracting a first historical running trajectory from the preprocessed first static data set, wherein the first historical running trajectory is any unextracted historical running trajectory; splicing any state in the first historical running trajectory, the state after any state within a preset time interval, the action and the environmental reward to obtain the trajectory segment; After obtaining the trajectory segment, the method further includes: generating a second static data set based on the trajectory segments and the respective performance labels; The step of training the diffusion model network based on each trajectory segment and each performance label includes: Randomly collecting each trajectory segment and each performance label corresponding to each trajectory segment from the second static data set; Extracting state trajectory segments from the trajectory segments, and updating the loss function of the diffusion model network and the loss function of the inverse dynamics network based on the state trajectory segments and the performance labels, and updating the network parameters of the diffusion model network and the network parameters of the inverse dynamics network; When the training result meets the second preset termination condition, the training is terminated; Among them, the second preset termination condition includes being greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the diffusion model network is constructed based on the trajectory segment, the performance label corresponding to the trajectory segment and the hyperparameters of the diffusion model network; the loss function of the inverse dynamics network is constructed based on the state, action in the trajectory segment and the hyperparameters of the inverse dynamics network.

2. The offline reinforcement learning training method according to claim 1, characterized in that: The performance labels corresponding to the various trajectory segments of the historical running trajectory are generated based on the trained value function network, including: Use the trained value function network as the first value function network; Inputting the terminal state of any trajectory segment into the first value function network to obtain the value corresponding to the terminal state output by the first value function network; A performance label of any trajectory segment is determined based on the environmental rewards corresponding to each state in any trajectory segment and the value corresponding to the terminal state.

3. The offline reinforcement learning training method according to claim 2, characterized in that: The training of the value function network based on the first static data set includes: Collecting the state and environment reward of the robot from the first static data set; Based on the collected state and environmental rewards, the loss function of the value function network is updated, and the network parameters of the value function network are updated; When the training result meets the first preset termination condition, the training is terminated to obtain the trained value function network; Among them, the first preset termination condition includes being greater than or equal to a preset number of training times, or the loss function of the value function network converges; the loss function of the value function network is constructed based on the state in the first static data set, the environmental reward and the hyperparameters of the value function network.

4. The offline reinforcement learning training method according to claim 1, characterized in that: The extracting the state trajectory segment from the trajectory segment comprises: Extract the states in each trajectory segment sequentially based on the time order; The states in the extracted trajectory segments are sequentially spliced ​​to obtain the state trajectory segment.

5. A motion prediction method, characterized in that: include: Inputting the state of the robot in the current prediction environment into the value function network of the action prediction model to obtain the value corresponding to the state output by the value function network; Generate a first state trajectory segment based on the Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise; Replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment; Inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments; The trajectory feasibility of each trajectory segment in the state trajectory segment set is combined into a trajectory feasibility set; Based on the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set, the state is input into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network; Wherein, the action prediction model is trained based on the offline reinforcement learning training method described in any one of claims 1 to 4.

6. An offline reinforcement learning training device, characterized in that: include: An acquisition module is used to acquire a first static data set of a robot controlled by a motion prediction model to be trained; the first static data set includes multiple historical running trajectories of the robot; the network of the motion prediction model to be trained includes a value function network, a diffusion model network and an inverse dynamics network; the value function network is used to train the strategy of the diffusion model; the diffusion model network is a strategy network represented by the diffusion model; the inverse dynamics network is used to parse the state output by the diffusion model and convert the state into an action that the robot can perform; A first training module, used for training the value function network based on the first static data set; A label module, used for generating a performance label corresponding to each trajectory segment of the historical running trajectory based on the trained value function network; A second training module, used for training the diffusion model network and the inverse dynamics network based on each trajectory segment and each performance label; The historical running trajectory includes the state and action of the robot at each historical moment, and the environmental reward of the robot interacting with the environment; the trajectory segment is obtained based on the following steps: Determine the value of each state based on the trained value function network; preprocessing the first static data set based on the Gaussian distribution of the value; Extracting a first historical running trajectory from the preprocessed first static data set, wherein the first historical running trajectory is any unextracted historical running trajectory; splicing any state in the first historical running trajectory, the state after any state within a preset time interval, the action and the environmental reward to obtain the trajectory segment; After obtaining the trajectory segment, the method further includes: generating a second static data set based on the trajectory segments and the respective performance labels; The step of training the diffusion model network based on each trajectory segment and each performance label includes: Randomly collecting each trajectory segment and each performance label corresponding to each trajectory segment from the second static data set; Extracting state trajectory segments from the trajectory segments, and updating the loss function of the diffusion model network and the loss function of the inverse dynamics network based on the state trajectory segments and the performance labels, and updating the network parameters of the diffusion model network and the network parameters of the inverse dynamics network; When the training result meets the second preset termination condition, the training is terminated; Among them, the second preset termination condition includes being greater than or equal to the preset number of training times, or the loss function of the diffusion model network converges; the loss function of the diffusion model network is constructed based on the trajectory segment, the performance label corresponding to the trajectory segment and the hyperparameters of the diffusion model network; the loss function of the inverse dynamics network is constructed based on the state, action in the trajectory segment and the hyperparameters of the inverse dynamics network.

7. An action prediction device, characterized in that: include: A value module, used to input the state of the robot in the current prediction environment into the value function network of the action prediction model, and obtain the value corresponding to the state output by the value function network; A generating module, configured to generate a first state trajectory segment based on the Gaussian distribution of the value; the first state trajectory segment carries Gaussian noise; A replacement module, used for replacing the state in the first state trajectory segment based on the historical state of the robot to obtain a second state trajectory segment; A collection module, used for inputting the second state trajectory segment and the state of the robot into the diffusion model network of the action prediction model to obtain a state trajectory segment set predicted by the diffusion model network; the state trajectory segment set includes a plurality of third state trajectory segments; A forming module, used for forming a trajectory feasibility set by forming the trajectory feasibility of each trajectory segment in the state trajectory segment set; An action module is used to input the state of the third state trajectory segment corresponding to the maximum feasibility in the trajectory feasibility set into the inverse dynamics network of the action prediction model to obtain the action of the robot output by the inverse dynamics network; Wherein, the action prediction model is trained based on the offline reinforcement learning training method described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the offline reinforcement learning training method as described in any one of claims 1 to 4 and the action prediction method as described in claim 5 are implemented.

Images