Method, device and medium for improving control accuracy of offline reinforcement learning robot

Abstract

The present invention discloses a method, device and medium for improving the control accuracy of an offline reinforcement learning robot, belonging to the field of robot control. The method comprises: step 1, obtaining an offline data set containing damaged data and undamaged data; step 2, using the offline data set to train an offline reinforcement learning model for controlling the robot with a robust variational Bayesian inference method until the cumulative value of the reward is maximized; step 3, deploying the offline reinforcement learning model trained in step 2 on the robot to control the robot to complete a predetermined operation task. The Bayesian inference framework of the method captures the uncertainty caused by the damage of diversified data in the offline data set, reduces the negative effect of damaged data on the strategy, significantly improves the robustness and performance of the model in a clean environment, and also improves the accuracy of the robot controlled by the offline reinforcement learning model.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a method for improving the control accuracy of an offline reinforcement learning robot. Background Art

[0002] Offline reinforcement learning algorithms aim to learn effective policies from a fixed, pre-collected dataset, thus avoiding real-time interactions with complex environments. This learning paradigm has significant advantages in scenarios where data collection is expensive, risky, or difficult to conduct in real time, such as healthcare, autonomous driving, and industrial automation. However, due to limitations in dataset collection, offline reinforcement learning algorithms often face the core challenge of policy distribution shift, that is, there is a distribution shift between the behavior policies in the offline dataset and the policies learned during robot training. This shift usually causes the machine to overestimate out-of-distribution (OOD) actions, thereby affecting the effectiveness of the learned policy.

[0003] To address this challenge, existing solutions often introduce uncertainty estimation techniques, such as action-value function-based integration methods or Bayesian inference, to measure the uncertainty of the action-value function or dynamic model. These techniques limit the machine-learned strategies to the range of behavioral strategies in the offline dataset to a certain extent, thereby improving the robustness of the strategy.

[0004] However, in practical applications, offline datasets are often damaged to varying degrees due to sensor failures, malicious attacks, etc. These damages may manifest as random noise, adversarial attacks, or other forms of data perturbations, affecting key elements in the dataset, such as states, actions, rewards, and transition dynamics. Classic offline reinforcement learning algorithms often assume that the dataset is clean and intact. Therefore, when faced with damaged data, the strategies learned by the machine tend to be similar to those in the damaged data, resulting in a significant drop in performance after the machine is deployed in a clean environment.

[0005] Although researchers have made some progress in the field of robust offline reinforcement learning, such as some methods that attempt to mitigate the impact of noise or adversarial attacks by enhancing robustness during testing, most of them focus on learning on non-offline datasets and defending against attacks during testing, lacking effective methods for dealing with damaged training datasets. In addition, current robust offline reinforcement learning methods for damaged data often only focus on a specific type of data damage, such as damage in state or transition dynamics, and cannot effectively deal with complex situations where multiple elements of a dataset are damaged at the same time.

[0006] In the field of robot control, existing offline reinforcement learning algorithms often face data corruption problems due to sensor failure or malicious attacks. These damaged data usually introduce high uncertainty, causing the performance of existing algorithms to drop significantly when actually deployed, thus affecting the accuracy of robot control.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a method, device and medium for improving the control accuracy of an offline reinforcement learning robot, which can improve the control accuracy of a robot using an offline reinforcement learning algorithm, thereby solving the above-mentioned technical problems existing in the prior art.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A method for improving the control accuracy of an offline reinforcement learning robot, comprising:

[0011] Step 1, obtain an offline dataset containing damaged data;

[0012] Step 2, mapping the offline data set to an action value function, estimating the uncertainty value of the posterior distribution of the action value function by using a variational Bayesian inference method, and adjusting the weights used by the damaged data based on the uncertainty value to train an offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized;

[0013] Step 3: deploy the offline reinforcement learning model trained in step 2 on the robot to control the robot to complete the predetermined operation task.

[0014] A processing device, comprising:

[0015] at least one memory for storing one or more programs;

[0016] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0017] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0018] Compared with the prior art, the method, device and medium for improving the control accuracy of an offline reinforcement learning robot provided by the present invention have the following beneficial effects:

[0019] By using the offline data set to train the offline reinforcement learning model of the robot control with a robust variational Bayesian inference method until the cumulative value of the reward is maximized, the Bayesian inference framework is used to capture the uncertainty caused by the damage of the diverse data in the offline data set, and all damaged data are modeled as the uncertainty of the action value function, and the posterior distribution of the action value function is approximated by using the offline data, and the damaged data and the undamaged data are distinguished by the entropy-based uncertainty measure, so as to adjust the influence of the damaged data in the process of strategy learning, reduce the negative effect of the damaged data on the strategy, significantly improve the robustness and performance of the offline reinforcement learning model in a clean environment, and also improve the accuracy of the robot controlled by the offline reinforcement learning model. The method of the present invention can overcome the limitations of the existing offline reinforcement learning algorithm in dealing with the complex situation that multiple elements in the data set are damaged at the same time, thereby enhancing its effectiveness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for improving the control accuracy of an offline reinforcement learning robot provided by an embodiment of the present invention.

[0022] Figure 2 A schematic diagram of a probability model of the decision-making process of the method provided in an embodiment of the present invention.

[0023] Figure 3 The method provided in the embodiment of the present invention is tested in the lane-v0 task of the CARLA simulator, and the performance score graph under random mixed damage is shown.

[0024] Figure 4 A schematic diagram of the framework of a method for improving the control accuracy of an offline reinforcement learning robot provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0026] First, the terms that may be used in this article are explained as follows:

[0027] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0028] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0029] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0030] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0031] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0032] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0033] like Figure 1 As shown, an embodiment of the present invention provides a method for improving the control accuracy of an offline reinforcement learning robot, comprising:

[0034] Step 1, obtain an offline dataset containing damaged data;

[0035] Step 2, mapping the offline data set to an action value function, estimating the uncertainty value of the posterior distribution of the action value function by using a variational Bayesian inference method, and adjusting the weights used by the damaged data based on the uncertainty value to train an offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized;

[0036] Step 3: deploy the offline reinforcement learning model trained in step 2 on the robot to control the robot to complete the predetermined operation task.

[0037] Preferably, in the above method, the offline data set containing damaged data obtained in step 1 includes: a damaged action data subset consisting of damaged action data, a damaged state data subset consisting of damaged state data, and a damaged reward data subset consisting of damaged reward data.

[0038] Preferably, in step 2 of the above method, the offline data set is mapped to the action value function in the following manner, the uncertainty value of the posterior distribution of the action value function is estimated by a variational Bayesian inference method, and the weight used by the damaged data is adjusted based on the uncertainty value to train the offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized, including:

[0039] Step 21, input the three damaged data in the damaged offline data set into the critic network and value network of the offline reinforcement learning model to generate an action value function Sum value function , the three types of damaged data are: damaged state data , damaged motion data and damaged reward data ;

[0040] Step 22: Generate the action-value function Sum value function The input is an integrated model composed of three multi-layer perceptrons in parallel to reconstruct the three damaged data to obtain three reconstructed data. The three reconstructed data are: reconstructed state data , Reconstructed action data And the reconstructed reward data , estimate the posterior distribution of the action-value function based on the three reconstructed data, and calculate the uncertainty value of the posterior distribution of the action-value function;

[0041] Step 23, adjusting the usage weight of the damaged data according to the uncertainty value of the posterior distribution of the calculated action-value function, and updating the critic network and the integrated model using the three reconstructed data according to the determined usage weight of the damaged data;

[0042] Step 24: reconstruct the state data Input into the updated critic network to generate a new action value function ;

[0043] Step 25: Using the new action-value function Update actor network;

[0044] Step 26, repeat steps 21 to 25 to train the critic network, value network and actor network of the offline reinforcement learning model until the output value of the value network is maximized, and the training of the offline reinforcement learning model is completed.

[0045] Preferably, in step 22 of the above method, the generated action value function is Sum value function The input is an integrated model consisting of three multi-layer perceptrons in parallel to reconstruct three types of damaged data to obtain three types of reconstructed data, including:

[0046] Using multi-layer perceptron Likelihood function for modeling impaired motion data , then based on the variational Bayesian likelihood function of the damaged action data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged action data is obtained Formula (6) shows that the damaged action distribution The loss function of the posterior distribution of the cumulative reward is estimated by taking the damaged action data subset as the observation data, and formula (6) is:

[0047] (6);

[0048] In the above formula (6), the meanings of the parameters are: is the loss function identifier, indicating that it is based on the reconstructed action data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; It is a multi-layer perceptron used to reconstruct damaged motion data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling the damaged motion data to obtain the damaged motion data distribution; is subject to the distribution of damaged action data The reconstructed action data of It follows the action value distribution The new action-value function is parameterized by express; is subject to the damaged state distribution in the offline dataset Damaged state data; is subject to the damaged reward distribution in the offline dataset of damaged reward data; It follows the distribution of damaged state transitions in the offline dataset Damaged state data The damaged state data afterwards; is the damaged action distribution obeyed by the damaged action data of the offline dataset;

[0049] Using multi-layer perceptron Modeling the likelihood function of corrupted reward data , then based on the variational Bayesian likelihood function of the damaged reward data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data is obtained Formula (7) shows that the damaged reward distribution The loss function of the posterior distribution of the cumulative reward is estimated by taking the damaged reward data subset as the observation data. Formula (7) is:

[0050] (7);

[0051] In the above formula (7), the meanings of the parameters are: is the loss function identifier, indicating that it is based on the reconstructed reward data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; is a multi-layer perceptron used to reconstruct corrupted reward data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling damaged reward data to obtain damaged reward data distribution; is the damaged reward distribution in the offline dataset; Is subject to the damaged reward data distribution Reconstructed reward data of ; It follows the action value distribution The new action-value function is parameterized by express; is subject to the damaged state distribution in the offline dataset Damaged state data; is subject to the distribution of damaged actions in the offline dataset The damaged action data; is subject to the damaged state distribution in the offline dataset Damaged state data The damaged state data afterwards;

[0052] Using multi-layer perceptron Likelihood function for modeling corrupted state data , then based on the variational Bayesian likelihood function of the damaged state data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged state data is obtained Formula (8) shows that the damaged state distribution The loss function of the posterior distribution of the cumulative reward is estimated by using the damaged state data subset as the observation data. Formula (8) is:

[0053] (8);

[0054] In the above formula (8), the meanings of the parameters are: Is the loss function identifier, indicating that it is based on the reconstructed state data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; It is a multi-layer perceptron used to reconstruct damaged state data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling the damaged state data to obtain the damaged state data distribution; is subject to the damaged state data distribution The reconstructed state data; It follows the action value distribution The new action-value function is parameterized by express; is subject to the distribution of damaged actions in the offline dataset The damaged action data; is subject to the damaged reward distribution in the offline dataset of damaged reward data.

[0055] Preferably, in step 22 of the above method, the first KL divergence in formula (6) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged action data, the first KL divergence in formula (7) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data, and the first KL divergence in formula (8) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged state data are replaced by the generalized Bayesian inference to obtain the damaged data reconstruction loss The first optimization task objective is: the calculation method of the damaged data reconstruction loss of the first item is as follows:

[0056] (9);

[0057] In the above formula (9), the meanings of the parameters are: is the expected function; It is an offline dataset; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in the new action value function Follow the action value distribution ; Indicates the corresponding damaged reward data distribution The mean of Indicates the corresponding damaged reward data distribution The standard deviation of Indicates the corresponding damaged state data distribution The mean of Indicates the corresponding damaged state data distribution The standard deviation of Represents the corresponding damaged action data distribution The mean of Represents the corresponding damaged action data distribution The standard deviation of ; the superscript T indicates the transpose of the matrix;

[0058] The maximum likelihood loss is obtained by applying the Huber function using the generalized Bayesian inference to maximize the logarithmic probability of the second term in formula (6) for the posterior distribution of the maximum cumulative reward corresponding to the damaged action data, the logarithmic probability of the second term in formula (7) for the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data, and the logarithmic probability of the second term in formula (8) for the posterior distribution of the maximum cumulative reward corresponding to the damaged state data. The second optimization objective is to calculate the maximum likelihood loss of the second term as shown in the following formula (10):

[0059] (10);

[0060] In the above formula (10), the meaning of each parameter is: is the expected function; It is an offline dataset; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in ; , , They are respectively subject to the damaged state data distribution , distribution of damaged action data , Damaged Reward Data Distribution Samples of Multilayer Perceptron Reconstructing the damaged motion data to obtain the damaged motion data distribution; Multilayer Perceptron Reconstructing the damaged state data to obtain the damaged state data distribution; Multilayer Perceptron Reconstructing the damaged reward data to obtain the damaged reward data distribution; So is the Huber function of the threshold; is a randomly sampled value in the interval [0,1]; and These are all new action-value functions The quantile function corresponds to The quantile value of Indicates that the quantile value carries gradient information, which is used to update the parameters of the action value function , Indicates that the quantile value does not carry gradient information, is only a constant, and is not used to update the parameters of the action value function ; represents the action value distribution.

[0061] Preferably, in step 22 of the above method, the posterior distribution of the action-value function is estimated according to the three reconstructed data in the following manner, including:

[0062] Reconstruct the loss from the damaged data With maximum likelihood loss Adding together all the damaged data, we get the formula (11) for the integrated loss function that uses the posterior distribution of the action value function to calculate the integrated loss function, which is used to update the parameters of the posterior distribution of the action value function. , then based on the parameters The posterior distribution of the updated action-value function, formula (11) is:

[0063] (11);

[0064] In the above formula (11), the meanings of the parameters are: is the modeling action-value function The parameters of the critic network; To reconstruct the damaged state data among the three damaged data Multilayer Perceptron; To reconstruct the damaged action data among the three types of damaged data Multilayer Perceptron; To reconstruct the damaged reward data among the three damaged data Multilayer Perceptron; is the reconstruction loss of damaged data; is the maximum likelihood loss.

[0065] Preferably, in step 22 of the above method, after the posterior distribution of the action-value function is updated, the uncertainty value of the posterior distribution of the action-value function is calculated by formula (12) in the following manner: , formula (12) is:

[0066] (12);

[0067] In the above formula (12), the meanings of the parameters are: , They are the n-1th and nth values ​​randomly sampled in the interval [0,1] respectively; N is the total number of samples, and a fixed value of 32 is used in actual use; is the new action-value function The quantile function corresponds to Percentile value of ; is the new action-value function The quantile function corresponds to Percentile value of ; , ,when When the subscript n is 1, it is equal to the subscript n is 1 ; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in .

[0068] Preferably, in step 23 of the above method, the usage weight of the damaged data is adjusted in such a way that the usage weight of the damaged data is inversely proportional to the uncertainty value of the calculated posterior distribution of the action-value function.

[0069] The embodiment of the present invention further provides a processing device, including:

[0070] at least one memory for storing one or more programs;

[0071] At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the above method.

[0072] The embodiments of the present invention further provide a readable storage medium storing a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0073] In summary, the method of the embodiment of the present invention uses the offline data set to train the offline reinforcement learning model that controls the robot with a robust variational Bayesian inference method until the cumulative value of the reward is maximized. The Bayesian inference framework is used to capture the uncertainty caused by the damage of diverse data in the offline data set, and all damaged data are modeled as the uncertainty of the action value function. The posterior distribution of the action value function is approximated by using offline data, and the damaged data and the undamaged data are distinguished by an entropy-based uncertainty measure, so as to adjust the influence of the damaged data in the process of strategy learning, thereby reducing the negative effect of the damaged data on the strategy, significantly improving the robustness and performance of the model in a clean environment, and also improving the accuracy of the robot controlled by the offline reinforcement learning model. The method of the present invention can overcome the limitations of existing offline reinforcement learning algorithms in dealing with the complex situation that multiple elements in the data set are damaged at the same time, thereby enhancing its effectiveness in practical applications.

[0074] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0075] Example 1

[0076] like Figure 1 , Figure 4As shown, an embodiment of the present invention provides a method for improving the control accuracy of an offline reinforcement learning robot. The method uses a Bayesian inference framework to capture the uncertainty caused by the damage of diverse data in an offline data set, distinguishes damaged data from undamaged data according to uncertainty metrics, and adjusts the influence of damaged data during strategy learning, thereby reducing the negative effect of damaged data on the strategy, significantly improving the robustness and performance of the offline reinforcement learning model in a clean environment, and further improving the robustness and performance of the controlled robot. The method includes the following steps:

[0077] Step 1, obtain an offline dataset containing damaged data;

[0078] Step 2, mapping the offline data set to an action value function, estimating the uncertainty value of the posterior distribution of the action value function by using a variational Bayesian inference method, and adjusting the weights used by the damaged data based on the uncertainty value to train an offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized;

[0079] Step 3: deploy the offline reinforcement learning model trained in step 2 on the robot to control the robot to complete the predetermined operation task.

[0080] In the above step 2, the offline data set is mapped to the action value function in the following manner, the uncertainty value of the posterior distribution of the action value function is estimated by the variational Bayesian inference method, and the weight used by the damaged data is adjusted based on the uncertainty value to train the offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized, including:

[0081] Step 21, input the three damaged data in the damaged offline data set into the critic network and value network of the offline reinforcement learning model to generate an action value function Sum value function , the three types of damaged data are: damaged state data , damaged motion data and damaged reward data ;

[0082] Step 22: Generate the action-value function Sum value function The input is an integrated model composed of three multi-layer perceptrons in parallel to reconstruct the three damaged data to obtain three reconstructed data. The three reconstructed data are: reconstructed state data , Reconstructed action data And the reconstructed reward data , estimate the posterior distribution of the action-value function based on the three reconstructed data, and calculate the uncertainty value of the posterior distribution of the action-value function;

[0083] Step 23, adjusting the usage weight of the damaged data according to the uncertainty value of the posterior distribution of the calculated action-value function, and updating the critic network and the integrated model using the three reconstructed data according to the determined usage weight of the damaged data;

[0084] Step 24: reconstruct the state data Input into the updated critic network to generate a new action value function ;

[0085] Step 25: Using the new action-value function Update actor network;

[0086] Step 26, repeating steps 21 to 25 to train the critic network, value network and actor network of the offline reinforcement learning model until the output value of the value network is maximized, and the training of the offline reinforcement learning model is completed. The specific analysis process of the method of the present invention is as follows:

[0087] 1. Problem Analysis:

[0088] The problem of training an offline reinforcement learning algorithm to control a robot using an offline dataset containing corrupted data can be considered as a general Markov decision process (MDP) with the tuple It indicates that, is the state space, is the action space, It is a reward space. is a state-action pair Under the condition, the transition probability distribution of the next state is, is the probability distribution of the initial state, and is the discount factor. Note and Represent the state space and action space A collection of probability distributions over subsets.

[0089] For simplicity, we will use uppercase letters to represent random variables and lowercase letters to represent the values ​​that random variables take. Specifically, Indicates that the distribution follows The random variable of the single-step reward, and This random variable A value of .

[0090] Assume that for any , the random variable of the single-step reward and its expectation are given by and Definition.

[0091] The goal of this invention is to learn the optimal strategy to maximize the discounted cumulative reward:

[0092] (1);

[0093] Based on this cumulative reward, define the value function , the expectation of the action-value function (i.e., cumulative reward) , the action value function (i.e., the cumulative reward random variable) is:

[0094] ,

[0095] (2);

[0096] here .

[0097] In the real world, data collected by sensors or humans may be corrupted in various ways due to sensor failure or malicious attacks. and Respectively represent the non-offline dataset and the offline dataset, both containing samples . Assume that the state of the non-offline dataset follows the state distribution , the state of the offline dataset follows the state distribution , the actions of the offline dataset follow the behavior strategy , the actions of the offline dataset are The rewards of the offline dataset are sampled from The next state of the offline dataset is sampled from The empirical state-action distribution of the non-offline dataset and the empirical state-action distribution of the offline dataset are expressed as and . Furthermore, for any and , the Bellman formula under damaged data is introduced as:

[0098] (3);

[0099] Consider here Represents the equivalence of probability, random variables The distribution law and The distribution pattern is the same.

[0100] The method of the present invention involves fitting the distribution of the action-value function, so the distributional reinforcement learning paradigm is introduced to transform the task of fitting the distribution of the action-value function into learning the quantile function (also known as the inverse cumulative probability function) of the distribution of the action-value function. Parameterize the action-value function (or the random variable that accumulates the reward) , the action value function of The representation of the quantile function can be written as ,in is a dimensional vector, each dimensional vector is uniformly distributed It is worth noting that The n-th dimension vector of is a cumulative reward value, and its corresponding cumulative probability is Based on the temporal difference learning paradigm and quantile regression, we get The recursive optimization objective is:

[0101] (4);

[0102] here , is the value function The quantile function of Represents a uniform distribution The vector sampled from Total number of; So is the Huber function of the threshold, that is, the Huber function:

[0103] (5);

[0104] 2. Bayesian Inference Based on Damaged Data:

[0105] Considering that (1) multiple types of impairments introduce high uncertainty to all elements of the dataset and (2) there is a clear correlation between each element and the cumulative reward (i.e., action value, Q value) (see Figure 2 Therefore, estimating the cumulative reward function (i.e., action-value function) using a variety of corrupted data will introduce high uncertainty.

[0106] In order to deal with this kind of high uncertainty problem caused by multiple damaged data (i.e., damaged state, action, reward, and state transition data), based on Figure 2The probabilistic graphical model shown in the figure proposes to use all the damaged data in the offline dataset as observation data and exploit the high correlation between these observation data and the accumulated rewards to accurately identify the uncertainty of the action-value function. Figure 2 It is a probabilistic graphical model of the decision process, where the nodes connected by solid lines represent the data in the dataset, while the Q values ​​(i.e., action values, cumulative rewards) connected by dashed lines do not belong to the offline dataset. These Q values ​​are usually the task objectives that the method of the present invention aims to estimate.

[0107] Based on this idea, this paper proposes robust variational Bayesian inference in reinforcement learning (called TRACER) to learn uncertainty-aware cumulative reward representations to capture the uncertainty in offline datasets to cope with various data corruption situations. Specifically, all corrupted data are modeled as cumulative reward representations. In order to capture this uncertainty, variational Bayesian inference is introduced to estimate the posterior distribution of the cumulative reward using all offline damaged data as observation data.

[0108] based on Figure 2 , redefine the action value function ,generally The function uses the mean function, is the distribution of action value functions. We start with the distribution of damaged actions We start with a dataset of actions and use them as observations to estimate the posterior distribution of the cumulative reward. Since the action data is correlated with the cumulative reward and all other elements (state, reward, state transition), the likelihood function of the corrupted action data can be used with a multilayer perceptron Expressed as Then, based on the variational Bayesian inference framework, the cumulative reward is maximized The posterior distribution of , thereby minimizing the evidence lower bound (ELBO), that is, formula (6):

[0109] (6);

[0110] in, , , They follow the damaged state distribution , Damaged Reward Distribution , Damage state distribution ; is the Kullback-Leibler (KL) divergence.

[0111] Then, as in formula (6), we use the damaged reward distribution The damaged reward dataset is used as observation data, and the model is modeled using a multi-layer perceptron Parameterized likelihood function for corrupted reward data , we get formula (7):

[0112] (7);

[0113] in, and They follow the damaged state distribution and the distribution of impaired actions .

[0114] Finally, the application follows the damaged state distribution The state data set is used as observation data and the model is used Parameterized likelihood function , we get formula (8):

[0115] (8);

[0116] in, and Follow the damaged action distribution and damaged reward distribution .

[0117] Because there is no , and The explicit expression of cannot directly calculate the first term in formula (6), formula (7) and formula (8), namely the KL divergence term. Therefore, the generalized Bayesian inference is introduced to replace the two distribution positions of the KL divergence in the above formulas, and the task objective of the first term that can be optimized, namely the damaged data reconstruction loss, is approximately obtained as follows:

[0118] (9);

[0119] in, Represents the corresponding distribution The mean and standard deviation pairs of the parameters, the subscript a, subscript r, and subscript s correspond to the parameters of the damaged action data, the parameters of the damaged reward data, and the parameters of the damaged state data, respectively.

[0120] In addition, for the second item in formula (6), formula (7) and formula (8), considering that the damaged data will introduce sample points with heavy-tail distribution, the robust Huber function is applied based on the general Bayesian inference to maximize the logarithmic probability, and the task objective of the second item, namely the maximum likelihood loss, is obtained as follows:

[0121] (10);

[0122] in, , , They are respectively distributed ,distributed ,distributed A sample of the reconstructed data, It is an offline dataset Each damaged data sample in .

[0123] Adding the above formula (9) and formula (10) together, we get the formula (11) for the integrated loss function that uses all damaged data to calculate the posterior distribution of the action value function. This integrated loss function is used to update the parameters of the posterior distribution of the action value function. , then based on the parameters The posterior distribution of the updated action-value function, formula (11) is:

[0124] (11);

[0125] After obtaining the updated posterior distribution of the action-value function, the uncertainty value of the posterior distribution of the action-value function can be further estimated by the following formula (12):

[0126] (12);

[0127] in, is the nth value randomly sampled in the interval [0,1]; N is the total number of samples, and a fixed value of 32 is used in actual use; is the action-value function The quantile function corresponds to Percentile value of ; , when n is 1, equal ; They are the damaged offline datasets The damaged state data samples, damaged action data samples, and damaged reward data samples in .

[0128] Compared with the existing offline reinforcement learning algorithm, the method of the present invention has the following advantages:

[0129] (1) Using a Bayesian framework to capture and quantify the uncertainty caused by data corruption, we can more accurately handle the uncertainty in offline data and make the machine more robust when facing noisy data;

[0130] (2) Introducing an entropy-based uncertainty measure that can distinguish between damaged data and undamaged data, thereby adjusting the loss function associated with damaged data and reducing the impact of damaged data on robustness;

[0131] (3) Applied in the field of robot control, it effectively enhances the robustness of offline reinforcement learning algorithms when facing various types of data corruption.

[0132] In order to verify the effectiveness of the method of this embodiment, the method of this embodiment is applied to different simulation environments and data sets for testing. These tests are intended to evaluate the robustness and performance of the method of the present invention in a variety of scenarios. Considering the high cost and low efficiency of training and testing reinforcement learning algorithms in real environments, experiments were conducted in the MuJoCo simulation environment and the CARLA simulation environment, and the data set provided by D4RL (Datasets for Deep Data-Driven Reinforcement Learning) was used as an offline data set.

[0133] 1. MuJoCo simulation environment:

[0134] MuJoCo stands for Multi-Joint dynamics with Contact. It was developed by Professor Emo Todorov of the University of Washington. It is used in the fields of optimal control, state estimation, system identification, etc. It has obvious advantages in the application of dynamic multi-point contact of robots (such as multi-finger dexterous hand operation). Unlike other engines that use robot models such as urdf or sdf, the MuJoCo engine team has developed a robot modeling format (MJCF) to support more environmental parameter configurations.

[0135] This embodiment mainly uses three robot control environments provided by MuJoCo to implement and test the algorithm of the present invention. In these task environments, the observable states are different physical quantities (such as position, angle, speed, etc.) of various parts of the simulated robot (such as legs, joints, etc.), and the controllable actions are the magnitude of the force used by specific parts (such as legs, head). Specifically, the three simulated robot control environments are:

[0136] (11) HalfCheetah: This is a two-dimensional bipedal robot environment. The task is to control the robot to run forward at the fastest speed. The observation space includes physical quantities such as the robot's position and speed, while the action space involves the forces controlling the leg joints. The observation space is 17-dimensional and the action space is 6-dimensional.

[0137] (12) Walker2d: In this environment, the task is to control a 2D humanoid robot to walk stably. The observation space contains the physical state of each part of the robot (such as legs, joints, etc.), and the action space involves the forces applied to these parts. The observation space is 17-dimensional and the action space is 6-dimensional.

[0138] (13) Hopper: This is a single-legged hopping robot simulation environment. The task is to control the robot to jump forward on one leg. Since the robot has only one leg, maintaining balance is a major challenge in this task. The observation space is 11-dimensional and the action space is 3-dimensional.

[0139] (II) CARLA simulation environment:

[0140] CARLA is an open source simulation platform based on Unreal Engine, mainly used for autonomous driving research. The CARLA environment usually simulates urban driving scenes, including complex dynamic environments such as traffic lights, pedestrians, and vehicles. We mainly tested the performance of the TRACER algorithm in the CARLA-lane scene. The task of CARLA-lane is to keep the lane in an "8" shaped path while avoiding collisions with other vehicles. The observation space is 48×48×3 dimensions, that is, the observation data is a 48×48 RGB image, and the action space is 3D.

[0141] (21) Experimental setup:

[0142] Experiments were conducted with the probability of random or adversarial damage to the four elements of the offline dataset: state, action, reward, and state transition. Random damage is to add random noise to the data (i.e., sampling noise in a uniform distribution), while adversarial damage is to use the gradient direction of the pre-trained value function on the current data as attack noise. It should be noted that, unlike other types of adversarial damage, adversarial reward damage is achieved by multiplying the original reward data by This is achieved by using a gradient approach instead of a gradient approach. The core of this invention focuses on random or adversarial mixed impairments, which means that random or adversarial impairments exist simultaneously in the four elements of the data set. In the experiment of this invention, the impairment rate is set , that is, a single element in the offline data set is randomly selected % damaged; in the case of mixed damage, there are 1-( %) 4 In addition, the damage scale is set , which means that for random damage, the noise obeys the interval [- , ] is uniformly distributed; for adversarial damage, it means that the attack noise with gradient is in the interval [- , ] range. =30, =1.0. Note that when the impairment rate is set to 30%, a total of 76% of the data is noisy under mixed impairment.

[0143] (III) Experimental effect display:

[0144] The "medium-replay-v2" dataset provided by D4RL is used to verify the robustness of different algorithms in the face of data corruption in offline reinforcement learning. The collection method of this dataset is very close to real-world applications, and is collected by robots trained with SAC (Soft Actor-Critic). Random or adversarial corruption is implemented on the offline dataset to construct data corruption that may be faced in the real world. Based on this corrupted data, different algorithms are used to train robots, and the trained robots are deployed in a clean test environment to study the robustness of different algorithms in the face of corrupted data.

[0145] For each environment of MuJoCo, we compare TRACER to the previous state-of-the-art methods:

[0146] (31) BC: an algorithm based on imitation learning;

[0147] (32) CQL: An offline reinforcement learning algorithm using double Q-network.

[0148] (33) IQL: an algorithm based on implicit Q-learning;

[0149] (34) EDAC: An uncertain offline reinforcement learning algorithm implemented by integrating Q networks (the number of ensembles > 2).

[0150] (35) MSG: An uncertainty method that combines diversity and robustness.

[0151] (36) UWMSG: A recently proposed offline reinforcement learning algorithm based on the combination of uncertainty and robustness.

[0152] (37) RIQL: A state-of-the-art offline robust reinforcement learning algorithm for diverse offline datasets.

[0153] Table 1 shows the performance of the proposed method (i.e., TRACER) when multiple data elements are damaged at the same time. This performance is the generalization performance in a clean test environment without damage problems. The proposed method achieved a significant performance improvement in all experimental settings, with an improvement of +21.1%, which shows the strong robustness of the proposed method to large-scale damaged data.

[0154] Table 1 shows the average scores and standard errors of the robot after 3 million training times when the data is damaged by random mixture (random) and adversarial mixture (advers). The highest average performance is marked in bold.

[0155] ,

[0156] ;

[0157] Tables 2 and 3 show the average performance of the TRACER method in a clean environment when a single type of data element is damaged. Compared with mixed damage, in these two tables, the data damage is small-scale and accounts for a smaller proportion of the data set. In this type of scenario, the TRACER method can also achieve generalization performance improvement in 14 of the 24 experimental settings, and obtain the highest average performance score. It can be seen that TRACER is also effective for small-scale, single-type element damage problems.

[0158] Table 2 shows the mean scores and standard errors of the agents after 3 million training runs when one element in the data is randomly corrupted. The highest mean performance is marked in bold.

[0159] ,

[0160] ;

[0161] Table 3 shows the average scores and standard errors of the agents after 3 million training runs when one element in the data is adversarially impaired. The highest average performance is marked in bold.

[0162] ,

[0163] ;

[0164] In addition, experiments were conducted in the autonomous driving simulator CARLA. The “lane-v0” task was selected to perform random mixed damage processing on the data. The parameters used were c=30, =1.0. The experimental results are as follows Figure 3 The performance score of the method of the present invention in the lane-v0 task of the CARLA simulator under random mixing impairment is shown. In this type of task, the method of the present invention achieves the best generalization performance in a clean environment.

[0165] Example 2

[0166] This embodiment provides a method for improving the control accuracy of an offline reinforcement learning robot, including:

[0167] Preparation stage:

[0168] A random or adversarial attack is performed on the offline data set to add noise, thereby generating a batch of damaged data. The robot learns a strategy based on this batch of damaged data, and the goal of the present invention is to maximize the cumulative value of the reward.

[0169] Training phase: Based on Figure 3The framework implements TRACER. The specific training process is as follows:

[0170] 1) Input the (damaged) offline data into the critic network and value network to generate the action-value function Value Function ;

[0171] 2) Input the (damaged) offline data into an integrated model consisting of three multi-layer perceptrons to reconstruct the offline data;

[0172] 3) Change the status Input into the actor network to generate the currently learned strategy ;

[0173] 4) Update the loss function of each network and repeat the training several times.

[0174] Verification phase:

[0175] The learning strategy trained on the damaged data is deployed on a robot running in a clean test environment to evaluate the robustness of the learning strategy. The robot interacts with the environment, calculates the total reward obtained after a certain number of decisions, and normalizes the reward as an indicator of the evaluation effect. The higher the average score, the better the robustness of the method of the present invention.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0177] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for improving the control accuracy of an offline reinforcement learning robot, characterized in that: include: Step 1, obtain an offline dataset containing damaged data; Step 2, mapping the offline data set to an action value function, estimating the uncertainty value of the posterior distribution of the action value function by using a variational Bayesian inference method, and adjusting the weights used by the damaged data based on the uncertainty value to train an offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized; In step 2, the offline data set is mapped to the action value function in the following manner, the uncertainty value of the posterior distribution of the action value function is estimated by a variational Bayesian inference method, and the weight used by the damaged data is adjusted based on the uncertainty value to train the offline reinforcement learning model for controlling the robot until the cumulative value of the reward is maximized, including: Step 21, input the three damaged data in the offline dataset into the critic network and value network of the offline reinforcement learning model to generate an action value function Sum value function , the three types of damaged data are: damaged state data , damaged motion data and damaged reward data ; Step 22: Generate the action-value function Sum value function The input is an integrated model composed of three multi-layer perceptrons in parallel to reconstruct the three damaged data to obtain three reconstructed data. The three reconstructed data are: reconstructed state data , Reconstructed action data And the reconstructed reward data , estimate the posterior distribution of the action-value function based on the three reconstructed data, and calculate the uncertainty value of the posterior distribution of the action-value function; Step 23, adjusting the usage weight of the damaged data according to the uncertainty value of the posterior distribution of the calculated action-value function, and updating the critic network and the integrated model using the three reconstructed data according to the determined usage weight of the damaged data; Step 24: reconstruct the state data Input into the updated critic network to generate a new action value function ; Step 25: Using the new action-value function Update actor network; Step 26, repeating steps 21 to 25 to train the critic network, value network, and actor network of the offline reinforcement learning model until the output value of the value network is maximized, and the training of the offline reinforcement learning model is completed; In step 22, the generated action value function is Sum value function The input is an integrated model consisting of three multi-layer perceptrons in parallel to reconstruct three types of damaged data to obtain three types of reconstructed data, including: Using multi-layer perceptron Likelihood function for modeling impaired motion data , then based on the variational Bayesian likelihood function of the damaged action data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged action data is obtained Formula (6) shows that the damaged action distribution The loss function of the posterior distribution of the cumulative reward is estimated by taking the damaged action data subset as the observation data, and formula (6) is: （6）； In the above formula (6), the meanings of the parameters are: is the loss function identifier, indicating that it is based on the reconstructed action data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; It is a multi-layer perceptron used to reconstruct damaged motion data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling the damaged motion data to obtain the damaged motion data distribution; is subject to the distribution of damaged action data The reconstructed action data of It follows the action value distribution The new action-value function is parameterized by express; is subject to the damaged state distribution in the offline dataset Damaged state data; is subject to the damaged reward distribution in the offline dataset of damaged reward data; It follows the distribution of damaged state transitions in the offline dataset Damaged state data The damaged state data afterwards; is the damaged action distribution obeyed by the damaged action data of the offline dataset; Using multi-layer perceptron Modeling the likelihood function of corrupted reward data , then based on the variational Bayesian likelihood function of the damaged reward data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data is obtained Formula (7) shows that the damaged reward distribution The loss function of the posterior distribution of the cumulative reward is estimated by taking the damaged reward data subset as the observation data. Formula (7) is: （7）； In the above formula (7), the meanings of the parameters are: is the loss function identifier, indicating that it is based on the reconstructed reward data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; is a multi-layer perceptron used to reconstruct corrupted reward data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling damaged reward data to obtain damaged reward data distribution; is the damaged reward distribution in the offline dataset; Is subject to the damaged reward data distribution Reconstructed reward data of ; It follows the action value distribution The new action-value function is parameterized by express; is subject to the damaged state distribution in the offline dataset Damaged state data; is subject to the distribution of damaged actions in the offline dataset The damaged action data; is subject to the damaged state distribution in the offline dataset Damaged state data The damaged state data afterwards; Using multi-layer perceptron Likelihood function for modeling corrupted state data , then based on the variational Bayesian likelihood function of the damaged state data, the posterior distribution of the maximum cumulative reward is inferred, and the loss function of the posterior distribution of the maximum cumulative reward corresponding to the damaged state data is obtained Formula (8) shows that the damaged state distribution The loss function of the posterior distribution of the cumulative reward is estimated by using the damaged state data subset as the observation data. Formula (8) is: （8）； In the above formula (8), the meanings of the parameters are: Is the loss function identifier, indicating that it is based on the reconstructed state data To optimize the action-value function The loss function of is the modeling action-value function The parameters of the critic network; It is a multi-layer perceptron used to reconstruct damaged state data; is the expected function; It is an offline dataset; is the action value distribution; is the KL divergence; Multilayer Perceptron Modeling the damaged state data to obtain the damaged state data distribution; is subject to the damaged state data distribution The reconstructed state data of It follows the action value distribution The new action-value function is parameterized by express; is subject to the distribution of damaged actions in the offline dataset The damaged action data; is subject to the damaged reward distribution in the offline dataset of damaged reward data; In step 22, the first KL divergence in formula (6) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged action data, the first KL divergence in formula (7) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data, and the first KL divergence in formula (8) for obtaining the posterior distribution of the maximum cumulative reward corresponding to the damaged state data are replaced by the generalized Bayesian inference to obtain the damaged data reconstruction loss The first optimization task objective is: the calculation method of the damaged data reconstruction loss of the first item is as follows: （9）； In the above formula (9), the meanings of the parameters are: is the expected function; It is an offline dataset; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in the new action value function Follow the action value distribution ; Indicates the corresponding damaged reward data distribution The mean of Indicates the corresponding damaged reward data distribution The standard deviation of Indicates the corresponding damaged state data distribution The mean of Indicates the corresponding damaged state data distribution The standard deviation of Represents the corresponding damaged action data distribution The mean of Represents the corresponding damaged action data distribution The standard deviation of ; the superscript T indicates the transpose of the matrix; The maximum likelihood loss is obtained by applying the Huber function using the generalized Bayesian inference to maximize the logarithmic probability of the second term in formula (6) for the posterior distribution of the maximum cumulative reward corresponding to the damaged action data, the logarithmic probability of the second term in formula (7) for the posterior distribution of the maximum cumulative reward corresponding to the damaged reward data, and the logarithmic probability of the second term in formula (8) for the posterior distribution of the maximum cumulative reward corresponding to the damaged state data. The second optimization objective is to calculate the maximum likelihood loss of the second term as shown in the following formula (10): （10）； In the above formula (10), the meaning of each parameter is: is the expected function; It is an offline dataset; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in ; , , They are respectively subject to the damaged state data distribution , distribution of damaged action data , Damaged Reward Data Distribution Samples of Multilayer Perceptron Reconstructing the damaged motion data to obtain the damaged motion data distribution; Multilayer Perceptron Reconstructing the damaged state data to obtain the damaged state data distribution; Multilayer Perceptron Reconstructing the damaged reward data to obtain the damaged reward data distribution; So is the Huber function of the threshold; is a randomly sampled value in the interval [0,1]; and These are all new action-value functions The quantile function corresponds to The quantile value of Indicates that the quantile value carries gradient information, which is used to update the parameters of the action value function , Indicates that the quantile value does not carry gradient information, is only a constant, and is not used to update the parameters of the action value function ; represents the action value distribution; In step 22, the posterior distribution of the action value function is estimated based on the three reconstructed data in the following manner, including: Reconstruct the loss from the damaged data With maximum likelihood loss Add together the integrated loss function that uses all the damaged data to calculate the posterior distribution of the action value function Formula (11) is used to update the parameters of the posterior distribution of the action value function through the integrated loss function , then based on the parameters The posterior distribution of the updated action-value function, formula (11) is: （11）； In the above formula (11), the meanings of the parameters are: is the modeling action-value function The parameters of the critic network; To reconstruct the damaged state data among the three damaged data Multilayer Perceptron; To reconstruct the damaged action data among the three types of damaged data Multilayer Perceptron; To reconstruct the damaged reward data among the three damaged data Multilayer Perceptron; is the reconstruction loss of damaged data; is the maximum likelihood loss; Step 3: deploy the offline reinforcement learning model trained in step 2 on the robot to control the robot to complete the predetermined operation task.

2. The method for improving the control accuracy of an offline reinforcement learning robot according to claim 1, characterized in that: The offline data set containing damaged data obtained in step 1 includes: a damaged action data subset consisting of damaged action data, a damaged state data subset consisting of damaged state data, and a damaged reward data subset consisting of damaged reward data.

3. The method for improving the control accuracy of an offline reinforcement learning robot according to claim 1, characterized in that: In step 22, after the posterior distribution of the action-value function is updated, the uncertainty value of the posterior distribution of the action-value function is calculated by formula (12) in the following manner: , formula (12) is: （12）； In the above formula (12), the meanings of the parameters are: , They are the n-1th and nth values ​​randomly sampled in the interval [0,1] respectively; N is the total number of samples, and a fixed value of 32 is used in actual use; is the new action-value function The quantile function corresponds to Percentile value of ; is the new action-value function The quantile function corresponds to Percentile value of ; , ,when When the subscript n is 1, it is equal to the subscript n is 1 ; They are offline datasets The damaged state data samples, damaged action data samples and damaged reward data samples in .

4. The method for improving the control accuracy of an offline reinforcement learning robot according to claim 1, characterized in that: In step 23, the usage weight of the damaged data is adjusted in such a way that the usage weight of the damaged data is inversely proportional to the uncertainty value of the calculated posterior distribution of the action-value function.

5. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method described in any one of claims 1 to 4.

6. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 can be implemented.

Images