Offline meta-reinforcement learning model training method, device, equipment, medium and product

Abstract

The present invention provides an offline meta-reinforcement learning model training method, device, equipment, medium and product, which relates to the field of artificial intelligence technology. The method includes: training a context encoder based on the similarity between different training tasks to minimize the distance between the representation vectors of the training tasks with a similarity higher than a threshold, and maximizing the distance between the representation vectors of the training tasks with a similarity lower than a threshold; training a Q function network based on conservative Q learning to minimize the standard Bellman error and minimize the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy; training a strategy network based on a behavior regularized actor judge to limit the difference between the training strategy and the behavior strategy; and obtaining a trained offline meta-reinforcement learning model based on the trained context encoder, Q function network and strategy network. The present invention realizes the enhancement of the performance, robustness and generalization ability of the offline meta-reinforcement learning model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an offline meta-reinforcement learning model training method, device, equipment, medium and product. Background Art

[0002] The offline meta-reinforcement learning (OMRL) model reduces the cost of interacting with environmental data during training by using offline datasets, while having the ability to generalize to new tasks. Therefore, it has important practical value in fields that require collecting sufficient and difficult-to-obtain exploratory interaction data (such as healthcare, autonomous driving, intelligent robots, etc.).

[0003] However, current OMRL models still face the problem of context transfer, which is the distribution difference between the behavior policy context used to generate offline datasets and the policy generation context in the current training process. On the one hand, context transfer can lead to an overestimation of the Q function, which in turn harms the quality of policy learning and the effect of generalization to new tasks. On the other hand, context transfer affects the ability of the policy to perform task inference.

[0004] Existing OMRL models encode transformation tuples to obtain embedding vectors, aiming to distinguish tasks by minimizing the distance between embedding vectors of the same task while maximizing the distance between embedding vectors of different tasks. However, this approach often ignores the heterogeneity of similarities between tasks, hindering the effective extraction of common knowledge by meta-strategies, and thus affecting the robustness of task reasoning. Summary of the invention

[0005] The present invention provides an offline meta-reinforcement learning model training method, device, equipment, medium and product, which are used to solve the defects of offline meta-reinforcement learning models in the prior art, such as poor performance, robustness and generalization ability, and enhance the performance, robustness and generalization ability of offline meta-reinforcement learning models.

[0006] The present invention provides an offline meta-reinforcement learning model training method, comprising:

[0007] Based on the similarity between different training tasks, the context encoder is trained to minimize the distance between the representation vectors corresponding to the training tasks whose similarity is higher than a threshold, and to maximize the distance between the representation vectors corresponding to the training tasks whose similarity is lower than the threshold;

[0008] Based on conservative Q-learning, the Q-function network is trained to minimize the gap between the Q-values ​​corresponding to the training strategy and the Q-values ​​corresponding to the behavior strategy while minimizing the standard Bellman error;

[0009] The policy network is trained based on the behavior regularized actor-critic BRAC to limit the difference between the training policy and the behavior policy;

[0010] A trained offline meta-reinforcement learning model is obtained according to the trained context encoder, the trained Q function network and the trained policy network.

[0011] In some embodiments, the training of the context encoder based on the similarity between different training tasks includes:

[0012] Based on the distance metric learning loss function, the similarity between different training tasks is incorporated to construct the first loss function;

[0013] The context encoder is trained based on the first loss function.

[0014] In some embodiments, the method of incorporating similarities between different training tasks based on the distance metric learning loss function and constructing the first loss function further includes:

[0015] Sample training tasks containing task information from the offline data set and extract attribute features of each training task;

[0016] According to the attribute characteristics of each training task, the attribute distance between different training tasks is calculated;

[0017] The attribute distance between different training tasks is determined as the similarity between different training tasks.

[0018] In some embodiments, the training of the Q function network based on conservative Q learning includes:

[0019] On the basis of the standard Bellman error function, a second loss function is constructed by combining the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy; the conservative Q learning minimizes the Q value corresponding to the training strategy and maximizes the Q value corresponding to the behavior strategy;

[0020] Based on the second loss function, the Q function network is trained.

[0021] In some embodiments, the step of training the policy network based on BRAC includes:

[0022] Based on the expectation of the state value function for the state, combined with the KL divergence between the training strategy and the behavior strategy, the third loss function is constructed;

[0023] Based on the third loss function, the policy network is trained.

[0024] The present invention also provides an offline meta-reinforcement learning model training device, comprising:

[0025] A first training module is used to train the context encoder based on the similarity between different training tasks to minimize the distance between the representation vectors corresponding to the training tasks with similarity higher than a threshold and maximize the distance between the representation vectors corresponding to the training tasks with similarity lower than the threshold;

[0026] The second training module is used to train the Q function network based on conservative Q learning, and minimize the difference between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error;

[0027] The third training module is used to train the policy network based on BRAC to limit the difference between the training policy and the behavior policy;

[0028] An acquisition module is used to obtain a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network.

[0029] In some embodiments, the first training module includes:

[0030] A first construction unit is used to integrate the similarity between different training tasks on the basis of the distance metric learning loss function to construct a first loss function;

[0031] The first training unit is used to train the context encoder based on the first loss function.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an offline meta-reinforcement learning model training method as described above is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described offline meta-reinforcement learning model training methods.

[0034] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described offline meta-reinforcement learning model training methods.

[0035] The offline meta-reinforcement learning model training method, device, equipment, medium and product provided by the present invention enhance the robustness and generalization ability of the context encoder through the similarity between different training tasks, alleviate the problem of overestimation of the Q-value function by introducing conservative Q learning, and control the deviation of the training strategy through behavioral regularization, thereby enhancing the performance, robustness and generalization ability of the offline meta-reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is one of the flow charts of the offline meta-reinforcement learning model training method provided by the present invention;

[0038] Figure 2 This is the second flow chart of the offline meta-reinforcement learning model training method provided by the present invention;

[0039] Figure 3 It is a structural schematic diagram of the offline meta-reinforcement learning model training device provided by the present invention;

[0040] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Rapidly adapting to new environments is one of the capabilities that autonomous agents must possess when applied to complex real-world environments. Meta-reinforcement learning (meta-RL) has emerged as a cutting-edge technology. By refining the common knowledge between training tasks, the agent can achieve near-optimal performance when facing new tasks through only a limited number of adaptation steps in the meta-adaptation phase, greatly improving the adaptation efficiency and flexibility of the agent. However, the widespread application of the meta-RL framework faces a major bottleneck: the training process of the meta-strategy is highly dependent on a large number of diverse task trajectories, which requires huge time and resource costs.

[0043] Taking the household kitchen robot scenario as an example, different families have different kitchen types and layouts. If an immature model is used for online interaction in every kitchen where the robot is purchased, it will cause a lot of inconvenience to the buyer and even cause the buyer's property loss. In addition, it costs a lot to build simulation environments for different types of kitchens, and there are problems such as difficulty in ensuring the reliability of migration from simulation environments to real environments.

[0044] To overcome the above difficulties, offline meta-reinforcement learning has been proposed and has received widespread attention. The offline meta-reinforcement learning model trains meta-strategies by using pre-collected data sets that do not require additional exploration. This not only significantly reduces the cost of data collection, but also avoids security risks in the meta-training exploration process, while maintaining the ability to generalize to new tasks.

[0045] However, current offline meta-RL models still face the problem of context transfer, which is the distribution difference between the behavior policy context used to generate offline datasets and the policy generation context in the current training process. On the one hand, context transfer can lead to an overestimation of the Q function, which in turn impairs the quality of policy learning and the effect of generalization to new tasks. On the other hand, context transfer affects the ability of the policy to perform task inference.

[0046] Existing offline meta-reinforcement learning models encode transformation tuples to obtain embedding vectors, aiming to distinguish tasks by minimizing the distance between embedding vectors of the same task while maximizing the distance between embedding vectors of different tasks. However, this approach often ignores the heterogeneity of similarities between tasks, hindering the effective extraction of common knowledge by meta-strategies, and thus affecting the robustness of task reasoning.

[0047] The present invention provides an offline meta-reinforcement learning model training method to improve the performance of the offline meta-reinforcement learning meta-strategy when trained using only an offline dataset and its ability to migrate to new tasks, that is, to enhance the performance, robustness and generalization ability of the offline meta-reinforcement learning model.

[0048] Figure 1 This is one of the flow charts of the offline meta-reinforcement learning model training method provided by the present invention, such as Figure 1 As shown, the present invention provides an offline meta-reinforcement learning model training method, comprising:

[0049] Step 110, based on the similarity between different training tasks, the context encoder is trained to minimize the distance between the representation vectors corresponding to the training tasks with a similarity higher than a threshold, and to maximize the distance between the representation vectors corresponding to the training tasks with a similarity lower than a threshold.

[0050] Specifically, in order to enhance the ability of context encoders to distinguish different tasks, the idea of ​​metric learning is introduced. The training process of the context encoder is guided by maximizing the distance between the task representations of different training tasks in the embedding space and minimizing the distance between the task representations of the same training task in the embedding space. However, relying solely on maximizing the distance between the embedding vectors of different training tasks may ignore the similarities between the training tasks, thereby limiting the extraction of common features between different training tasks.

[0051] Therefore, based on the similarity between different training tasks, the context encoder is trained to minimize the distance between the representation vectors corresponding to the training tasks with a similarity higher than a threshold, and maximize the distance between the representation vectors corresponding to the training tasks with a similarity lower than a threshold, so that the representation vectors of the training tasks with high similarity in the embedding space are closer, while the representation vectors of the training tasks with low similarity in the embedding space maintain a larger distance, thereby training a context encoder that can effectively and robustly distinguish different training tasks and fully consider the common characteristics of the training tasks, thereby enhancing the robustness and generalization ability of the context encoder.

[0052] Step 120 , based on conservative Q learning, the Q function network is trained to minimize the difference between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error.

[0053] Specifically, CQL (conservative Q-learning) is introduced to train the Q-function network. While minimizing the standard Bellman error, it reduces overly optimistic Q-value estimates by minimizing the gap between the Q-value corresponding to the training strategy and the Q-value corresponding to the behavioral strategy.

[0054] Step 130 , based on the behavior regularized actor-critic BRAC, the policy network is trained to limit the difference between the training policy and the behavior policy.

[0055] Specifically, a behavior regularized actor critic (BRAC) is used to train the policy network, and the deviation of the training policy is controlled by limiting the difference between the training policy and the behavior policy.

[0056] Step 140, obtaining a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network.

[0057] For example, in the scenario of a household kitchen robot, the policy network controls the robot to perform actions such as moving, cooking, and serving dishes based on the robot's own state vector (such as the environment, its own posture information, etc.) and the task representation vector generated by the context encoder. When migrating to a new kitchen, the household kitchen robot takes appropriate actions based on the new state vector and the task representation vector inferred from running in the new scenario.

[0058] The offline meta-reinforcement learning model training method provided by the present invention enhances the robustness and generalization ability of the context encoder through the similarity between different training tasks, alleviates the problem of overestimation of the Q-value function by introducing conservative Q-learning, and controls the offset of the training strategy through behavioral regularization, thereby enhancing the performance, robustness and generalization ability of the offline meta-reinforcement learning model.

[0059] In some embodiments, training the context encoder based on similarities between different training tasks includes:

[0060] Based on the distance metric learning loss function, the similarity between different training tasks is incorporated to construct the first loss function;

[0061] Based on the first loss function, the context encoder is trained.

[0062] Specifically, the present invention aims to learn a network parameter Deterministic context encoder , can be from the context Infer the representation vector of the current task By optimizing the context encoder, it can efficiently learn a robust representation of the task based on context information, thereby clearly distinguishing different tasks in the embedding space.

[0063] Assume that each task can be represented by its corresponding transfer quadruple The only certainty is that Indicates the status, Indicates action, Indicates reward, Represents the next state. Based on this, a mapping is constructed: that is, using the context encoder from the training task Offline dataset The set of transfer quads sampled in In the above example, we learn a representation vector (i.e., embedding vector) ,in represents the number of quads in the sampled transfer quad, Represents the first Four tuples, Represents the training task The corresponding The state in the quadruple, Represents the training task The corresponding The actions in the quadruple, Indicates The reward in the quadruplet, Represents the training task The corresponding The next state in the quadruple, Represents the training task The representation vector of Indicated by On the set of four-tuples consisting of four-tuples, the representation vector of each four-tuple is expectations, The representation vector for each quadruple is a quadruple The context encoder for the input The output of , and the expectation of all outputs, can get the representation vector of the sampled four-tuple set.

[0064] By integrating the similarity between different training tasks into the distance metric learning loss function, we can obtain a new distance metric learning loss function that integrates task similarity information, namely the first loss function. The expression of the first loss function is as follows:

[0065]

[0066] In the formula, represents the first loss function, denotes the network parameters of the context encoder, with the superscript Indicates that the first loss function is an enhancement of the loss function based on distance metric learning. The acronym for distance metric learning. represents the number of training tasks, Indicates training tasks, Indicates training tasks, represents the indicative function, if If the content is true, the value is 1, otherwise the value is 0. Represents the training task The corresponding label, Represents the training task The corresponding label, Represents the training task The representation vector of Represents the training task The representation vector of Represents the training task With training tasks The similarity between This is to avoid small hyperparameters that divide by zero.

[0067] After constructing the first loss function, the parameters of the context encoder are adjusted according to the first loss function to obtain a trained context encoder.

[0068] The offline meta-reinforcement learning model training method provided by the present invention introduces a distance metric learning method and takes task similarity as part of the loss function to guide the context encoder to more accurately capture the common features between tasks. This design not only enhances the sensitivity of the context encoder to the similarity between tasks, but also significantly improves the robustness and generalization ability of the context encoder.

[0069] In some embodiments, based on the distance metric learning loss function, the similarity between different training tasks is incorporated, and before constructing the first loss function, the method further includes:

[0070] Sample training tasks containing task information from the offline data set and extract attribute features of each training task;

[0071] According to the attribute characteristics of each training task, the attribute distance between different training tasks is calculated;

[0072] The attribute distance between different training tasks is determined as the similarity between different training tasks.

[0073] Specifically, first, training tasks containing task information are sampled from the offline dataset, and attribute features of each training task are extracted (such as the target position when the target position is used as the criterion for distinguishing different tasks). The distance measurement method such as norm is used to calculate the attribute distance between different training tasks according to the attribute characteristics of each training task; finally, the attribute distance between different training tasks is determined as the similarity between different training tasks.

[0074] For example, for the training task With training tasks , then the training task and training tasks The similarity between them can be expressed as:

[0075]

[0076] In the formula, Represents the training task With training tasks The similarity between Represents the training task The attribute characteristics of Represents the training task Attribute characteristics.

[0077] The offline meta-reinforcement learning model training method provided by the present invention calculates the attribute distance between different training tasks, determines the attribute distance between different training tasks as the similarity between different training tasks, and realizes the quantification of the similarity between different training tasks.

[0078] In some embodiments, based on conservative Q learning, training the Q function network includes:

[0079] On the basis of the standard Bellman error function, a second loss function is constructed by combining the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy; the conservative Q learning minimizes the Q value corresponding to the training strategy and maximizes the Q value corresponding to the behavior strategy;

[0080] Based on the second loss function, the Q function network is trained.

[0081] Specifically, the present invention aims to learn a conservative lower bound of Q value to address the problem of overestimation of the value of out-of-distribution state-action pairs in offline reinforcement learning. Specifically, the present invention learns a conservative Q function such that The expected Q value of is the state value function The lower bound of the Q function is obtained by A Q-value minimization term is added and training is performed in an iterative update manner, which minimizes the Q-value while optimizing the standard Bellman error function.

[0082] make is the target Q value, where Indicates reward, represents the discount factor, Indicates that the sampling state from the offline dataset is , training strategy Select the action to perform When In the actor-critic network framework, based on the standard Bellman error function, the second loss function is constructed by combining the gap between the Q value under the behavior strategy and the Q value under the training strategy. The expression of the second loss function is as follows:

[0083]

[0084] In the formula, represents the second loss function, Represents the action-value function The corresponding network parameters are: represents the mathematical expectation, Represents the offline dataset corresponding to the current training task, Represents the representation vector in the training task Next, the state-action pair The corresponding Q value is, represents the target Q value, is a hyperparameter used to adjust the weight of the regularization term. Indicates in status and the representation vector of the training task Next select action behavioral strategies.

[0085] The first term of this formula is the standard Bellman error function, that is, in the current task dataset The mean square time difference error (TD error) on represents the square of the TD error, by minimizing the next state The corresponding Q value From the status To the next state Transfer Rewards The sum of the state The corresponding Q value The difference between them is used to optimize the Q function.

[0086] In the second term of this formula, Used to estimate the current training strategy in state The Q value under represents the Q value under the behavior strategy, that is, from the behavior strategy The sampling state in the data distribution is Action When expectations.

[0087] Conservative Q-learning adds a minimization term to the original standard Bellman error function. In this minimization term, the Q value corresponding to the state-action pair selected by the current training strategy is penalized by minimization, and the Q value corresponding to the state-action pair in the offline data set is compensated by maximization to reduce the estimation of the Q value of the state-action pair out of the distribution. In this way, conservative Q-learning addresses the problem of overestimation of the Q value of the state-action pair out of the distribution caused by the inconsistent distribution of the training strategy and the behavior strategy in offline reinforcement learning.

[0088] After constructing the second loss function, the parameters of the Q function network are adjusted according to the second loss function to obtain a trained Q function network.

[0089] The offline meta-reinforcement learning model training method provided by the present invention reduces overly optimistic Q-value estimates by minimizing the gap between the Q-value under the current strategy and the Q-value under the behavior strategy. By integrating CQL into the offline meta-reinforcement learning model, the Q-value is estimated more conservatively, effectively alleviating the problem of overestimation of the Q function of out-of-distribution state-action pairs, thereby ensuring more stable learning and better generalization to new tasks.

[0090] In some embodiments, training a policy network based on a behavior regularized actor-critic BRAC includes:

[0091] Based on the expectation of the state value function for the state, combined with the KL divergence between the training strategy and the behavior strategy, the third loss function is constructed;

[0092] Based on the third loss function, the policy network is trained.

[0093] Specifically, BRAC is used to optimize the training strategy , aiming to maintain the training strategy and behavioral strategies , avoiding the training strategy from making too many out-of-distribution exploration behaviors that deviate too much from the distribution of the offline dataset. In the actor-critic network framework, based on the expectation of the state value function for the state, combined with the KL divergence between the training strategy and the behavior strategy, the third loss function is constructed. The expression of the third loss function is as follows:

[0094]

[0095] In the formula, represents the third loss function, Represents the training strategy The corresponding network parameters are: represents the mathematical expectation, Indicates in status and the representation vector of the training task Next, according to the training strategy Select Action The probability of Represents the representation vector in the training task Next, the state-action pair The corresponding Q value is, Represents the offline dataset corresponding to the current training task, is a tunable hyperparameter, Indicates in status and the representation vector of the training task The probability of taking each action under the training strategy, Indicates in status and the representation vector of the training task The next behavior is the probability of taking each action according to the strategy, Indicates calculation of KL divergence.

[0096] The first term of this formula Represents the state value function in an offline setting In the offline dataset corresponding to the current training task The transfer quad contained in Medium status Specifically, for each state , all possible actions in this state , according to the current training strategy Select Action Probability Multiply by the action value function , and obtain the value function in this state , and then for all By finding the expectation, we can get the state value function The negative sign in front of this term is due to maximizing this term, but deep neural networks are currently implemented using the gradient descent method to minimize the target loss.

[0097] The second term of this formula is a hyperparameter, By minimizing the training strategy In Status and the representation vector of the training task The probability and behavior strategy of taking each action In Status and the representation vector of the training task The KL divergence (Kullback-Leibler divergence) between the probabilities of taking each action under the same distribution is used to minimize the difference between the training strategy and the behavior strategy, limit the degree to which the training strategy deviates from the behavior strategy during training, and reduce the selection of out-of-distribution actions.

[0098] The BRAC method is a policy gradient algorithm in the original actor-critic framework (i.e. The KL divergence between the training policy and the behavior policy is added to the first term of The addition of the KL divergence measures the difference between the current training policy and the behavior policy. Minimizing the KL divergence between the two policies can make the training policy choose actions with larger Q values ​​for a given state as much as possible during optimization, while not deviating too far from the behavior policy and making too many out-of-distribution exploration behaviors.

[0099] After constructing the third loss function, the parameters of the policy network are adjusted according to the third loss function to obtain a trained policy network.

[0100] The offline meta-reinforcement learning model training method provided by the present invention minimizes the difference between the training strategy and the behavior strategy through behavior regularization, limits the degree to which the training strategy deviates from the behavior strategy during the training process, and reduces the selection of out-of-distribution actions.

[0101] Figure 2 This is the second flow chart of the offline meta-reinforcement learning model training method provided by the present invention. Figure 2 As shown, the present invention provides an offline meta-reinforcement learning model training method, comprising the following steps:

[0102] Step 210, inputting an offline data set and a context-based offline meta-reinforcement learning model. Looping through steps 220 to 250 until convergence;

[0103] Step 220: Collect a training task set.

[0104] Sample training tasks containing task information from the offline dataset.

[0105] In step 230, the transition tuple of each training task is sampled as context.

[0106] In step 240 , the context encoder encodes the context as a task representation.

[0107] The context sampled from the offline dataset corresponding to the training task is used as the input of the context encoder, which infers the task representation.

[0108] Step 250, update the context encoder, policy network and Q function network according to the task representation.

[0109] Taking the task representation as input, the context encoder is optimized based on the distance metric learning method with task similarity constraints; the Q function network (i.e., the critic network) is updated according to conservative Q learning; the training strategy is optimized using BRAC to update the policy function (i.e., the actor network).

[0110] Step 260: output a conservative offline meta-reinforcement learning model with task similarity constraints.

[0111] Through the above process, conservative Q-learning and behavior regularization are combined with offline meta-reinforcement learning, and task similarity distance metric learning is integrated to propose a conservative offline meta-reinforcement learning method based on task similarity constraints. This method avoids overestimation of Q values ​​while ensuring the stability of learning and good generalization ability for new tasks.

[0112] The offline meta-reinforcement learning model training method provided by the present invention has the following advantages:

[0113] First, efficient adaptation and low cost advantages. The present invention adopts a model-free context-based offline meta-reinforcement learning model as a research benchmark. This type of method only needs to adapt to a limited number of steps on the new task to quickly reach a near-optimal performance level; at the same time, since a pre-collected data set is used to train the meta-strategy, it not only reduces the cost of data collection, but also avoids unsafe exploration during the meta-training process. It is particularly suitable for scenarios where real-time interactive training is not possible and safety requirements are high. The method proposed in the present invention not only enables the intelligent agent to quickly adapt to the environment in real-life scenarios, but also significantly reduces the time cost of data interaction, and has broad practical value.

[0114] Second, enhanced robustness and generalization. The present invention incorporates task similarity constraints into the training of the context encoder. By introducing a distance metric learning method and taking task similarity as part of the loss function, the present invention guides the context encoder to more accurately capture the common features between tasks. This design not only enhances the context encoder's sensitivity to similarities between tasks, but also significantly improves its robustness and generalization capabilities, allowing the intelligent agent to perform task inference more robustly when faced with unknown or similar new tasks, and thus adapt and learn more stably and effectively.

[0115] Third, enhanced performance. The present invention introduces a conservative Q-learning method in the training of the Q-function network, and adds a regularization term on the basis of the original Q-function. This method ensures that the expected value of the strategy under the Q-function is a lower bound of its true value, thereby alleviating the common problem of overestimation in traditional Q-learning methods. This improvement provides more accurate guidance for strategy training, thereby improving the average return and generalization ability of the offline meta-reinforcement learning model.

[0116] In summary, the present invention constructs an offline meta-reinforcement learning framework with high efficiency, low cost, strong robustness and good generalization performance by combining a model-free context-based offline meta-reinforcement learning method, context encoder training based on task similarity constraints and a conservative Q-learning method. This method broadens the application field of meta-reinforcement learning and provides strong technical support for the autonomous adaptation and learning of intelligent agents in complex and changing environments.

[0117] The offline meta-reinforcement learning model training device provided by the present invention is described below. The offline meta-reinforcement learning model training device described below and the offline meta-reinforcement learning model training method described above can refer to each other.

[0118] Figure 3 is a schematic diagram of the structure of the offline meta-reinforcement learning model training device provided by the present invention, such as Figure 3 As shown, the present invention provides an offline meta-reinforcement learning model training device, comprising:

[0119] A first training module 310 is used to train the context encoder based on the similarity between different training tasks to minimize the distance between the representation vectors corresponding to the training tasks with a similarity higher than a threshold, and maximize the distance between the representation vectors corresponding to the training tasks with a similarity lower than the threshold;

[0120] A second training module 320 is used to train the Q function network based on conservative Q learning to minimize the difference between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error;

[0121] a third training module 330 for training the policy network based on the behavior regularized actor-critic BRAC to limit the difference between the training policy and the behavior policy;

[0122] The acquisition module 340 is used to obtain a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network.

[0123] In some embodiments, the first training module 310 includes:

[0124] A first construction unit is used to integrate the similarity between different training tasks on the basis of the distance metric learning loss function to construct a first loss function;

[0125] The first training unit is used to train the context encoder based on the first loss function.

[0126] In some embodiments, the apparatus further comprises:

[0127] A processing module is used to sample training tasks containing task information from the offline data set and extract attribute features of each training task;

[0128] A calculation module, used to calculate the attribute distance between different training tasks according to the attribute characteristics of each training task;

[0129] The determination module is used to determine the attribute distance between different training tasks as the similarity between different training tasks.

[0130] In some embodiments, the second training module 320 includes:

[0131] The second construction unit is used to construct a second loss function based on the standard Bellman error function and in combination with the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy; the conservative Q learning minimizes the Q value corresponding to the training strategy and maximizes the Q value corresponding to the behavior strategy;

[0132] The second training unit is used to train the Q function network based on the second loss function.

[0133] In some embodiments, the third training module 330 includes:

[0134] A third construction unit is used to construct a third loss function based on the expectation of the state value function for the state and the KL divergence between the training strategy and the behavior strategy;

[0135] The third construction unit is used to train the policy network based on the third loss function.

[0136] It should be noted here that the above-mentioned offline meta-reinforcement learning model training device provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

[0137] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the offline meta-reinforcement learning model training method, which includes: based on the similarity between different training tasks, training the context encoder to minimize the distance between the representation vectors corresponding to the training tasks with similarity higher than a threshold, and maximizing the distance between the representation vectors corresponding to the training tasks with similarity lower than the threshold; based on conservative Q learning, training the Q function network to minimize the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error; based on the behavior regularization actor critic BRAC, training the policy network to limit the difference between the training strategy and the behavior strategy; and obtaining a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network.

[0138] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0139] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the offline meta-reinforcement learning model training method provided by the above methods, the method including: based on the similarity between different training tasks, training the context encoder to minimize the distance between the representation vectors corresponding to the training tasks with a similarity higher than a threshold, and maximizing the distance between the representation vectors corresponding to the training tasks with a similarity lower than the threshold; based on conservative Q learning, training the Q function network to minimize the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error; based on the behavior regularization actor critic BRAC, training the policy network to limit the difference between the training strategy and the behavior strategy; according to the trained context encoder, the trained Q function network and the trained policy network, a trained offline meta-reinforcement learning model is obtained.

[0140] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the offline meta-reinforcement learning model training method provided by the above-mentioned methods, the method comprising: training a context encoder based on the similarity between different training tasks to minimize the distance between representation vectors corresponding to training tasks with a similarity higher than a threshold, and maximizing the distance between representation vectors corresponding to training tasks with a similarity lower than the threshold; training a Q function network based on conservative Q learning to minimize the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error; training a policy network based on a behavior regularized actor critic BRAC to limit the difference between the training strategy and the behavior strategy; obtaining a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A household kitchen robot control method based on an offline meta-reinforcement learning model, characterized in that: include: Based on the similarity between different training tasks, the context encoder is trained to minimize the distance between the representation vectors corresponding to the training tasks whose similarity is higher than a threshold, and to maximize the distance between the representation vectors corresponding to the training tasks whose similarity is lower than the threshold; Based on conservative Q-learning, the Q-function network is trained to minimize the gap between the Q-values ​​corresponding to the training strategy and the Q-values ​​corresponding to the behavior strategy while minimizing the standard Bellman error; The policy network is trained based on the behavior regularized actor-critic BRAC to limit the difference between the training policy and the behavior policy; Obtaining a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network; Get the state vector of the home kitchen robot; The state vector is input into the trained offline meta-reinforcement learning model, the trained context encoder encodes the state vector into a task representation vector, and the trained policy network controls the household kitchen robot to perform the action of moving, cooking or serving dishes according to the task representation vector and the state vector; The training of the context encoder based on the similarity between different training tasks includes: Based on the distance metric learning loss function, the similarity between different training tasks is incorporated to construct the first loss function; The context encoder is trained based on the first loss function.

2. The household kitchen robot control method based on offline meta-reinforcement learning model according to claim 1 is characterized in that: The method further includes: integrating the similarity between different training tasks on the basis of the distance metric learning loss function and constructing the first loss function; Sample training tasks containing task information from the offline data set and extract attribute features of each training task; According to the attribute characteristics of each training task, the attribute distance between different training tasks is calculated; The attribute distance between different training tasks is determined as the similarity between different training tasks.

3. The household kitchen robot control method based on offline meta-reinforcement learning model according to claim 1 is characterized in that: The training of the Q function network based on conservative Q learning includes: On the basis of the standard Bellman error function, a second loss function is constructed by combining the gap between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy; the conservative Q learning minimizes the Q value corresponding to the training strategy and maximizes the Q value corresponding to the behavior strategy; Based on the second loss function, the Q function network is trained.

4. The household kitchen robot control method based on offline meta-reinforcement learning model according to claim 1 is characterized in that: The behavior regularized actor-critic BRAC is used to train the policy network, including: Based on the expectation of the state value function for the state, combined with the KL divergence between the training strategy and the behavior strategy, the third loss function is constructed; Based on the third loss function, the policy network is trained.

5. A household kitchen robot control device based on an offline meta-reinforcement learning model, characterized in that: include: A first training module is used to train the context encoder based on the similarity between different training tasks to minimize the distance between the representation vectors corresponding to the training tasks with similarity higher than a threshold and maximize the distance between the representation vectors corresponding to the training tasks with similarity lower than the threshold; The second training module is used to train the Q function network based on conservative Q learning to minimize the difference between the Q value corresponding to the training strategy and the Q value corresponding to the behavior strategy while minimizing the standard Bellman error; The third training module is used to train the policy network based on the behavior regularized actor-critic BRAC to limit the difference between the training policy and the behavior policy; An acquisition module, used to obtain a trained offline meta-reinforcement learning model according to the trained context encoder, the trained Q function network and the trained policy network; A control module is used to obtain a state vector of the household kitchen robot; input the state vector into the trained offline meta-reinforcement learning model, the trained context encoder encodes the state vector into a task representation vector, and the trained policy network controls the household kitchen robot to perform the action of moving, cooking or serving dishes according to the task representation vector and the state vector; The first training module includes: A first construction unit is used to integrate the similarity between different training tasks on the basis of the distance metric learning loss function to construct a first loss function; The first training unit is used to train the context encoder based on the first loss function.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the household kitchen robot control method based on the offline meta-reinforcement learning model as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the household kitchen robot control method based on the offline meta-reinforcement learning model as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the household kitchen robot control method based on the offline meta-reinforcement learning model as described in any one of claims 1 to 4 is implemented.

Images